A robot motion strategy model optimization method and related device

Abstract

The present application discloses a robot motion strategy model optimization method and related devices, which relate to the field of artificial intelligence. The method comprises: performing preliminary training on the robot motion strategy model based on an autonomous interactive playback data memory library to obtain a preliminary trained robot motion strategy model; judging whether there are any scenes that do not meet the requirements according to the situation when the second robot performs the task; if so, retraining the preliminary trained robot motion strategy model based on a hybrid interactive playback data memory library for the scenes that do not meet the requirements to obtain a trained robot motion strategy model; the present application collects human demonstration data for the scenes where the preliminary trained robot motion strategy model does not perform the tasks that meet the requirements, so as to perform post-hybrid training in a targeted manner, so that the robot motion strategy model can meet the training requirements faster, improve the training efficiency of the robot motion strategy model, shorten the training cycle, and reduce the training cost.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a robot motion strategy model optimization method and related devices. Background Art

[0002] At present, when robots perform motion planning, they are generally based on a trained strategy model. Regarding the training of strategy models, two methods are often used: traditional reinforcement learning algorithms and traditional imitation learning algorithms based on human demonstration data. Among them, the traditional reinforcement learning algorithm is that the robot interacts with the training environment autonomously and conducts random exploration. It often requires a lot of interactions and the collection of a large amount of training data to achieve good training results; and the traditional imitation learning algorithm based on human demonstration data also needs to collect a large amount of demonstration data to cover as many environments and task states as possible, which requires a lot of manpower costs. In summary, the existing training methods all require high costs, such as computing costs or manpower costs, and the training cycle is long and the efficiency is low. Summary of the invention

[0003] The purpose of this application is to provide a robot motion strategy model optimization method and related devices, which can improve the training efficiency of the robot motion strategy model, shorten the training cycle, and reduce the training cost.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a robot motion strategy model optimization method, comprising:

[0006] Based on the autonomous interaction replay data memory library, the robot motion strategy model is preliminarily trained to obtain a preliminarily trained robot motion strategy model; the autonomous interaction replay data memory library includes first sample data; the first sample data is the interaction data between the first robot and the environment during the preliminary training process; the first robot is a robot based on a deep Q learning model; the deep Q learning model is constructed based on the robot motion strategy model.

[0007] According to the situation when the second robot performs the task, determine whether there are any scenarios that do not meet the requirements; the second robot is a robot based on a preliminary trained robot motion strategy model; the second robot is used to select the control action corresponding to the largest action value estimate value for output based on the action value estimate value of each control action output by the preliminary trained robot motion strategy model.

[0008] If so, the initially trained robot motion strategy model is retrained based on a hybrid interactive playback data memory library to obtain a trained robot motion strategy model; the hybrid interactive playback data memory library includes second sample data and third sample data; the second sample data is human demonstration data; the human demonstration data is the interaction data between the third robot and the environment in the scenario that does not meet the requirements; the third robot is a robot controlled by a human; the third sample data is the interaction data between the first robot and the environment during the retraining process.

[0009] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described robot motion strategy model optimization methods.

[0010] In a third aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned robot motion strategy model optimization methods.

[0011] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0012] The present application provides a robot motion strategy model optimization method and related devices. First, an autonomous training method is used to perform preliminary training on the robot motion strategy model to obtain a preliminary trained robot motion strategy model, so that it can achieve certain performance, such as learning obstacle avoidance or navigation skills in some basic scenarios; then, based on the situation when a second robot of the preliminary trained robot motion strategy model performs a task, human demonstration data is collected, so as to carry out targeted post-hybrid training, that is, based on a hybrid interactive playback data memory library, the preliminary trained robot motion strategy model is retrained. This training stage actually also includes an imitation learning training method, so that the robot motion strategy model can meet the training requirements faster, improve the training efficiency of the robot motion strategy model, shorten the training cycle, and reduce the training cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 A schematic diagram of a flow chart of a robot motion strategy model optimization method provided in one embodiment of the present application;

[0015] Figure 2 A schematic diagram of a robot motion strategy model provided in an embodiment of the present application;

[0016] Figure 3 A schematic diagram of the training process of the first stage of a robot motion strategy model provided by an embodiment;

[0017] Figure 4 A schematic diagram of the second phase of training of a robot motion strategy model provided by an embodiment;

[0018] Figure 5 A schematic diagram of a flow chart of another robot motion strategy model optimization method provided in one embodiment of the present application;

[0019] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0022] In an exemplary embodiment, Figure 1 As shown, a robot motion strategy model optimization method is provided, which includes the following steps 001 to 003 in the embodiment of the present application. Among them:

[0023] Step 001, based on the autonomous interaction playback data memory library, preliminarily train the robot motion strategy model to obtain a preliminarily trained robot motion strategy model; the autonomous interaction playback data memory library includes first sample data; the first sample data is the interaction data between the first robot and the environment during the preliminary training process; the first robot is a robot based on a deep Q learning model; the deep Q learning model is constructed based on the robot motion strategy model.

[0024] Step 002, based on the situation when the second robot performs the task, determine whether there is a scene that does not meet the requirements; the second robot is a robot based on a preliminary trained robot motion strategy model; the second robot is used to select the control action corresponding to the largest action value estimate value for output based on the action value estimate value of each control action output by the preliminary trained robot motion strategy model.

[0025] Step 003, if yes, then based on the hybrid interactive playback data memory library, the initially trained robot motion strategy model is retrained to obtain a trained robot motion strategy model; the hybrid interactive playback data memory library includes second sample data and third sample data; the second sample data is human demonstration data; the human demonstration data is the interaction data between the third robot and the environment in the scenario that does not meet the requirements; the third robot is a robot controlled by a human; the third sample data is the interaction data between the first robot and the environment during the retraining process.

[0026] See also Figure 2 As an optional implementation, the robot motion strategy model is an action value estimation network model; the action value estimation network model includes an input data preprocessing layer, a stacked convolutional layer, a bidirectional gated recurrent unit, a temporal state fusion layer, a first fully connected layer, a second fully connected layer and an action value estimation layer; wherein the input data preprocessing layer, the stacked convolutional layer, the bidirectional gated recurrent unit and the temporal state fusion layer are connected in sequence; the output of the temporal state fusion layer is respectively connected to the first fully connected layer and the second fully connected layer; the outputs of the first fully connected layer and the second fully connected layer are both connected to the action value estimation layer; the first fully connected layer is used for value estimation; the value estimation is the value estimation of the input state; the second fully connected layer is used for action advantage estimation of each control action; each of the control actions is a control action corresponding to the input state.

[0027] As an optional implementation, the action value estimation layer is used to estimate the action value of each control action, specifically including:

[0028] Estimate the value of the output of the first fully connected layer and the action advantage estimate of each of the control actions output by the second fully connected layer After adding, subtract the mean of the action advantage estimates of all control actions corresponding to the input state to obtain the action value estimate of each control action .

[0029] As an optional implementation, step 001 specifically includes:

[0030] Step 001.1, establish a first robot based on a deep Q learning model; the deep Q learning model includes a current Q network, a target Q network, an autonomous interactive playback data memory, an exploration module and a first loss function module; the current Q network and the target Q network both adopt the same network structure as the robot motion strategy model.

[0031] Step 001.2, randomly initialize the initial state of the first robot in the environment.

[0032] Step 001.3, storing the interaction data between the first robot and the environment in the form of a data group in the autonomous interaction playback data memory library; wherein the current Q network is used to calculate the action value estimation value of each control action according to the current state of the environment; the exploration module is used to select the output control action according to the action value estimation value of each control action; each of the data groups includes the environment state S at a certain moment t , the control action a corresponding to the environmental state at a certain moment t , the environmental reward R corresponding to the corresponding control action t and the next moment's environmental state S t+1 .

[0033] Step 001.4, when the number of the data groups in the autonomous interactive playback data memory reaches a preset threshold, randomly select a data group (S t 、a t , R t , S t+1 ), update the network parameters of the current Q network and the target Q network.

[0034] Step 001.5, judging whether the conditions for ending this round of training are met based on the task execution of the first robot; if not, returning to step 001.3.

[0035] Step 001.6, if yes, then end this round of training, add 1 to the training round, and return to step 001.2 until the training round reaches the preset round, and determine the current Q network as the robot motion strategy model for preliminary training.

[0036] As an optional implementation, step 001.4 specifically includes:

[0037] Step 001.4.1, the environmental state S in the data group t+1 Input to the target Q network and calculate the estimated value of the target action value.

[0038] Step 001.4.2, the environmental state S in the data group t Input to the current Q network and calculate the control action at The corresponding action value estimate.

[0039] Step 001.4.3, according to the first loss function module, the control action a t The corresponding environmental reward R t , the control action a t The corresponding action value estimate and the target action value estimate are used to calculate the loss value.

[0040] Step 001.4.4, update the network parameters of the current Q network according to the loss value and the back propagation algorithm.

[0041] Step 001.4.5: When the network parameters of the current Q network are updated a preset number of times, the network parameters of the target network are updated based on the network parameters of the current Q network.

[0042] In this application, the training of the robot motion strategy model (referred to as strategy model) for reinforcement learning is divided into two stages, wherein step 001 belongs to the first stage and step 003 belongs to the second stage.

[0043] As an optional implementation, in step 001, the network parameters of the robot motion strategy model are updated according to a first loss function and a back propagation algorithm; the first loss function is a temporal difference loss function.

[0044] That is to say, in the first stage of training (or optimization), the temporal difference error loss function is used to autonomously train the reinforcement learning strategy model. Specifically:

[0045] The action value estimation network based on the deep recurrent neural network is trained using the reinforcement learning method of deep Q-value learning. The deep neural network combined with the recurrent neural network structure is a common and very effective method for using deep neural networks as navigation models to cope with dynamic environments. The reasoning process of the recurrent neural network module in the deep recurrent network is shown in formula (3)-formula (9). The training process is as follows Figure 3 As shown, after starting training (201), the number of rounds (also called training rounds, or rounds for short) is first initialized (202), and the robot is randomly initialized in the training environment (203). Random initialization can improve the richness of robot training scenarios and improve the robustness of the strategy model obtained by robot training in multiple scenarios. The robot interacts with the training environment to generate interactive data. (204); Each set of interaction data Including the environment state before and after the interaction, the execution action and the environment reward. Each set of interaction data As a training sample; Store in autonomous interactive playback data memory (205); When the data memory When the number of samples in the training set reaches a threshold, a batch of training samples is extracted from the training set (206), and a loss function is calculated based on the environment reward (207) and the action value estimation network module parameters are updated through back propagation (208). Based on the preset conditions such as the robot task and safety, it is determined whether to end this round of training (209). If not, the robot will continue to interact with the environment to collect training sample data, and then return (204). On the contrary, if it is determined that this round of training is over, the number of rounds is increased by 1 (210). It is determined whether the number of rounds has reached the set number of rounds. (211); if not, return to (203) and start a new round of training; if so, end the training (212).

[0046] The mission objectives and safety objectives of a robot's operation are usually the key to judging whether it stops each time it performs a mission. Mission objectives (task preset conditions) include reaching a set spatial target point or flying more than a certain number of meters in a certain direction. Safety objectives (safety preset conditions) usually include avoiding collisions or staying within a certain number of meters of a certain object.

[0047] Among them, the temporal difference loss function Estimated value of the maximum target action corresponding to the state of the next time node Rewards for interacting with the current time node environment R The sum of the actual action value estimate , as shown in the following formula (1):

[0048] ; Formula (1)

[0049] in, Indicates the input status of the current time node; Represents the action value estimation network based on the input state The action that the selected robot performs; Indicates the input state of the next time node; Indicates input status The corresponding discrete actions; Represents the set of trainable parameters of the stacked convolutional layers and bidirectional gated recurrent units in the action value estimation network; Represents the trainable parameters of the fully connected layer corresponding to the value estimate in the action value estimation network; Represents the trainable parameters of the fully connected layer corresponding to the action advantage estimate in the action value estimation network; Represents the neural network parameter learning rate.

[0050] Each discrete control action (i.e., discrete action) corresponding to the input state is pre-defined based on the environment and task.

[0051] In this embodiment, the network loss function is set to the temporal difference error, and the Adam optimizer is used for training. The learning rate and momentum are 0.003, 0.9, and 0.99 respectively (the Adam algorithm calculates the exponential decay factors of the first-order moment and the second-order moment of the gradient through the two momentums of 0.9 and 0.99 respectively); the model is trained in batch training mode with a data batch size of 128, and a total of 1500 rounds of environmental interaction are performed to collect data and train.

[0052] When the robot performs autonomous navigation after preliminary training, the denoised and resized environmental depth image is sequentially passed through the stacked convolutional layer and the bidirectional gated recurrent unit network module to obtain the temporal environmental state hidden feature vector. The details are as follows:

[0053] See Figure 3 When the robot performs motion planning based on the trained action value estimation network based on the deep recurrent neural network (101), it first inputs the time series environment perception data and preprocesses it to obtain the input state matrix (102), where is the number of time nodes in the set time series data. Specifically, the depth camera obtains multiple time steps ( ) of the environment depth image, perform image denoising and size transformation (i.e., preprocessing) to adapt the action value estimation network proposed in this application.

[0054] The expression of the action value estimation network is shown in the following formula (2), where the input state , the action value module calculates the value estimate (108) and action advantage estimation (109), then add and subtract Discrete actions The corresponding action advantage estimates are averaged and then fused to obtain the action value estimate (110). For input status The corresponding discrete actions (i.e. optional actions),

[0055] ;Formula (2)

[0056] in, Represents the action value estimation network based on the input state The action that the selected robot performs, ;The action performed by the robot selected by the action value estimation network; A set of trainable parameters for the stacked convolutional layer (103) and the bidirectional gated recurrent unit (104); is the trainable parameter of the fully connected layer estimated by the corresponding value (106); similarly, are the trainable parameters of the fully connected layer corresponding to the action advantage estimate (107), N is the number of discrete output actions of the motion strategy model.

[0057] Stacked convolutional layers (103) and bidirectional gated recurrent units (104) are used to extract the input state The spatial and temporal characteristics of . First, the data is compressed through the stacked convolution layer (103) and the state feature matrix is ​​extracted through the convolution kernel. Then, the state features in the state feature matrix are sequentially input into the bidirectional gated recurrent unit (104) to obtain the time feature state values ​​of multiple time steps.

[0058] The bidirectional gated recurrent unit mainly consists of two gating structures: the update gate and the reset gate, and the candidate hidden state at the current moment and the current hidden state Two calculated variables.

[0059] for The bidirectional gated recurrent unit operation at a certain moment in the step sequence, the state feature vector at that moment First, after the reset gate operation, we get , as shown in the following formula (3):

[0060] ; Formula (3)

[0061] in, is the hidden state calculated by the bidirectional gated recurrent unit operation at the previous moment. If it is the first moment, is a zero vector of the same dimension, and To reset the parameters of the fully connected layer in the gate, is the bias vector, As the activation function, the Sigmoid function is used here, and its expression is:

[0062] ; Formula (4)

[0063] At the same time, the state eigenvector After the operation of the update gate, we get , as shown in the following formula (5):

[0064] ; Formula (5)

[0065] in, and To update the parameters of the fully connected layer in the gate, is the bias vector.

[0066] Candidate hidden states The calculation of is shown in formula (6). The current time step resets the gate output The hidden state at the previous time step Element-wise multiplication is performed by multiplying two vectors or matrices of the same dimension one by one. Then, the result of element-wise multiplication is added to the hidden state of the current time step, and the candidate hidden state is calculated through a fully connected layer with an activation function tanh. ,

[0067] ;Formula (6)

[0068] in, and is the fully connected layer parameter of the corresponding variable, is the bias vector.

[0069] Reset gate output Controls the hidden state of the previous time step How to flow into the candidate hidden state of the current time step. The hidden state of the previous time step may contain all the historical information of the time series up to the previous time step. Therefore, the reset gate can be used to discard historical information that is irrelevant to prediction.

[0070] The expression of the tanh function is:

[0071] ; Formula (7)

[0072] The update gate output You can control the hidden state of the current time step How should the candidate hidden state containing the information of the current time step be Updated as shown in formula (8):

[0073] ;Formula (8)

[0074] The above is the calculation process of the forward gated recurrent unit. The bidirectional gated recurrent unit cited in this application also includes a reverse gated recurrent unit. First, Reverse the time sequence in , and then repeat formula (3) to formula (8) for each time node in the set to obtain .

[0075] The temporal environment state hidden feature vector ( and ) are fused through the time series state fusion layer to obtain a feature vector, which is then input into two fully connected layers (i.e., the first fully connected layer and the second fully connected layer) for calculation to obtain value estimation and action advantage estimation. The value estimation and action advantage estimation are fused to obtain the final action value estimation.

[0076] As shown in formula (9), the time series state fusion layer (105) combines the feature vectors of different time nodes and Calculate the mean respectively, and further calculate the mean of the two forward and reverse propagation mean feature matrices to obtain the final state feature vector H ,

[0077] ; Formula (9)

[0078] The state feature vectors are input into the fully connected layers (106) and (107) to obtain the value estimation and action advantage estimation The action value estimate is obtained through formula (2): , and use the action with the largest estimated value among the discrete actions As the action to be executed by the robot at the current time step.

[0079] See also Figure 4 In this application, in the second stage, human demonstration data must first be collected. In this process, the person first observes the performance of the robot in performing tasks after the first stage of training, and then begins to collect a small number of samples of human expert data (301). The person (expert) judges the robot's strategic performance, that is, the degree of training, by observing the video recording of the robot performing tasks or the real-time navigation screen. In some simple scenarios, the strategy obtained by the robot's autonomous training may have been completed very well, so this part does not need to record or collect expert data; if it is observed that the robot's performance is insufficient, then experience data collection is carried out in these scenarios in a targeted manner.

[0080] The process of collecting human demonstration data includes: initializing the number of rounds (302); Randomly initialize the (robot) state (303); Humans (actually humans controlling the robot) interact with the (training) environment to generate interaction data (304); Each set of interaction data Including the environment state before and after the interaction, the execution action and the environment reward. Each set of interaction data As a training sample; Store in hybrid interactive playback data memory (305); Based on the robot's task and safety and other preset conditions, determine whether to end this round of training (306); If not, the robot will continue to interact with the environment to collect training sample data, that is, return to (304); On the contrary, if it is determined that this round of training is over, the number of rounds is increased by 1 (307); Determine whether the number of rounds has reached the set number of rounds ( Should be much smaller than ) (308); if not, return to (303) and start a new round of training; if yes, that is, enough demonstration data of people have been collected, then end the collection cycle (309). After enough demonstration data has been collected, enter the second stage of local strategy optimization process (i.e. step 003), which is also called the post-optimization stage.

[0081] As an optional implementation, in step 003, the network parameters of the robot motion strategy model are updated according to a second loss function and a back propagation algorithm; the second loss function is a weighted sum of a temporal difference loss function, a cross entropy loss function, and an L2 regularization loss function.

[0082] That is to say, in this application, the local strategy optimization process in the second stage (i.e., step 003) combines human demonstration data with data of autonomous interaction of reinforcement learning, integrates multiple loss functions, and simultaneously performs imitation learning and deep Q-value learning to further optimize the strategy model. This is equivalent to multi-task learning of the strategy model, which can effectively reduce the overfitting of the model to human demonstration data or autonomous interaction data of reinforcement learning. At the same time, by reasonably adjusting the weights of each loss function in training, the trained strategy model can achieve the ability to surpass the decision-making of robots or people alone.

[0083] Specifically, in the local strategy optimization process of the second stage, the loss function is shown in formula (10), which includes the temporal difference loss function , cross entropy loss function And the L2 regularization loss function to prevent model overfitting The temporal difference loss function uses the collected data to train the long-term value estimation ability of the policy model; the cross entropy loss function is only for the extracted demonstration data, which is used to efficiently imitate the human strategy; the L2 regularization loss function prevents the cross entropy loss function from causing the network model to overfit the human strategy represented by the human demonstration data. , and is the hyperparameter for balancing the three loss functions.

[0084] ; Formula (10)

[0085] As an optional implementation, in the local strategy optimization process, data samples in the mixed interaction playback data memory are extracted according to the sample priority. The priority of human demonstration data and robot autonomous interaction data will be calculated based on the environmental reward value of the round in which the sample data is collected. Specifically, the priority is proportional to the size of the round environment reward value, and all data in each round will be given the same sample priority. The local strategy optimization process (retraining) specifically includes: First, initialize the number of rounds (310); Randomly initialize the robot state (311); The robot interacts with the training environment to generate new autonomous interaction data (312); Each set of interaction data Including the environment state before and after the interaction, the execution action and the environment reward. Each set of interaction data As a training sample; Store in hybrid interactive playback data memory (313); Extract data samples from the mixed interactive playback data memory according to sample priority (314); Priority of human demonstration data at the beginning of local strategy optimization training It should be significantly higher than the priority of robot autonomous interaction data ; Calculate the loss function of the batch data (315), perform back propagation based on the loss and update the network parameters (316); Determine whether to end this round of training based on the preset conditions such as the robot's task and safety (317); If it is not ended, the robot will continue to interact with the environment to collect training sample data, that is, return (312); On the contrary, if it is determined that this round of training is over, add 1 to the number of rounds (318); Determine whether the number of rounds has reached the set number of rounds (319); if not, return to (311) and start a new round of training; if so, end the training (320).

[0086] In this embodiment, a total of The network loss function at this stage is set to combine the temporal difference error loss function, the cross entropy loss function and the L2 regularization loss function; Set to 1, Set to 0.01, The Adam optimizer is used for training, and its learning rate and momentum are 0.003, 0.9, and 0.99 respectively. The model is trained in batches, with a batch size of 128. The round environment interacts to collect data and conduct training.

[0087] As an optional implementation, after collecting the demonstration data of a person, the average reward value of the demonstration data (i.e., the first sample data) of the person is calculated using formula (11): :

[0088] ; Formula (11)

[0089] in, is the total number of demonstration data samples for humans, The environmental reward obtained by the robot when interacting with the environment for each sample collection.

[0090] Similarly, as shown in formula (12), the average reward value of the robot's autonomous interaction data (i.e., the second sample data) is The same method is used for calculation. is the total number of autonomous interaction data samples, The environmental reward obtained by the robot interacting with the environment when collecting each autonomous interaction sample,

[0091] ; Formula (12)

[0092] According to the above calculation and , the probability of sample extraction is calculated by formula (13) and . and Applicable to the datasets and In the training process, as the robot continues to collect more autonomous interaction data and the performance of its policy model task improves, the environmental reward value of each autonomous interaction data will usually continue to increase. According to formula (13), The increase will lead to The size of gradually increases, and the probability of autonomous interaction data being extracted continues to increase. Ultimately, it is expected that after sufficient training, a robot autonomous policy model that exceeds the performance of a human policy model will be obtained.

[0093] ; Formula (13)

[0094] Among them, whether the performance of the human strategy has been surpassed can be determined by judging whether the mean of the task rewards corresponding to the human demonstration data is greater than the mean of the task rewards obtained by the autonomous strategy.

[0095] Cross Entropy Loss Function The expression is as follows:

[0096] ; Formula (14)

[0097] in, is a symbolic function, when When the action is the same as in the human demonstration data, is 1; otherwise, is 0; Execute actions for the robot policy model probability.

[0098] The robot motion strategy model is an action value estimation network based on a deep recurrent neural network , in order to obtain the action The corresponding execution probability is The inferred action value is estimated using the Softmax function, as shown in formula (15):

[0099] ; Formula (15)

[0100] The L2 regularized loss function is defined as the sum of the squares of all trainable network parameters:

[0101] . Formula (16)

[0102] After the robot motion strategy model is trained, the robot compares the action value estimates of each discrete action based on the trained robot motion strategy model, and selects the action with the largest action value estimate to execute.

[0103] See also Figure 5 As an optional implementation, the robot motion strategy model optimization method may further include the following steps:

[0104] In step S11, the training of the reinforcement learning strategy model is divided into two stages. In the first stage, the reinforcement learning strategy model is autonomously trained using a temporal difference error loss function.

[0105] In step S12, when the robot performs autonomous navigation after preliminary training, the denoised and resized environmental depth image is sequentially passed through the stacked convolutional layer and the bidirectional gated recurrent unit network module to obtain the temporal environmental state hidden feature vector.

[0106] In step S13, the temporal environment state hidden feature vector is fused through the temporal state fusion layer to obtain a feature vector, and the feature vector is input into the fully connected layer for calculation to obtain a value estimate and an action advantage estimate, and the value estimate and the action advantage estimate are fused to obtain the final action value estimate.

[0107] In step S14, the second stage combines the human demonstration data with the data of autonomous interaction of reinforcement learning, integrates multiple loss functions, and simultaneously performs imitation learning and deep Q-value learning to further optimize the strategy model.

[0108] In step S15, the action value estimates of the robot's discrete actions are compared according to the strategy model, and the robot selects the action with the largest action value estimate to execute.

[0109] This application uses human demonstration data to improve the design of deep reinforcement learning methods based on value functions, and constructs a deep reinforcement learning method that integrates human demonstration data and reinforcement learning autonomous interaction data, so that action strategy models for complex environments and long-term tasks can be efficiently trained.

[0110] The present application has made important and substantial improvements to the reinforcement learning method based on deep convolutional networks, so that the optimization method of the action value estimation network based on deep recurrent neural networks of the present application can effectively improve the training effect of the model through phased data collection and loss function design without the need to collect a large amount of human demonstration data and a large amount of autonomous interaction data.

[0111] This application first divides the training of the model into two stages. In the first stage, the policy model autonomously interacts with the environment through a deep Q-value learning method and trains the model using a temporal difference loss function. After obtaining a model with basic motion strategy planning capabilities, people observe the planning capabilities of the policy model and summarize the environmental states where the model's autonomous planning fails. In the second stage, first based on the observation results, a small amount of human demonstration data is collected and added to the fusion playback data memory library, and then the robot will autonomously interact with the environment to continue to collect autonomous interaction data. Finally, training samples are extracted according to sample priority to train the model.

[0112] In the second stage, autonomous interaction data is still collected to build a fusion playback data memory library. On the one hand, it can prevent the robot strategy model from forgetting the excellent action strategy that has been trained in the first stage; on the other hand, it can prevent the problem of unstable training caused by the large distribution gap between human demonstration data and autonomous interaction data. In addition, this application also improves the feature extraction model, integrating a deep recurrent neural network module to enhance the model's ability to extract time features from time series input data.

[0113] This application proposes a method for training a robot motion strategy model based on a deep recurrent neural network by combining human demonstration experience data with autonomous interaction data of reinforcement learning. The method includes autonomous training based on deep Q-value learning in the first stage and deep Q-value learning combined with human demonstration data in the second stage. The deep reinforcement learning algorithm itself is used to guide the intelligent agent to train from scratch. After reaching a certain performance, that is, learning obstacle avoidance or navigation skills for some basic scenes, people collect small batches of demonstration data based on their training results. This application adds a lightweight recurrent neural network module to the deep network to improve the navigation ability of the robot decision network model in a dynamic environment. And a loss function for multi-loss fusion is proposed, which takes into account the pros and cons of the two training algorithms of imitation learning and reinforcement learning.

[0114] The present application also provides an application scenario, which applies the above-mentioned robot motion strategy model optimization method. Specifically: the robot motion strategy model optimization method provided in this embodiment can be applied in the robot motion strategy model optimization scenario.

[0115] A robot motion strategy model optimization method of the present application has the following beneficial effects:

[0116] (1) On the basis of effectively combining human demonstration data and robot autonomous interaction data to improve the model task performance, a two-stage training method of autonomous training followed by hybrid training is used to effectively reduce the dependence on a large amount of human demonstration data and avoid overfitting of the model to the demonstration data.

[0117] (2) The hybrid loss function proposed in this application considers multiple issues of model training at the same time. The temporal difference loss function trains the policy model's ability to estimate long-term action values; the cross entropy loss function effectively improves the policy model's ability to imitate human demonstration strategies; and the L2 regularization loss function prevents the policy model from overfitting to human demonstration strategies.

[0118] (3) This application introduces a lightweight deep recurrent neural network into the action value estimation network to improve the network model's ability to extract temporal information features and further enhance its ability to cope with dynamic environments and long-term tasks.

[0119] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to train related data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a robot motion strategy model optimization method is implemented.

[0120] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0121] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0122] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0123] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0124] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0125] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0126] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A robot motion strategy model optimization method, characterized in that: The robot motion strategy model optimization method comprises: Based on the autonomous interaction playback data memory library, the robot motion strategy model is preliminarily trained to obtain a preliminarily trained robot motion strategy model; the autonomous interaction playback data memory library includes first sample data; the first sample data is the interaction data between the first robot and the environment during the preliminary training process; the first robot is a robot based on a deep Q learning model; the deep Q learning model is constructed based on the robot motion strategy model; the robot motion strategy model is an action value estimation network model; the action value estimation network model includes an input data preprocessing layer, a stacked convolutional layer, a bidirectional gated recurrent unit, a temporal state fusion The first fully connected layer, the second fully connected layer and the action value estimation layer are connected in sequence; wherein the input data preprocessing layer, the stacked convolution layer, the bidirectional gated recurrent unit and the time series state fusion layer are connected in sequence; the output of the time series state fusion layer is connected to the first fully connected layer and the second fully connected layer respectively; the output of the first fully connected layer and the second fully connected layer are both connected to the action value estimation layer; the first fully connected layer is used for value estimation; the value estimation is the value estimation of the input state; the second fully connected layer is used for action advantage estimation of each control action; each of the control actions is the control action corresponding to the input state; According to the situation when the second robot performs the task, it is judged whether there is a scene that does not meet the requirements; the second robot is a robot based on the preliminarily trained robot motion strategy model; the second robot is used to select the control action corresponding to the maximum action value estimation value for output according to the action value estimation value of each control action output by the preliminarily trained robot motion strategy model; If yes, then based on the hybrid interactive playback data memory library, the initially trained robot motion strategy model is retrained to obtain a trained robot motion strategy model; the hybrid interactive playback data memory library includes second sample data and third sample data; the second sample data is human demonstration data; the human demonstration data is the interaction data between the third robot and the environment in the scenario that does not meet the requirements; the third robot is a robot controlled by a human; the third sample data is the interaction data between the first robot and the environment during the retraining process; The step of "preliminarily training the robot motion strategy model based on the autonomous interactive playback data memory library to obtain a preliminarily trained robot motion strategy model" specifically includes: Establishing a first robot based on a deep Q learning model; the deep Q learning model includes a current Q network, a target Q network, an autonomous interactive playback data memory library, an exploration module and a first loss function module; the current Q network and the target Q network both adopt the same network structure as the robot motion strategy model; Randomly initialize the initial state of the first robot in the environment; The interaction data between the first robot and the environment is stored in the autonomous interaction playback data memory in the form of a data group; wherein the current Q network is used to calculate the action value estimation value of each control action according to the current state of the environment; the exploration module is used to select the output control action according to the action value estimation value of each control action; each of the data groups includes the environment state S at a certain moment t , the control action a corresponding to the environmental state at a certain moment t , the environmental reward R corresponding to the corresponding control action t and the next moment's environmental state S t+1 ; When the number of the data groups in the autonomous interaction playback data memory reaches a preset threshold, a data group (S) is randomly selected from the autonomous interaction playback data memory. t 、a t , R t , S t+1 ), update the network parameters of the current Q network and the target Q network; According to the task execution of the first robot, determine whether the end condition of this round of training is met; if not, return to the step of "storing the interaction data between the first robot and the environment in the form of a data group in the autonomous interaction playback data memory library"; If so, end this round of training, add 1 to the number of training rounds, and return to step "randomly initialize the initial state of the first robot in the environment" until the number of training rounds reaches the preset number of rounds, and determine the current Q network as the robot motion strategy model for preliminary training.

2. The robot motion strategy model optimization method according to claim 1, characterized in that: The action value estimation layer is used to estimate the action value of each control action, specifically including: Estimate the value of the output of the first fully connected layer and the action advantage estimate of each of the control actions output by the second fully connected layer After adding, subtract the mean of the action advantage estimates of all control actions corresponding to the input state to obtain the action value estimate of each control action .

3. The robot motion strategy model optimization method according to claim 1, characterized in that: In the step "retraining the initially trained robot motion strategy model based on the hybrid interactive playback data memory to obtain a trained robot motion strategy model", the network parameters of the robot motion strategy model are updated according to the second loss function and the back propagation algorithm; the second loss function is the weighted sum of the temporal difference loss function, the cross entropy loss function and the L2 regularization loss function.

4. The robot motion strategy model optimization method according to claim 1, characterized in that: In the step "retraining the initially trained robot motion strategy model based on the hybrid interactive playback data memory to obtain a trained robot motion strategy model", sample data in the hybrid interactive playback data memory is extracted according to the sample data extraction probability; wherein the sample data extraction probability is calculated according to the sample data extraction probability calculation formula; the sample data extraction probability calculation formula is as follows: ； in, is the probability of extracting the first sample data; is the probability of extracting the second sample data; is the average reward of the first sample data; is the average reward of the second sample data.

5. The robot motion strategy model optimization method according to claim 1, characterized in that: In the step "based on the autonomous interactive playback data memory library, the robot motion strategy model is preliminarily trained to obtain a preliminarily trained robot motion strategy model", the network parameters of the robot motion strategy model are updated according to the first loss function and the back propagation algorithm; the first loss function is a temporal difference loss function.

6. The robot motion strategy model optimization method according to claim 1, characterized in that: Step "When the number of the data groups in the autonomous interactive playback data memory reaches a preset threshold, randomly select a data group (S) from the autonomous interactive playback data memory t 、a t , R t , S t+1 ), update the network parameters of the current Q network and the target Q network", specifically including: The environmental state S in the data set t+1 Input to the target Q network and calculate the estimated value of the target action; The environmental state S in the data set t Input to the current Q network and calculate the control action a t The corresponding action value estimate; According to the first loss function module, the control action a t The corresponding environmental reward R t , the control action a t The corresponding action value estimate and the target action value estimate are used to calculate the loss value; Update the network parameters of the current Q network according to the loss value and the back propagation algorithm; When the network parameters of the current Q network are updated for a preset number of times, the network parameters of the target network are updated based on the network parameters of the current Q network.

7. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the robot motion strategy model optimization method according to any one of claims 1 to 6.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the robot motion strategy model optimization method described in any one of claims 1 to 6 is implemented.

Images