Robot Behavior Decision Method and Device Based on Hierarchical Deep Reinforcement Learning Model

Abstract

The present application provides a robot behavior decision-making method and device based on a hierarchical deep reinforcement learning model. By simulating the hierarchical organization mechanism of the human brain, the present application proposes a deep hierarchical reinforcement learning model for solving the sparse reward problem of robots. The model includes two parts: a top-level module and a bottom-level module. During the process of the robot's environmental cognition, when it is in an environment with sparse rewards, the upper module sets sub-goals for the lower module according to the interaction between the agent and the environment. At the same time, the upper module can perceive the environment and predict the state transition of the agent. In addition, predicted rewards and reward gains are added to the deep hierarchical reinforcement learning model to accelerate the learning speed of the lower module. In the lower module, a dimensionality reduction network is designed to encode and map state information. Therefore, the deep hierarchical reinforcement learning model can effectively solve the sparse reward problem of the agent.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular, to a robot behavior decision-making method and device based on a hierarchical deep reinforcement learning model. Background Art

[0002] In the prior art, with the development of technology, mobile robots have been more and more widely used in human production and life, such as in industries, agriculture, military, medical, domestic service, finance, space exploration, education fields, etc. To meet these different task requirements, mobile robots must have good behavior decision-making performance.

[0003] Among different robot behavior decision-making methods, reinforcement learning has been widely used due to advantages such as self-learning and dynamic stochastic optimization, such as autonomous driving of autonomous vehicles, overtaking decision-making during the autonomous driving process, learning behavior modeling of military logistics agents, robot navigation and obstacle avoidance, voice-visual gaze control of robots, multi-robot path planning, task allocation of multi-robot systems, robot assembly tasks, etc.; relevant research indicates that the extraction of significant events in episodic memory can effectively shorten the thinking time during the behavior decision-making process compared with ordinary events, indicating that the significant events stored in the hippocampus can preferentially propagate reward information, allowing decision-makers to learn action strategies faster.

[0004] With the development of deep learning algorithms, deep reinforcement learning algorithms have emerged and have been increasingly applied in mobile robot behavior decision-making, such as driving planning and vehicle following of autonomous moving vehicles, collaborative behavior decision-making of multiple autonomous underwater vehicles in offensive and defensive confrontation tasks, obstacle avoidance of small unmanned aerial vehicles, online trajectory planning of autonomous vehicles, coordinated behavior of multiple cooperative agents, path following and obstacle avoidance of autonomous surface vehicles, multi-agent task allocation, target tracking of autonomous surface and underwater vehicles, multi-target tracking of unmanned aerial vehicle swarms, robust motion control of bipedal robots, etc.

[0005] As the tasks that robots need to handle become more and more complex, the hierarchical reinforcement learning idea has gradually been applied in the field of mobile robot autonomous behavior decision-making, such as route and scheduling decision-making of autonomous vehicles, behavior decision-making of knowledge-injected context-driven dialogue agents for disease diagnosis, determination of sub-goals during the continuous navigation of agents, replay of forgotten experiences of agents, autonomous decision-making and motion planning of intelligent vehicles, trajectory and behavior planning of autonomous vehicles, etc.

[0006] Although these reinforcement learning algorithms have achieved certain success in their respective research objects, the research on the behavior decision-making of mobile robots is still lacking, and the existing behavior decision-making methods are relatively simple. If the robot does not receive rewards during the environmental interaction process, the contribution of samples to reinforcement learning will be very small. Directly using sparse reward samples for robot learning will slow down the iterative process and even make it difficult for the reinforcement learning algorithm to converge.

[0007] Therefore, studying how to solve the negative impacts brought by sparse rewards, such as low learning rate and poor environmental learning performance, is of great importance for improving the learning speed and strategy of reinforcement learning. Summary of the Invention

[0008] An object of the present application is to provide a robot behavior decision-making method and device based on a hierarchical deep reinforcement learning model to solve the problems of low learning rate and poor environmental learning performance caused by sparse rewards in the environmental interaction process of mobile robots in the prior art.

[0009] According to one aspect of the present application, there is provided a robot behavior decision-making method based on a hierarchical deep reinforcement learning model, including:

[0010] Obtain the current state information and the target task;

[0011] Construct a hierarchical deep reinforcement learning model, the hierarchical deep reinforcement learning model includes a top layer module and a bottom layer module, the top layer module includes a decision management network and a goal-oriented management mechanism; the bottom layer module includes a dimensionality reduction network, a reward processing mechanism, and a deep Q network;

[0012] Based on the target task, input the current state information into the bottom layer module to obtain a behavior decision and execute it;

[0013] Obtain the state information at the next moment, input the current state information and the state information at the next moment into the dimensionality reduction network, and splice them with the behavior decision corresponding to the current state information to obtain the current state transition sequence, and the top layer module obtains a reward prediction value based on the current state transition sequence;

[0014] Repeat the above steps until the robot collides, obtain the score of this round and the scores of all previous rounds before completing the target task of this round, input all the round scores into the management goal-oriented mechanism to obtain a reward gain value, and continuously train the hierarchical deep reinforcement learning model with all the reward prediction values and the reward gain value to obtain the optimized hierarchical deep reinforcement learning model after learning.

[0015] Further, in the above-mentioned robot behavior decision-making method based on the hierarchical deep reinforcement learning model, the construction of the hierarchical deep reinforcement learning model includes constructing a dimensionality reduction network, the dimensionality reduction network includes three subnets, and each subnet includes a convolutional layer, an activation function, and a pooling layer;

[0016] The three subnets are sequentially connected in series to a fully connected neural network and a Tanh activation function is introduced.

[0017] Further, in the above-mentioned robot behavior decision-making method based on the hierarchical deep reinforcement learning model, the obtaining of the next moment state information, and inputting the current state information and the next moment state information into the dimensionality reduction network and then splicing with the behavior decision corresponding to the current state information to obtain the current state transition sequence includes:

[0018] Input the current state information into the dimensionality reduction network to obtain the current low-dimensional feature information;

[0019] Input the next moment state information into the dimensionality reduction network to obtain the next moment low-dimensional feature information;

[0020] Splice the behavior decision corresponding to the current state information with the current low-dimensional feature information and the next moment low-dimensional feature information and output the current state transition sequence.

[0021] Further, in the above-mentioned robot behavior decision-making method based on the hierarchical deep reinforcement learning model, repeating the above steps until the robot collides, obtaining the score of this round and the scores of all previous rounds before completing the target task of this round, inputting all the round scores into the management goal-oriented mechanism to obtain the reward gain value, and continuously training the hierarchical deep reinforcement learning model with all the reward prediction values and the reward gain value to obtain the optimized hierarchical deep reinforcement learning model, includes:

[0022] Input the current state transition sequence into the decision management network to obtain the reward prediction value;

[0023] Obtain the environmental reward signal after executing the behavior decision;

[0024] Repeat the above steps until the robot collides, obtain the score of this round and the scores of all previous rounds before completing the target task of this round, and input all the round scores into the management goal-oriented mechanism to obtain the reward gain value;

[0025] Put all the obtained state transition sequences, reward prediction values, reward gain values, and environmental reward signals into the experience pool for storage and update;

[0026] Continuously train the hierarchical deep reinforcement learning model using the data in the experience pool to obtain the optimized hierarchical deep reinforcement learning model after learning.

[0027] Further, in the above-mentioned robot behavior decision-making method based on the hierarchical deep reinforcement learning model, continuously training the hierarchical deep reinforcement learning model using the data in the experience pool to obtain the optimized hierarchical deep reinforcement learning model after learning includes:

[0028] Use all the state transition sequences, reward prediction values, and environmental reward signals in the experience pool to jointly form a loss function, and optimize the parameters of the decision management network through backpropagation;

[0029] Train and optimize the deep Q network using the reward prediction values and reward gain values obtained from the state transition sequences in the experience pool to obtain the optimized hierarchical deep reinforcement learning model after learning.

[0030] According to another aspect of the present application, there is also provided a computer-readable medium storing computer-readable instructions, which, when executed by a processor, cause the processor to implement the method as described in any one of the above.

[0031] According to another aspect of the present application, there is also provided a robot behavior decision-making device based on a hierarchical deep reinforcement learning model, the device including:

[0032] One or more processors;

[0033] A computer-readable medium for storing one or more computer-readable instructions,

[0034] When the one or more computer-readable instructions are executed by the one or more processors, the one or more processors implement the method as described in any one of the above.

[0035] Compared with the prior art, in the present application, first, the current state information and the target task are obtained; then, a hierarchical deep reinforcement learning model is constructed. The hierarchical deep reinforcement learning model includes a top-level module and a bottom-level module. The top-level module includes a decision management network and a goal-oriented management mechanism; the bottom-level module includes a dimensionality reduction network, a reward processing mechanism, and a deep Q network; then, based on the target task, the current state information is input into the bottom-level module to obtain a behavior decision and execute it. Then, the next moment state information is obtained, and the behavior decision corresponding to the current state information, the current state information, and the next moment state information are input into the dimensionality reduction network to obtain the current state transition sequence. The top-level module obtains a reward prediction value based on the current state transition sequence; finally, the above steps are repeated until the robot collides. The score of this round and the scores of all previous rounds before completing the target task of this round are obtained. All the round scores are input into the management goal-oriented mechanism to obtain a reward gain value. The hierarchical deep reinforcement learning model is continuously trained using all the reward prediction values and the reward gain value to obtain the optimized hierarchical deep reinforcement learning model. That is, by simulating the hierarchical organization mechanism of the human brain, a deep hierarchical reinforcement learning model for solving the sparse reward problem of robots is proposed. The model includes two parts: a top-level module and a bottom-level module. During the process of the robot's environmental cognition, when it is in an environment with sparse rewards, the upper module sets sub-goals for the lower module according to the interaction between the agent and the environment. At the same time, the upper module can perceive the environment and predict the state transition of the agent. In addition, predicted rewards and reward gains are added to the deep hierarchical reinforcement learning model to accelerate the learning speed of the lower module. In the lower module, a dimensionality reduction network is designed to encode and map state information. Therefore, the deep hierarchical reinforcement learning model can effectively solve the sparse reward problem of the agent. Description of the Drawings

[0036] Other features, objectives, and advantages of the present application will become more obvious by reading the detailed description of the non-limiting embodiments with reference to the following drawings:

[0037] Figure 1 A schematic flowchart of a robot behavior decision-making method based on a hierarchical deep reinforcement learning model according to an aspect of the present application is shown;

[0038] Figure 2 A schematic structural diagram of a hierarchical deep reinforcement learning model according to an aspect of the present application is shown;

[0039] Figure 3 A schematic structural diagram of a dimensionality reduction network in a hierarchical deep reinforcement learning model according to an aspect of the present application is shown;

[0040] Figure 4 Shows a cumulative score graph of an application scenario of a robot behavior decision-making method based on a hierarchical deep reinforcement learning model according to one aspect of the present application;

[0041] Figure 5 Shows a graph of the change in the training error of the DQN average network in an application scenario of a robot behavior decision-making method based on a hierarchical deep reinforcement learning model according to one aspect of the present application.

[0042] Like reference numerals in the drawings denote like or similar components. Detailed implementation manners

[0043] The present application will be further described in detail below with reference to the drawings.

[0044] In a typical configuration of the present application, the terminal, the devices of the service network, and the trusted party all include one or more processors (such as a central processing unit (CPU)), an input / output interface, a network interface, and a memory.

[0045] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM), and / or non-volatile memory in the form of, for example, read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0046] A computer-readable medium includes permanent and non-permanent, removable and non-removable media that can store information by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, Phase-Change RAM (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technologies, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disk (DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer-readable medium does not include transitory media such as modulated data signals and carrier waves.

[0047] Figure 1 A schematic flowchart of a robot behavior decision-making method based on a hierarchical deep reinforcement learning model according to an aspect of the present application is shown. This method is applicable to various motion scenarios of a mobile robot. The method includes steps S11, S12, S13, S14, and S15, which specifically include:

[0048] Step S11, obtaining the current state information and the target task;

[0049] Step S12, constructing a hierarchical deep reinforcement learning model. The hierarchical deep reinforcement learning model includes a top layer module and a bottom layer module. The top layer module includes a Decision-Managing Network (DMN) and a Goal-Directed Managing Mechanism (GDMM); the bottom layer module includes a Reduction Network (RN), a reward processing mechanism, and a Deep Q Network (DQN);

[0050] Step S13: Input the current state information into the underlying module based on the target task to obtain a behavior decision and execute it.

[0051] Step S14: Obtain the state information at the next moment, input the current state information and the state information at the next moment into the dimensionality reduction network, and splice them with the behavior decision corresponding to the current state information to obtain the current state transition sequence. The top-level module obtains a reward prediction value based on the current state transition sequence.

[0052] Step S15: Repeat the above steps until the robot collides. Obtain the score for this round and the scores for all previous rounds before completing the target task for this round. Input all the round scores into the management target-oriented mechanism to obtain a reward gain value. Continuously train the hierarchical deep reinforcement learning model using all the reward prediction values and the reward gain value to obtain the optimized hierarchical deep reinforcement learning model.

[0053] For the above steps S11 to S15, first, obtain the current state information and the target task; then, construct a hierarchical deep reinforcement learning model. The hierarchical deep reinforcement learning model includes a top-level module and an underlying module. The top-level module includes a decision management network and a target-oriented management mechanism; the underlying module includes a dimensionality reduction network, a reward processing mechanism, and a deep Q network, as Figure 2As shown; then, based on the target task, the current state information is input into the underlying module to obtain a behavior decision and execute it. Then, the state information at the next moment is obtained, and the behavior decision corresponding to the current state information, the current state information, and the state information at the next moment are input into the dimensionality reduction network to obtain the current state transition sequence. The top-level module obtains the reward prediction value based on the current state transition sequence; finally, the above steps are repeated until the robot collides, the score for this round and the scores for all previous rounds before completing the target task for this round are obtained, and all the round scores are input into the management goal-oriented mechanism to obtain the reward gain value. The hierarchical deep reinforcement learning model is continuously trained using all the reward prediction values and reward gain values to obtain the optimized hierarchical deep reinforcement learning model. That is, by simulating the hierarchical organization mechanism of the human brain, a deep hierarchical reinforcement learning model for solving the sparse reward problem of robots is proposed. This model consists of two parts: a top-level module and a bottom-level module. During the process of a robot's environmental cognition, when it is in a sparse reward environment, the upper module sets sub-goals for the lower module according to the interaction between the agent and the environment. At the same time, the upper module can perceive the environment and predict the state transition of the agent. In addition, predicted rewards and reward gains are added to this deep hierarchical reinforcement learning model to accelerate the learning speed of the lower module. In the lower module, a dimensionality reduction network is designed to encode and map state information. Therefore, this deep hierarchical reinforcement learning model can effectively solve the sparse reward problem of the agent.

[0054] For example, in an application scenario of the present invention, the Skiing game in Atari. The scenario of the Skiing game belongs to a typical sparse reward environment. This game requires the player to control a virtual character (i.e., the robot) to ski and avoid obstacles such as trees and rocks. The score is determined by the skiing time of the virtual character. The longer the skiing time, the more times the virtual character has avoided obstacles, and the higher the score for this round. First, obtain the current state information s t and the target task V;

[0055] Then, a hierarchical deep reinforcement learning model M is constructed. This invention proposes a deep hierarchical reinforcement learning model for solving the sparse reward problem of robots by simulating the hierarchical organization mechanism of the human brain. The model includes two parts: a top-level module and a bottom-level module. In the process of robot environmental cognition, when it is in an environment with sparse rewards, the upper-level module sets sub-goals for the bottom-level module according to the interaction between the agent and the environment. At the same time, the upper-level module can perceive the environment and predict the state transition of the agent. The hierarchical deep reinforcement learning model M includes a top-level module and a bottom-level module. The top-level module includes a decision management network and a goal-oriented management mechanism; the bottom-level module includes a dimensionality reduction network, a reward processing mechanism and a deep Q network. Here, a dimensionality reduction network is designed in the bottom-level module to encode and map state information so that the deep hierarchical reinforcement learning model can effectively solve the sparse reward problem of the agent. As shown in Figure 2, DMN is a model with memory function. The input is a decision management network that has been processed by a dimensionality reduction network. Current status information after processing Action taken a t and through the dimensionality reduction network Next state after processing State transition sequence The output is the reward r predicted by the top module t * , this value represents the response signal of the top-level module to the state transition of the bottom-level module, and also represents the evaluation of the top-level module on the state transition of the bottom-level module at the current moment. Another part of the top-level module is GDMM, which is inspired by the working principle of the hierarchical architecture of the human brain and is used to handle tasks such as setting sub-goals for the bottom-level modules and evaluating the execution of sub-goals of the bottom-level modules. The job of GDMM is to collect and analyze the scores sc at the end of each round. When the score sc of the bottom-level module at the end of the current round is higher than any previous record, the top-level module will issue a reward gain r to the RPM of the bottom-level module. ep buff , acting on the training process and the next round, thereby indirectly providing goal guidance to the DQN parameter update of the underlying module.

[0056] Then, based on the target task, the current state information is input into the underlying module to obtain the behavior decision a t Then, the bottom module obtains the next moment state information s t+1 , and the behavior decision a corresponding to the current state information t , the current state information s t And the next moment state information s t+1 Input the dimensionality reduction network to obtain the current state transition sequence The top-level module is based on the current state transition sequence to obtain the reward prediction value r t * ; Finally, repeat the above steps until the robot collides, obtain the score of this round and the scores of all previous rounds sc1, sc2,..., ssc before completing the target task of this round ep Input the scores of all rounds into the management target-oriented mechanism to obtain the reward gain value r ep buff , using all the reward prediction values {r t *}(t = 1, 2, 3...T) and the reward gain value r ep buff Continuously train the hierarchical deep reinforcement learning model to obtain the optimized hierarchical deep reinforcement learning model. That is, the predicted reward and reward gain are added to the deep hierarchical reinforcement learning model to accelerate the learning speed of the bottom-level module

[0057] Furthermore, in the step S12 of constructing the hierarchical deep reinforcement learning model, a dimensionality reduction network is constructed. The dimensionality reduction network includes three subnets, and each subnet includes a convolutional layer, an activation function, and a pooling layer

[0058] The three subnets are sequentially connected in series to a fully connected neural network and a Tanh activation function is introduced

[0059] For example, as Figure 3 shown, the dimensionality reduction network includes three subnets, and each subnet includes a convolutional layer, an activation function, and a pooling layer; the size of the convolutional kernel is 20×20; the ReLU activation function is selected as the activation function; the maximum pooling is selected for the pooling layer. After the three subnets are sequentially connected in series, they are connected to a fully connected network. The number of fully connected neural networks is 2, and then the Tanh activation function is used. The state information s t passes through the RN to output a one-dimensional vector φ(st) of 210 elements. All the weight parameters in the dimensionality reduction network are sampled from a normal distribution with a mean of 0 and a variance of 0.5, and each layer of the network is sampled independently. That is, a series of neural networks are used to construct the dimensionality reduction network (Reduction Network, RN) to realize the encoding mapping from high-dimensional information to low-dimensional information, so that the deep hierarchical reinforcement learning model can effectively solve the sparse reward problem of the agent

[0060] Following the above embodiment, in the step S14, obtaining the next moment state information, and after inputting the current state information and the next moment state information into the dimensionality reduction network and splicing them with the behavior decision corresponding to the current state information to obtain the current state transition sequence, includes

[0061] Input the current state information into the dimensionality reduction network to obtain the current low-dimensional feature information

[0062] Input the next moment state information into the dimensionality reduction network to obtain the next moment low-dimensional feature information

[0063] The behavior decision a corresponding to the current state information t and the current low-dimensional feature information and the next moment low-dimensional feature information are concatenated and output to obtain the current state transition sequence Implement the encoding mapping from high-dimensional information to low-dimensional information, which is beneficial to the subsequent model training process.

[0064] Following the above embodiments, in step S15, repeat the above steps until the robot collides, obtain the score of this round and the scores of all previous rounds before completing the target task of this round, input all the round scores into the management target-oriented mechanism to obtain the reward gain value, and continuously train the hierarchical deep reinforcement learning model with all the reward prediction values and reward gain values to obtain the optimized hierarchical deep reinforcement learning model, including:[[]]

[0065] Input the current state transition sequence into the decision management network to obtain the reward prediction value;

[0066] After executing the behavior decision, obtain the environmental reward signal; here, the environmental reward signal feedback by the environment is not the final system total reward R. The system total reward R is composed of the reward signal r from the environment t , the predicted reward r of the top-level module DMN t * and the reward gain r of GDMM ep buff and is jointly composed, and the system total reward is synthesized in RPM. The formula for the system total reward Rt at the current moment is shown. The final system total reward Rt participates in the parameter update of the deep Q network.

[0067]

[0068] Repeat the above steps until the robot collides, obtain the score of this round and the scores of all previous rounds before completing the target task of this round, input all the round scores into the management target-oriented mechanism to obtain the reward gain value; here, in GDMM, the initial gain is 0; the hyperparameter bufftime is set to 0.2;

[0069]

[0070] Store and update all the obtained state transition sequences, reward prediction values, reward gain values, and environmental reward signals in the experience pool;

[0071] Continuously train the hierarchical deep reinforcement learning model using the data in the experience pool to obtain the optimized hierarchical deep reinforcement learning model.

[0072] Following the above embodiment, continuously train the hierarchical deep reinforcement learning model using the data in the experience pool to obtain the optimized hierarchical deep reinforcement learning model, including:

[0073] Use all the state transition sequences, reward prediction values, and environmental reward signals in the experience pool to jointly form a loss function, and optimize the parameters of the decision-making management network through backpropagation; here, the DMN network includes an LSTM model with two layers of Recurrent Neural Network (RNN) and a fully connected neural network. The fully connected neural network is connected to a ReLU activation function at the back. During training, choose to use the root mean square to calculate the loss function; choose the Adam algorithm as the parameter optimizer, and set the learning rate to 0.01.

[0074] s t+1 =env(s t ,a t )

[0075]

[0076]

[0077] θ1=θ1+optimizer(L(θ1))

[0078] For the human brain with a hierarchical architecture, it is more about first formulating clear goals, and then gradually achieving the goals and obtaining the best steps by continuously trial and error to accumulate experience. The goal-oriented trial and error process, compared with the non-goal-oriented trial and error process, has more subjective initiative in exploring the environment by the model during the task while ensuring that the randomness remains unchanged.

[0079] Train and optimize the deep Q network using the reward prediction values and reward gain values obtained from the state transition sequences in the experience pool to obtain the optimized hierarchical deep reinforcement learning model.

[0080] Here, the deep Q-network of the underlying module, similar to other deep reinforcement learning methods, evaluates the value of taking actions in the current state. The deep Q-network of this model also has two features: an experience replay process and a target network. The network is divided into two subnets. One is the evaluation network (Q Evaluate Network, QEN), and the other is the target network (QTarget Network, QTN). The two networks have the same structure, and the model parameters of the evaluation network are denoted as θ2. In this model, the action output a t is represented by the following formula, and the ε-greedy algorithm is used to solve the "exploration-exploitation" balance problem.

[0081]

[0082] where: probability—the generated random number;

[0083] ε—the parameter in the ε-greedy algorithm, with a specific range between 0.7 and 0.9. s t represents the state of the agent at time t, and π represents the action policy used by the agent. q eval represents the evaluated Q value.

[0084] The loss function during the training of the deep Q-network is expressed as follows, where LossFunc represents the type of the loss function, and loss represents the error value generated during training.

[0085]

[0086] loss = LossFunc(q target - q eval )

[0087] where q target represents the target Q value, R represents the reward value, and γ is the discount factor.

[0088] In the deep Q-network, QEN and QTN use exactly the same fully connected neural network structure, and the number of neurons in the middle layer is set to 40. The weights of each layer are independently initialized using a normal distribution with a standard deviation of 0.1. ε used to handle the "exploration-exploitation" balance problem is set to 0.7. The size of the experience pool is set to 2000; 32 pieces of data are randomly selected from the experience pool for training during training. The parameters of QTN are updated from QEN every 100 times. The discount factor γ is set to 0.9. During training, the root mean square is selected to calculate the loss function, and the Adam algorithm is selected as the parameter optimizer, with the learning rate set to 0.01.

[0089] In addition, the error value loss generated during training has the following relationship.

[0090]

[0091] where N batch represents the number of state transition sequences in a batch extracted during one training, and i represents the i-th state transition sequence in this batch.

[0092] It is not difficult to see from the above formula that when the environment is in a sparse environment and the agent does not reach the goal (worst case) set by GDMM in a round, due to r t * still exists, so there is still the "ability" to train the network. It can also be seen from the inequality that by using the two augmented reward signals obtained from the state transition sequence and the network maximum value one-step update strategy, the network training error value can be appropriately increased, and the learning ability of the agent in the sparse environment can be improved. RMP receives the environmental feedback reward signal r t , the predicted reward r t * and the reward gain r ep buff , outputs the total system reward Rt, and sends the state transition sequence and the reward r t to the experience pool for storage and update of the original content. After the end of this round, the sampled information in the experience pool is sent to the deep Q network and DMN for parameter training. After the training ends, this round ends, and GDMM collects the score sc of this round and calculates the reward gain for the next round. When the number of rounds reaches the threshold, this round of tasks ends.

[0093] In an embodiment of the present invention, the scenario of the Skiing game belongs to a typical sparse reward environment. This game requires the player to control a virtual character (i.e., the agent) to ski and avoid obstacles such as trees and rocks. The score is determined by the skiing time of the virtual character. The longer the skiing time, the more times the virtual character avoids obstacles, and the higher the score of this round. Frames are extracted from the game at equal time intervals, brought into the calculation model, and actions are output to control the virtual character. Therefore, the skiing time can be converted into the number of times of controlling the skiing action of the virtual character. The longer the skiing time and the more times the action is executed without collision, the higher the score of the game. In a multi-round, when the virtual character controlled by the player collides, the game terminates, this round ends, and the virtual character keeps the model parameters and returns to the starting point and starts a new round.

[0094] In this environment, the number of elements in the action set is 3, that is, the virtual character has three actions: left, right, and forward. The state space is this game scene. The state st represents the game frame at time t, and each frame is an RGB image with a width of 160 pixels and a height of 210 pixels. When the game frame is input into the computing model, it is a 210*160*3 matrix.

[0095] The parameter settings are as follows:

[0096] In the dimensionality reduction network, a convolutional layer, an activation function, and a pooling layer form a subnet. The size of the convolutional kernel is 20*20; the ReLU activation function is selected; the max pooling is selected for the pooling layer. After three subnets are connected in series, they are connected to 2 fully connected networks. The number of fully connected neural networks is 2. The 2 fully connected neural networks are connected in series, and the Tanh activation function is used after each fully connected neural network. The state information st passes through the dimensionality reduction network RN and outputs a one-dimensional vector φ(st) with 210 elements. All the weight parameters in the dimensionality reduction network are sampled from a normal distribution with a mean of 0 and a variance of 0.5. Each layer of the network is sampled independently.

[0097] The DMN network contains an LSTM with two layers of recurrent neural networks (RNN) and one layer of fully connected neural network. The ReLU activation function is used after the fully connected neural network. During training, the root mean square is used to calculate the loss function; the Adam algorithm is selected as the parameter optimizer, and the learning rate is set to 0.01.

[0098] In GDMM, the initial gain is 0; the hyperparameter bufftime is set to 0.2;

[0099] In the deep Q network, QEN and QTN use exactly the same fully connected neural network structure, and the number of neurons in the middle layer is set to 40. The weights of each layer are independently initialized from a normal distribution with a standard deviation of 0.1. ε used to handle the "exploration-exploitation" balance problem is set to 0.7. The size of the experience pool is set to 2000; 32 pieces of data are randomly selected from the experience pool for training during training. The QTN parameters are updated from QEN every 100 times. The discount factor γ is set to 0.9. During training, the root mean square is used to calculate the loss function, and the Adam algorithm is selected as the parameter optimizer, and the learning rate is set to 0.01.

[0100] Each experiment takes 50 rounds as one cycle and is executed for 10 cycles in total. When the model runs, the CPU used is Intel COREi5, and the GPU used is NVIDIA GeForce RTX 2080 Ti. It is implemented using the Python language.

[0101] Compare the model of this application with the original DQN, Double DQN (DDQN), Actor-Critic (AC), and NoisyNet (NN) algorithms. Run 10 rounds in the Skiing environment, with 50 episodes per round, and also record the number of actions executed N ep r and the training error value e of the DQN network ep r . When training the Actor with the AC algorithm, the root mean square is selected to calculate the loss function; the Adam algorithm is selected as the parameter optimizer, and the learning rate is set to 0.01. NoisyNet adopts the method of adding a Gaussian noise network to each layer of the network

[0102] Figure 4 Shows the cumulative score graphs of four algorithms, Actor-Critic (AC), Double DQN (DDQN), DQN, and NoisyNet (NN), in a 10-round game with 50 episodes per round. The changing trends of SNep for the four algorithms are represented by SNep-AC, SNep-DDQN, SNep-DQN, and SNep-NN respectively in Figure 4 . The shaded area represents the distribution of the cumulative scores for each episode. The upper boundary of the distribution is the changing trend of MaxNep for each model, and the lower boundary is the changing trend of MinNep. To enhance the comparability, we also plot the changing trend of SNep for the validation experiment of the model proposed in this application in the figure. From Figure 4 it can be seen that during the task execution, SNep, MaxNep, and MinNep of the four algorithms are all gradually increasing, and the growth trends of the four algorithms are similar. The MaxNep and MinNep curves are relatively close, that is, the numerical differences are not significant. However, compared with the algorithm proposed in this application, the growth rates of these four algorithms are all slower. That is, under the condition that the cumulative score exceeds 10 to the power of 5, the number of episodes required for the four algorithms is more than that of the algorithm proposed in this article

[0103] Figure 5 Represents the comparison graph of the training error of the DQN network, the underlying module of the four algorithms and the algorithm proposed in this article during task execution. The black dotted line in the figure represents the changing trend graph of Se of the algorithm proposed in this application ep and its values are all consistent with the validation experiment; while the other lines respectively represent the changing trend graphs of Se of the four algorithms ep (represented by Seep-AC, Seep-DDQN, Seep-DQN, and Seep-NN respectively).

[0104] From Figure 5It can be seen that Seep-NN and Seep-DQN remain relatively stable within 50 rounds. However, there are several times when the network error training values are relatively large (specifically, NoisyNet: 8 times and DQN: 6 times), indicating that for these two algorithms, the network error does not fluctuate and decrease within 50 consecutive rounds. For the Double DQN algorithm, its Seep-DDQN shows a gradually increasing trend starting from the 41st round, indicating that under such tasks, this algorithm is difficult to meet the performance requirements. For the AC algorithm, its Seep-AC shows an oscillating downward state starting from the 30th round, indicating that this algorithm has the ability to decrease the error during the task but requires more rounds. Compared with the algorithm proposed in this invention represented by the black dashed line, the network training capabilities of these four algorithms are all lower than that of the algorithm proposed in this invention.

[0105] For the DQN model, in a sparse environment, the number of positive rewards is small and the distribution is sparse. It is difficult for the agent to obtain positive rewards, resulting in slow updates of the state value and action value. Therefore, it is difficult for the agent to learn. Even though DDQN changes the update strategy to reduce the overestimation of action values by DQN, its performance is still poor due to the characteristics of the sparse environment. NoisyNet mainly increases the degree of environmental exploration by adding a noise network to the deep Q-network. However, the number of positive rewards in the sparse environment is itself small, and the Skiing game has the characteristic that the state is not unique when the task terminates. Therefore, the part of the state transition sequence that helps the agent learn does not change significantly after adding NoisyNet, so the impact on the agent's learning is not great. Actor-Critic has the best performance among the four compared models, but the experimental results of this model are still lower than the model proposed in this article. This is because in the Actor-Critic model, Actor learns with reference to Critic, and Critic learns with reference to the cumulative expected reward of the agent in the environment. The sparse environment affects the cumulative expected reward, thus affecting Critic, and further affecting the learning of Actor that takes it as a reference. Therefore, the characteristics of the sparse environment result in poor performance of both Actor and Critic.

[0106] The above data and analysis all show that in a sparse reward environment such as the Skiing game, the execution performance of the algorithm proposed in this patent is better than the other four algorithms.

[0107] According to another aspect of the present application, there is also provided a computer-readable medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the processor is enabled to implement the method for controlling user confrontation as described above.

[0108] According to another aspect of the present application, there is also provided a robot behavior decision-making device, characterized in that the device includes:

[0109] One or more processors;

[0110] A computer-readable medium for storing one or more computer-readable instructions,

[0111] When the one or more computer-readable instructions are executed by the one or more processors, the one or more processors implement the method for controlling user confrontation on the device as described above.

[0112] Herein, for the detailed content of each embodiment of the device, reference may specifically be made to the corresponding part of the embodiment of the method for controlling user confrontation on the device end as described above, and details are not repeated herein.

[0113] In summary, by simulating the hierarchical organization mechanism of the human brain, a deep hierarchical reinforcement learning model for solving the sparse reward problem of robots is proposed. The model includes two parts: a top-level module and a bottom-level module. During the process of robot environmental cognition, when it is in a sparse reward environment, the upper module sets sub-goals for the lower module according to the interaction between the agent and the environment. At the same time, the upper module can perceive the environment and predict the state transition of the agent. In addition, predicted rewards and reward gains are added to the deep hierarchical reinforcement learning model to accelerate the learning speed of the lower module. In the lower module, a dimensionality reduction network is designed to encode and map state information. Therefore, the deep hierarchical reinforcement learning model can effectively solve the sparse reward problem of the agent.

[0114] It should be noted that the present application can be implemented in software and / or a combination of software and hardware. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of the present application can be executed by a processor to implement the steps or functions described above. Similarly, the software program (including related data structures) of the present application can be stored in a computer-readable recording medium, such as a RAM memory, a magnetic or optical drive, or a floppy disk and the like. In addition, some steps or functions of the present application can be implemented using hardware, for example, as a circuit that cooperates with the processor to execute each step or function.

[0115] In addition, a part of this application can be applied as a computer program product, such as computer program instructions. When executed by a computer, through the operation of this computer, it can call or provide the methods and / or technical solutions according to this application. The program instructions for calling the methods of this application may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium, and / or stored in the working memory of a computer device running according to the program instructions. Here, an embodiment according to this application includes a device, which includes a memory for storing computer program instructions and a processor for executing the program instructions. When the computer program instructions are executed by the processor, it triggers the device to run based on the methods and / or technical solutions of the foregoing multiple embodiments according to this application.

[0116] For those skilled in the art, it is obvious that this application is not limited to the details of the above exemplary embodiments, and without departing from the spirit or basic characteristics of this application, this application can be implemented in other specific forms. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of this application is defined by the appended claims rather than the above description. Therefore, it is intended to cover all changes falling within the meaning and scope of the equivalent elements of the claims in this application. Any reference signs in the claims should not be regarded as limiting the claimed rights. In addition, obviously, the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices stated in the device claims can also be implemented by one unit or device through software or hardware. First, second, etc. are used to denote names and do not denote any particular order.

Claims

1. A robot behavior decision-making method based on a hierarchical deep reinforcement learning model, characterized in that, The method includes: Obtaining the current state information and the target task; Constructing a hierarchical deep reinforcement learning model, which includes a top-level module and a bottom-level module. The top-level module includes a decision management network and a goal-oriented management mechanism; the bottom-level module includes a dimensionality reduction network, a reward processing mechanism, and a deep Q network; Based on the target task, input the current state information into the bottom-level module to obtain a behavior decision and execute it; Obtain the state information at the next moment, input the current state information and the state information at the next moment into the dimensionality reduction network, and then splice them with the behavior decision corresponding to the current state information to obtain the current state transition sequence. The top-level module obtains a reward prediction value based on the current state transition sequence; Repeat the above steps until the robot collides. Obtain the score of this round and the scores of all previous rounds before completing the target task of this round. Input all the round scores into the management goal-oriented mechanism to obtain a reward gain value. Continuously train the hierarchical deep reinforcement learning model with all the reward prediction values and the reward gain value to obtain the optimized hierarchical deep reinforcement learning model, including: Input the current state transition sequence into the decision management network to obtain a reward prediction value; Obtain an environmental reward signal after executing the behavior decision; Repeat the above steps until the robot collides. Obtain the score of this round and the scores of all previous rounds before completing the target task of this round. Input all the round scores into the management goal-oriented mechanism to obtain a reward gain value; Put all the obtained state transition sequences, reward prediction values, reward gain values, and environmental reward signals into an experience pool for storage and update; Continuously train the hierarchical deep reinforcement learning model with the data in the experience pool to obtain the optimized hierarchical deep reinforcement learning model, including: Use all the state transition sequences, reward prediction values, and environmental reward signals in the experience pool to jointly form a loss function, and optimize the parameters of the decision management network through backpropagation; Train and optimize the deep Q network from the reward prediction values and reward gain values obtained from the state transition sequences in the experience pool to obtain the optimized hierarchical deep reinforcement learning model.

2. The method according to claim 1, characterized in that, The construction of the hierarchical deep reinforcement learning model includes the construction of a dimensionality reduction network. The dimensionality reduction network includes three subnets, and each subnet includes a convolutional layer, an activation function, and a pooling layer; The three subnets are sequentially connected in series to a fully connected neural network and a Tanh activation function is introduced.

3. The method according to claim 2, characterized in that, The obtaining of the state information at the next moment, inputting the current state information and the state information at the next moment into the dimensionality reduction network, and then splicing them with the behavior decision corresponding to the current state information to obtain the current state transition sequence includes: Input the current state information into the dimensionality reduction network to obtain the current low-dimensional feature information; Input the state information at the next moment into the dimensionality reduction network to obtain the low-dimensional feature information at the next moment; Splice the behavior decision corresponding to the current state information with the current low-dimensional feature information and the low-dimensional feature information at the next moment to output the current state transition sequence.

4. A computer-readable medium having computer-readable instructions stored thereon, which, when executed by a processor, cause the processor to implement the method according to any one of claims 1 to 3.

5. A robot behavior decision-making device based on a hierarchical deep reinforcement learning model, characterized in that, The apparatus comprises: one or more processors; a computer-readable medium for storing one or more computer-readable instructions, which, when executed by the one or more processors, cause the one or more processors to implement the method according to any one of claims 1 to 3.

Images