Offline reinforcement learning method and control method for robot motion decision-making

Through the integrated random network distillation method and the dual-delay deterministic depth strategy gradient algorithm, the problems of high cost and difficult to control the constraints in offline reinforcement learning of robots are solved, and the training time and computing power consumption are reduced, which significantly improves training efficiency .

CN118504652BActive Publication Date: 2024-11-19UNIV OF ELECTRONICS SCI & TECH OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410519825.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-28
Publication Date
2024-11-19
Estimated Expiration
2044-04-28

AI Technical Summary

Technical Problem

The existing offline reinforcement learning methods of robots are costly, difficult to control, and are sensitive to parameters, resulting in long training time and high computing power consumption.

Method used

The integrated random network distillation method and the dual-delay deterministic depth strategy gradient algorithm are adopted to decouple the integrated part and the reinforcement learning training part through counter-exploration reward modeling and timing differential target calculation to reduce the training time and computing power requirements.

Benefits of technology

It significantly reduces the training time of offline reinforcement learning algorithms, reduces the requirements for computing power equipment, and avoids the problems of excessive or weak constraints caused by hyperparameter setting.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118504652B_ABST
    Figure CN118504652B_ABST
Patent Text Reader

Abstract

The present invention discloses an offline reinforcement learning method and control method for robot motion decision-making. The present invention decouples the integrated part from the reinforcement learning training part and puts it into the anti-exploration reward modeling part, thereby greatly reducing the time cost of offline reinforcement learning algorithm training and greatly reducing the requirements for computing power equipment. The present invention predicts the state-action pairs input in training and calculates the corresponding anti-exploration rewards through an integrated random network distillation method. By utilizing the generalization of the neural network itself to constrain the exploration of the intelligent agent, there is no need to control the strength of the divergence constraint, avoiding the problem of too strong or too weak constraints caused by improper hyperparameter settings.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of robot control, and in particular to an off-line reinforcement learning method and a control method for robot motion decision-making. Background Art

[0002] Reinforcement learning is a branch of machine learning that has shown excellent performance in intelligent agent control and dynamic decision-making tasks in recent years. Reinforcement learning does not learn knowledge directly from data, but takes actions based on feedback from the environment. The process is a continuous interactive process, the purpose of which is to obtain the maximum cumulative reward from the beginning of interaction with the environment to the end of interaction. Deep reinforcement learning combines traditional reinforcement learning methods based on Markov decision processes with deep learning technology, which greatly improves the ability of reinforcement learning to solve complex problems. It has now been implemented in many real-world applications such as games and robot control and has achieved remarkable results. However, there are always certain obstacles to the implementation of reinforcement learning. For example, traditional reinforcement learning methods usually require the intelligent agent to interact with the simulation environment continuously and learn the optimal strategy through trial and error. However, this real-time interactive method may have many challenges in real-world tasks, such as high data collection costs, low simulation efficiency, safety risks, and the difference in the fields of simulation and real applications because the simulation cannot be completely equivalent to the real situation. These problems have caused certain challenges to the implementation of online reinforcement learning.

[0003] Offline reinforcement learning is a popular sub-research area of ​​reinforcement learning. It requires the agent to be trained from a static, deterministic data set, and no additional exploration or simulation is allowed during the entire strategy training phase. Because the offline form is more efficient and safe, it solves the problems of low sampling rate and uncontrollable exploration in traditional reinforcement learning. The agent fully understands the Markov decision process through offline data sets and constructs a strategy to obtain the most cumulative rewards in actual interactions. Previous mainstream offline reinforcement learning methods can be divided into two categories. One is a divergence constraint-based method that prevents the distribution of learning strategies and behavioral strategies from deviating too far by adding explicit or implicit divergence constraints to offline learning strategies. The other is an integrated offline reinforcement learning method that utilizes the generalization characteristics of neural networks and uses an integrated Q-value network to fit the uncertainty of state actions by predicting the Q-value of state-action pairs in offline datasets. This eliminates the need to add constraints to strategies and directly penalizes strategy learning through estimated uncertainty. However, if this type of method wants to produce ideal results on difficult tasks, it may require a network integration size of dozens or hundreds, which consumes a huge amount of computing power and video memory, resulting in a surge in experimental costs. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the offline reinforcement learning method and control method for robot motion decision-making provided by the present invention solve the problems of excessively high cost of existing robot offline reinforcement learning, difficulty in controlling constraint strength, and sensitivity to parameters.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0006] An offline reinforcement learning method for robot motion decision-making is provided, which comprises the following steps:

[0007] S1. Obtain an offline data set of robot motion decision and preprocess it to obtain a preprocessed offline data set; each offline data includes the robot's input state, output action, state at the next moment, and current reward;

[0008] S2. Build a target network, fix the parameters of the target network, and build k test networks with the same structure as the target network. Based on the preprocessed offline data set, train the test network through the integrated random network distillation method to obtain the trained prediction network.

[0009] S3. Based on the double-delayed deterministic deep policy gradient algorithm, a policy network, a first value network, a second value network, a target policy network consistent with the policy network structure, a first target value network consistent with the first value network structure, and a second target value network consistent with the second value network structure are constructed;

[0010] S4, select the next moment state from the offline data set and input it into the target policy network to obtain the next moment action output by the target policy network;

[0011] S5. The next-moment action and the corresponding next-moment state output by the target policy network are used as the input of the trained test network, and the mean square error values ​​of the outputs of all currently trained test networks and the output of the target network are added to obtain the first anti-exploration reward.

[0012] S6. Input the next-moment action and the corresponding next-moment state output by the target strategy network into the two target value networks; select the smallest output of the two target value networks and calculate it with the first anti-exploration reward to obtain the time difference target of the value function;

[0013] S7, inputting the input state and output action corresponding to the next moment state selected in step S4 into two value networks, respectively obtaining the first outputs of the two value networks; calculating the update gradient of the value network parameters according to the time difference target and the first output of the value network, and updating the value network;

[0014] S8, determine whether the current value network update times reaches the update times threshold, if so, update the strategy network parameters and go to step S9; otherwise, return to step S4;

[0015] S9, determine whether the training is ended, if so, end the offline reinforcement learning of the robot motion strategy; otherwise, update the parameters of the target value network and the target strategy network, and return to step S4.

[0016] Furthermore, the specific method of obtaining the trained prediction network in step S2 includes the following sub-steps:

[0017] S2-1, build the target network, fix the parameters of the target network, and build k test networks with the same structure as the target network;

[0018] S2-2, select batch data containing only input states and output actions from the preprocessed offline data set, input the selected batch data into the target network and the test network respectively, and obtain the output of the target network and the output of each test network accordingly;

[0019] S2-3. Obtain the mean square error loss according to the output of the test network and the output of the target network, and update the parameters of the test network by the mean square error loss until the output of the prediction network converges to obtain the pre-trained test network.

[0020] Furthermore, the calculation expression of the first anti-exploration reward in step S5 is:

[0021]

[0022] in represents the first anti-exploration reward; k is the number of test networks; is the output of the i-th trained test network; is the output of the target network; represents the L2 norm.

[0023] Furthermore, the calculation expression of the timing difference target in step S6 is:

[0024]

[0025] in represents the temporal difference target; Indicates the current reward; and are all constants; Reward for the first anti-exploration; Represents the outputs of the two target value networks.

[0026] Furthermore, the updated gradient calculation expression of the value network parameters in step S7 is:

[0027]

[0028] in It means to find the gradient; represents the parameters of the j-th value network; represents the training data used in the current training batch, Indicates the number of training data used in the current training batch; represents the loss function used to find the gradient of the j-th value network parameter; Indicates that the input is When is the first output of the jth value network; represents the temporal difference target; represents the input state corresponding to the next moment state selected in step S4, Indicates the next state selected in step S4 The corresponding output action, r represents the current reward.

[0029] Furthermore, the specific method for updating the policy network parameters in step S8 includes the following sub-steps:

[0030] S8-1, input the input state into the policy network, and use the input state and the output corresponding to the policy network as the input of the trained test network, and add the mean square error value of the output of the currently trained test network to obtain the second anti-exploration reward;

[0031] S8-2. Input the input state and the output corresponding to the policy network into the two value networks, and obtain the second outputs of the two value networks respectively; select the smallest second output of the two value networks, and calculate it with the second anti-exploration reward to obtain the policy network parameter update gradient, and update the policy network parameters.

[0032] Furthermore, the calculation method of the second anti-exploration reward in step S8-1 is the same as the calculation method of the first anti-exploration reward.

[0033] Furthermore, the calculation expression of the policy network parameter update gradient in step S8-2 is:

[0034]

[0035] in It means to find the gradient; represents the training data used in the current training batch, Indicates the number of training data used in the current training batch; Indicates that the input is is the second output of the jth value network; is a constant; Denotes the second anti-exploration reward.

[0036] Furthermore, in step S9, the update expressions of the parameters of the target value network and the target strategy network are respectively:

[0037]

[0038] in Represents the updated parameters of the j-th target value network; Represents the parameters of the current j-th value network; is a constant greater than 0 and less than 1; The parameters before the j-th target value network update; Represents the updated parameters of the target policy network; Represents the current policy network parameters; Indicates the parameters of the target policy network before updating.

[0039] A robot control method based on an offline reinforcement learning method for robot motion decision-making is provided, which generates a robot control method through a policy network obtained through training to complete robot control.

[0040] The beneficial effects of the present invention are:

[0041] 1. The present invention decouples the integration part from the reinforcement learning training part and puts it into the anti-exploration reward modeling part, thereby greatly reducing the time cost of offline reinforcement learning algorithm training and greatly reducing the requirements for computing power equipment.

[0042] 2. The present invention predicts the state-action pairs input during training and calculates the corresponding anti-exploration rewards through an integrated random network distillation method. It constrains the exploration of intelligent agents by utilizing the generalization of the neural network itself, without controlling the strength of the divergence constraint, thus avoiding the problem of over-strong or over-weak constraints caused by improper hyperparameter settings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a schematic diagram of the process of this method;

[0044] Figure 2 Schematic diagram of the structure of the integrated random network distillation method. DETAILED DESCRIPTION

[0045] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0046] like Figure 1 As shown, the offline reinforcement learning method for robot motion decision-making includes the following steps:

[0047] S1. Obtain an offline data set of robot motion decision and preprocess it to obtain a preprocessed offline data set; each offline data includes the robot's input state, output action, state at the next moment, and current reward;

[0048] S2. Build a target network, fix the parameters of the target network, and build k test networks with the same structure as the target network. Based on the preprocessed offline data set, train the test network through the integrated random network distillation method to obtain the trained prediction network.

[0049] S3. Based on the double-delayed deterministic deep policy gradient algorithm, a policy network, a first value network, a second value network, a target policy network consistent with the policy network structure, a first target value network consistent with the first value network structure, and a second target value network consistent with the second value network structure are constructed;

[0050] S4, select the next moment state from the offline data set and input it into the target policy network to obtain the next moment action output by the target policy network;

[0051] S5. The next-moment action and the corresponding next-moment state output by the target policy network are used as the input of the trained test network, and the mean square error values ​​of the outputs of all currently trained test networks and the output of the target network are added to obtain the first anti-exploration reward.

[0052] S6. Input the next-moment action and the corresponding next-moment state output by the target strategy network into the two target value networks; select the smallest output of the two target value networks and calculate it with the first anti-exploration reward to obtain the time difference target of the value function;

[0053] S7, inputting the input state and output action corresponding to the next moment state selected in step S4 into two value networks, respectively obtaining the first outputs of the two value networks; calculating the update gradient of the value network parameters according to the time difference target and the first output of the value network, and updating the value network;

[0054] S8, determine whether the current value network update times reaches the update times threshold, if so, update the strategy network parameters and go to step S9; otherwise, return to step S4;

[0055] S9, determine whether the training is ended, if so, end the offline reinforcement learning of the robot motion strategy; otherwise, update the parameters of the target value network and the target strategy network, and return to step S4.

[0056] like Figure 2 As shown, the specific method of obtaining the trained prediction network in step S2 includes the following sub-steps:

[0057] S2-1, build the target network, fix the parameters of the target network, and build k test networks with the same structure as the target network;

[0058] S2-2, select batch data containing only input states and output actions from the preprocessed offline data set, input the selected batch data into the target network and the test network respectively, and obtain the output of the target network and the output of each test network accordingly;

[0059] S2-3. Obtain the mean square error loss according to the output of the test network and the output of the target network, and update the parameters of the test network by the mean square error loss until the output of the prediction network converges to obtain the pre-trained test network.

[0060] In the specific implementation process, the input of the target network and the test network involved in the integrated random network distillation method is a state-action pair, and the output is a tensor of fixed dimension. The dimension size of this tensor is a training hyperparameter that can be changed. This tensor has no explicit physical meaning, which is the result of the network encoding the input. The parameters of the target network are fixed and not updated after random initialization, so the target network is similar to a random network, and its output has no specific meaning and is only used as a distillation target. This is also the origin of the name of the random network distillation algorithm. Therefore, the target network and the test network have no necessary structural connection with the subsequent network. The specific structure of the target network and the test network can be set at will while meeting the above requirements.

[0061] The calculation expression of the first anti-exploration reward in step S5 is:

[0062]

[0063] in represents the first anti-exploration reward; k is the number of test networks; is the output of the i-th trained test network; is the output of the target network; represents the L2 norm.

[0064] The calculation expression of the timing difference target in step S6 is:

[0065]

[0066] in represents the temporal difference target; Indicates the current reward; and are all constants; Reward for the first anti-exploration; Represents the outputs of the two target value networks.

[0067] The updated gradient calculation expression of the value network parameters in step S7 is:

[0068]

[0069] in It means to find the gradient; represents the parameters of the j-th value network; represents the training data used in the current training batch, Indicates the number of training data used in the current training batch; represents the loss function used to find the gradient of the j-th value network parameter; Indicates that the input is When is the first output of the jth value network; represents the temporal difference target; represents the input state corresponding to the next moment state selected in step S4, Indicates the next state selected in step S4 The corresponding output action, r represents the current reward.

[0070] The specific method for updating the policy network parameters in step S8 includes the following sub-steps:

[0071] S8-1, input the input state into the policy network, and use the input state and the output corresponding to the policy network as the input of the trained test network, and add the mean square error value of the output of the currently trained test network to obtain the second anti-exploration reward;

[0072] S8-2. Input the input state and the output corresponding to the policy network into the two value networks, and obtain the second outputs of the two value networks respectively; select the smallest second output of the two value networks, and calculate it with the second anti-exploration reward to obtain the policy network parameter update gradient, and update the policy network parameters.

[0073] The calculation method of the second anti-exploration reward in step S8-1 is the same as the calculation method of the first anti-exploration reward.

[0074] The calculation expression of the policy network parameter update gradient in step S8-2 is:

[0075]

[0076] in It means to find the gradient; represents the training data used in the current training batch, Indicates the number of training data used in the current training batch; Indicates that the input is is the second output of the jth value network; is a constant; Denotes the second anti-exploration reward.

[0077] The update expressions of the parameters of the target value network and the target strategy network in step S9 are:

[0078]

[0079] in Represents the updated parameters of the j-th target value network; Represents the parameters of the current j-th value network; is a constant greater than 0 and less than 1; The parameters before the j-th target value network update; Represents the updated parameters of the target policy network; Represents the current policy network parameters; Indicates the parameters of the target policy network before updating.

[0080] The robot control method generates a robot control method through a trained strategy network to complete the robot control.

[0081] In the specific implementation process, the specific robot motion decision simulation environment can be determined first, and a certain strategy can be used in the simulation to collect interaction data in the format of (state, action, reward, next state). This strategy can be a strategy that has been interactively trained online in the simulation environment, a strategy with rules written by humans, or a random action strategy. In general, there is no restriction on the decision level of the strategy. This paper uses a public data set. For the state data in the data set, because its large numerical range is not conducive to the training stability of the neural network, this paper normalizes all states in the data. The specific normalization method is to subtract the mean of each dimension of the state from the value of the dimension and then divide it by the standard deviation, thereby completing the preprocessing of the data.

[0082] In one embodiment of the present invention, the Halfcheetah task requires an agent to control a robot with multiple joints and body parts. The goal is to let the agent control the quadruped robot to coordinate the joints and learn to run forward quickly by applying torque on the joints. The task assigns positive rewards according to the distance moved forward, and negative rewards are fed back if it moves backward. In this task, the robot's state information is generally represented by a 17-dimensional real number tensor. The real value of each dimension represents a certain physical meaning, including the coordinates of the top of the robot, the angle and angular velocity of each joint, the speed component of the robot on each coordinate axis, and other physical information to model the robot's state information; at the same time, the agent needs to output a real number tensor with a dimension of 6 in each interaction with the environment. The real value of each dimension represents the torque applied to the 6 controllable joints of the quadruped robot, that is, to control the robot's maneuverability and trajectory.

[0083] In order to verify the effect of the present invention in robot motion control, we selected the D4RL dataset, which is very common in offline reinforcement learning. The offline dataset was developed, collected, and evaluated by researchers at Stanford University. It is characterized by providing a high-quality, diverse, and standardized reinforcement learning dataset, covering a variety of tasks and environments that can be used to evaluate and compare different offline reinforcement learning algorithms. It is a very mainstream offline simulation dataset related to robot motion in academia. We mainly use the MuJoCo, a multi-legged robot control simulation dataset, to evaluate the decision-making ability of the present invention in robot motion control.

[0084] The D4RL dataset uses a standardized score to measure the ability of the agent in the task. This score scales the cumulative reward of the agent's performance in each task of the offline dataset to a score between 0 and 100, where 0 represents the average return obtained by the agent using random sampling in the action space (the value of more than 100 episodes is counted). And 100 corresponds to the average return of the expert agent in the task. In the experiments of this article, the following MuJoCo environment is used:

[0085] 1. HalfCheetah: This task is a robot with multiple joints and body parts. The goal is to let the agent control the quadruped robot to coordinate the joints and apply torque on the joints so that the robot can learn to run fast. This task assigns positive rewards based on the distance moved forward, and negative rewards are fed back if it moves backward. The state tensor dimension size in this task is 17, and the action vector dimension size is 6.

[0086] 2. Hopper: This task is a robot with a single spring leg and four main torso parts. The goal is to apply torque to the joints connecting the four parts, and the agent needs to learn to control the robot to jump to move forward as quickly as possible. The state tensor dimension size in this task is 11, and the action vector dimension size is 3.

[0087] 3. Walker2d: This task is based on the Hopper design. The agent needs to control a bipedal robot, which consists of 7 main body parts. The goal is to apply torque to the joints connecting the 7 body parts to control the robot to walk forward stably. The state tensor dimension size in this task is 17, and the action vector dimension size is 6.

[0088] We selected some of the more mainstream offline reinforcement learning methods for comparison on the MuJoCo task. The comparison results are shown in Table 1.

[0089] Table 1: Comparison of standardized scores of mainstream offline reinforcement learning methods on the Mujoco environment

[0090]

[0091] The methods in the first part of Table 1, such as BC, CQL, IQL, etc., are all classic offline reinforcement learning methods based on divergence constraints, while methods such as SAC-N and EDAC are classic offline reinforcement learning methods based on integration. And the description following the task represents the quality of the dataset, for example, halfcheetah-medium is a medium-quality dataset, while halfcheetah-medium-expert is a dataset mixed with expert data, with higher quality. It can be seen that the standardized scores of the present invention (ERND-TD3) on the three robot control tasks are significantly higher than the offline reinforcement learning methods based on divergence constraints; the performance effect is basically consistent with the offline reinforcement learning method based on integration.

[0092] In order to compare the integration efficiency, this implementation also performed the SAC-N algorithm and counted the running time and GPU memory usage. The hopper-medium offline dataset was selected for the task of this experiment, and the Nvidia RTX 2080s graphics card was used. The training time of one cycle (1000 gradient descent time steps) and the average memory usage of the SAC method, IQL method, SAC-500 and ERND-TD3 (integration size 3) algorithms were counted. The comparison results are shown in Table 2.

[0093] Table 2: Comparison of the efficiency of the algorithm based on the present invention and the traditional ensemble method

[0094]

[0095] As can be seen from Table 2, the present invention (ERND-TD3) has a very obvious improvement in both running time and GPU cost compared with the SAC-N method, while the effect is basically the same, which is enough to illustrate the effectiveness of this method in robot motion decision-making.

Claims

1. An offline reinforcement learning method for robot motion decision making, characterized in that: The following steps are involved: S1. Obtain an offline data set of robot motion decision and preprocess it to obtain a preprocessed offline data set; each offline data includes the robot's input state, output action, state at the next moment, and current reward; S2, build the target network, fix the parameters of the target network, and build k test networks with the same structure as the target network; Based on the preprocessed offline dataset, the test network is trained by the integrated random network distillation method to obtain the trained prediction network; S3. Based on the double-delayed deterministic deep policy gradient algorithm, a policy network, a first value network, a second value network, a target policy network consistent with the policy network structure, a first target value network consistent with the first value network structure, and a second target value network consistent with the second value network structure are constructed; S4, select the next moment state from the offline data set and input it into the target policy network to obtain the next moment action output by the target policy network; S5. The next-moment action and the corresponding next-moment state output by the target policy network are used as the input of the trained test network, and the mean square error values ​​of the outputs of all currently trained test networks and the output of the target network are added to obtain the first anti-exploration reward. S6, inputting the next moment action and the corresponding next moment state output by the target strategy network into the two target value networks; Select the smallest output of the two target value networks and calculate it with the first anti-exploration reward to obtain the temporal difference target of the value function; S7, inputting the input state and output action corresponding to the next moment state selected in step S4 into the two value networks, respectively obtaining the first outputs of the two value networks; According to the temporal difference target and the first output of the value network, the update gradient of the value network parameters is calculated and the value network is updated; S8, determine whether the current value network update times reaches the update times threshold, if so, update the strategy network parameters and go to step S9; otherwise, return to step S4; S9, judging whether the training is finished, if so, ending the offline reinforcement learning of the robot motion strategy; Otherwise, update the parameters of the target value network and the target strategy network, and return to step S4.

2. The offline reinforcement learning method for robot motion decision-making according to claim 1, characterized in that: The specific method of obtaining the trained prediction network in step S2 includes the following sub-steps: S2-1, build the target network, fix the parameters of the target network, and build k test networks with the same structure as the target network; S2-2, select batch data containing only input states and output actions from the preprocessed offline data set, input the selected batch data into the target network and the test network respectively, and obtain the output of the target network and the output of each test network accordingly; S2-3. Obtain the mean square error loss according to the output of the test network and the output of the target network, and update the parameters of the test network by the mean square error loss until the output of the prediction network converges to obtain the pre-trained test network.

3. The offline reinforcement learning method for robot motion decision-making according to claim 1, characterized in that: The calculation expression of the first anti-exploration reward in step S5 is: ; in represents the first anti-exploration reward; k is the number of test networks; is the output of the i-th trained test network; is the output of the target network; represents the L2 norm.

4. The offline reinforcement learning method for robot motion decision-making according to claim 1, characterized in that: The calculation expression of the timing difference target in step S6 is: ; in represents the temporal difference target; Indicates the current reward; and are all constants; Reward for the first anti-exploration; Represents the outputs of the two target-value networks.

5. The offline reinforcement learning method for robot motion decision-making according to claim 1, characterized in that: The updated gradient calculation expression of the value network parameters in step S7 is: ; in It means to find the gradient; represents the parameters of the j-th value network; represents the training data used in the current training batch, Indicates the number of training data used in the current training batch; represents the loss function used to find the gradient of the j-th value network parameter; Indicates that the input is When is the first output of the jth value network; represents the temporal difference target; represents the input state corresponding to the next moment state selected in step S4, Indicates the next state selected in step S4 The corresponding output action, r represents the current reward.

6. The offline reinforcement learning method for robot motion decision-making according to claim 1, characterized in that: The specific method for updating the policy network parameters in step S8 includes the following sub-steps: S8-1, input the input state into the policy network, and use the input state and the output corresponding to the policy network as the input of the trained test network, and add the mean square error value of the output of the currently trained test network to obtain the second anti-exploration reward; S8-2, input the input state and the output corresponding to the policy network into the two value networks, and obtain the second outputs of the two value networks respectively; Select the smallest second output of the two value networks and calculate it with the second anti-exploration reward to obtain the policy network parameter update gradient and update the policy network parameters; S8-3. Update the parameters of the target value network according to the current value network parameters; update the parameters of the target strategy network according to the current strategy network parameters.

7. The offline reinforcement learning method for robot motion decision-making according to claim 6, characterized in that: The calculation method of the second anti-exploration reward in step S8-1 is the same as the calculation method of the first anti-exploration reward.

8. The offline reinforcement learning method for robot motion decision-making according to claim 6, characterized in that: The calculation expression of the policy network parameter update gradient in step S8-2 is: ; in It means to find the gradient; represents the training data used in the current training batch, Indicates the number of training data used in the current training batch; Indicates that the input is is the second output of the jth value network; is a constant; Denotes the second anti-exploration reward.

9. The offline reinforcement learning method for robot motion decision-making according to claim 6, characterized in that: The update expressions of the parameters of the target value network and the target strategy network in step S9 are: ; in Represents the updated parameters of the j-th target value network; Represents the parameters of the current j-th value network; is a constant greater than 0 and less than 1; is the parameter before the j-th target value network update; Represents the updated parameters of the target policy network; Represents the current policy network parameters; Indicates the parameters of the target policy network before updating.

10. A robot control method based on the offline reinforcement learning method of robot motion decision-making according to any one of claims 1 to 9, characterized in that: The robot control method is generated by the trained strategy network to complete the robot control.