Robot motion control method based on improved depth deterministic strategy gradient algorithm

Through the adaptive experience playback mechanism and priority sampling strategy, the training efficiency and stability of the DDPG algorithm are improved, and the problems of long training time and low learning efficiency in robot motion control are solved, and faster convergence speed and higher Control performance.

CN119036475AInactive Publication Date: 2024-11-29INSPUR SOFTWARE TECH CO LTD

Patent Information

Application Number
CN202411534993.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-31
Publication Date
2024-11-29
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The robot motion control method based on the deep deterministic strategy gradient algorithm has a long training time, low learning efficiency in complex environments, and is difficult to maintain a stable motion control effect.

Method used

Introduce an adaptive experience playback mechanism and a priority sampling strategy for empirical data, calculate the priority of empirical data through time difference error and instant reward, and dynamically adjust sample priority to improve training efficiency and stability.

Benefits of technology

It significantly improves sample utilization efficiency and training speed, and enhances the robot's adaptability to complex environments and the stability of behavioral control strategies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119036475A_ABST
    Figure CN119036475A_ABST
Patent Text Reader

Abstract

The invention discloses a robot motion control method based on an improved depth deterministic strategy gradient algorithm, and belongs to the technical field of deep reinforcement learning and robot motion control. Empirical data in an empirical buffer pool are screened through a self-adaptive empirical playback mechanism; the utilization efficiency of the empirical data is improved through a priority sampling strategy of the empirical data; the self-adaptive experience playback mechanism is used for designing a plurality of experience buffer areas based on dimensions of a state space and an action space of the robot, and designing a calculation mode of classification thresholds among the buffer areas; according to the priority sampling strategy of the empirical data, a mode of combining time difference errors and instant rewards is adopted, the sample priority of each piece of empirical data in an empirical buffer pool is calculated, and the empirical data are distributed to different empirical buffer areas. The convergence speed is increased, the performance of the final motion control strategy is improved, and the adaptability of the robot to the complex environment dynamic state and the stability of the behavior control strategy are enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of deep reinforcement learning and robot motion control, and in particular to a robot motion control method based on an improved deep deterministic policy gradient algorithm. Background Art

[0002] With the rapid development of artificial intelligence and robotics, the application areas of mobile robots have been continuously expanded, covering catering, medical care, education, transportation, industry and other fields. Robot motion control is the basis for robots to achieve intelligent operation and complete tasks efficiently. Ensuring stable and safe robot motion control has become the focus of current robotics research. Traditional robot motion control methods include proportional integral differential control algorithm, sliding mode control algorithm and fuzzy control algorithm, which can achieve stable control in a known and fixed environment, but these algorithms require a lot of time to manually adjust parameters and set control strategies, and have a high dependence on the environmental model of the robot. Therefore, it is difficult to maintain a stable motion control effect when facing a complex and unknown environment.

[0003] In recent years, robot motion control methods based on deep reinforcement learning have made certain progress. Deep reinforcement learning has good perception and decision-making capabilities, and is suitable for dealing with complex unknown environmental problems and sequential decision-making problems. Among them, the Deep Deterministic Policy Gradient (DDPG) algorithm performs well in dealing with high-dimensional continuous control problems. It trains the best robot motion behavior control strategy by training deep neural networks to fit the policy function, and is widely used in the field of robot motion control. However, in the DDPG algorithm, the traditional experience replay mechanism is adopted. The experience data in the experience buffer pool is selected according to the random uniform sampling strategy, and the importance of the experience data is not effectively evaluated, thereby reducing the sample utilization efficiency of the training data, which leads to the problem of low learning efficiency and long training time of the robot. The quality of the experience data in the experience buffer pool is crucial to the model training effect. High-quality experience data can significantly accelerate the learning process of the robot's optimal control strategy. Therefore, the robot motion control technology based on the DDPG algorithm still needs further research and optimization to improve its training efficiency and control performance. Summary of the invention

[0004] The technical task of the present invention is to address the above shortcomings and provide a robot motion control method based on an improved deep deterministic policy gradient algorithm. By improving the DDPG algorithm, the utilization efficiency of training data is improved, so that the robot can more effectively explore the action space during training, thereby accelerating the convergence speed and improving the performance of the final motion control strategy, enhancing the robot's adaptability to complex environmental dynamics and the stability of the behavior control strategy.

[0005] The technical solution adopted by the present invention to solve its technical problem is: A robot motion control method based on an improved deep deterministic policy gradient algorithm screens the experience data in the experience buffer pool through an adaptive experience replay mechanism; and improves the utilization efficiency of the experience data through a priority sampling strategy for the experience data.

[0006] The adaptive experience replay mechanism designs multiple experience buffers based on the dimensions of the robot's state space and action space, and designs a calculation method for the classification threshold between each buffer; The priority sampling strategy of the experience data adopts a combination of time difference error and immediate reward to calculate the sample priority of each experience data in the experience buffer pool and distribute the experience data to different experience buffers.

[0007] Furthermore, the adaptive experience replay mechanism evaluates the importance of the experience data and selects the experience data that is more helpful to the robot motion training process according to the set sampling strategy; during the training process, the sample priority of the updated experience data can be dynamically adjusted to adapt to the needs of the environment and tasks; In the early stage of robot model training, experience data is mainly concentrated in the initial accumulation stage, and the model will not enable the adaptive experience replay mechanism; the action is selected by combining OU noise and policy network, and the obtained experience data is stored in the experience buffer pool; when the amount of experience data in the experience buffer pool reaches 5000, the adaptive experience replay mechanism is enabled.

[0008] Furthermore, the priority sampling strategy of the experience data selects as much high-priority experience data as possible according to a certain strategy, while also ensuring that low-priority experience data is not ignored, so as to enhance the robot's environmental exploration ability and learning efficiency and improve the stability of the robot's motion control strategy.

[0009] Furthermore, the specific implementation steps of the method include: Step 1: Initialize the robot motion simulation environment and DDPG model: Step 1.1: Initialize the MuJoCo robot motion simulation environment and its related components; Step 1.2: Initialize the value network of the DDPG model , Strategy Network , target value network and target strategy network Related parameters, set the number of batches N and the maximum time step of the empirical data ; Step 2: Design multiple experience buffers to store the experience data in the experience buffer pool, and determine the number of experience buffers required based on the dimensions of the robot's state space and action space; Step 3: Initialize OU noise at the beginning of robot model training and select actions based on the policy network and OU noise ,in, represents the time-dependent OU noise, represents the policy network, Represents the parameters of the policy network and performs actions Get rewarded , next state And the robot movement end mark done, the experience data is stored in the experience buffer; Step 4: When the number of experience data in the experience buffer reaches 5000, the adaptive experience replay mechanism is enabled and the experience data are distributed to different experience buffers according to the proposed sampling strategy; Step 5: According to the classification threshold and the priority of the experience data, the new experience data generated during the interaction between the robot and the environment are allocated to different experience buffers; the experience data in the same buffer have similar priorities; Step 6: During model training, ensure that the model fully utilizes high-priority data while also taking into account low-priority experience data to improve exploration and generalization capabilities and accelerate the optimization and convergence speed of the robot motion control strategy; Step 7: Pass Calculate the target Q value, update the policy network, value network parameters, target policy network, and target value network parameters; represents the target Q value, Indicated in Execute actions when Rewards received, is the discount factor, is Time After resetting the robot's motion environment state, a new cycle is entered by entering step 4.

[0010] Furthermore, in step 1.1, the MuJoCo robot motion simulation environment and its related components are initialized. The robot motion simulation environment includes: A swimming robot model simulation environment with a 2-dimensional action space and an 8-dimensional state space; A single-leg robot model simulation environment with 3-dimensional action dimensions and 11-dimensional state dimensions; A two-legged robot model simulation environment with a 17-dimensional state space and a 6-dimensional action space; A semi-cheetah robot simulation environment with a 17-dimensional state space and a 6-dimensional action space; The simulation environment enables the robot to walk forward by controlling the movement of joints.

[0011] Furthermore, in step 2, as the dimension increases, the number of buffers also increases accordingly to cope with more complex robot motion control tasks; the experience buffers are initialized and numbered in sequence as 1, 2, 3, ..., k, where the experience data of buffer No. 1 has the highest priority, and the experience data of buffer No. k has the lowest priority; the number k of the experience buffers is calculated by the following formula: , Among them, m and n are hyperparameters. In order to prevent the number k of experience buffers from being set too large, the hyperparameters m and n are used to control the upper limit of k, thereby avoiding excessive computation and storage overhead caused by setting too many experience buffers. In step 5, the calculation formula for the classification threshold of k-1 experience buffers of k experience buffers is as follows: ; in, Represents the amount of experience data in the jth experience buffer.

[0012] Furthermore, in step 6, the sampling strategy of the empirical data is designed as follows: The number of samples in a Batch is N, which is specifically expressed as: ; in, represents the number of experience buffers sampled from the i-th buffer. The calculation formula for the number of experience data sampled from k experience buffers is as follows: ; in, represents the total number of experience data in the kth buffer, u and v are dynamic adjustment factors used to adjust the probability of selecting experience data from buffers of different priorities; since the samples in the buffer are not sorted by priority, uniform random sampling is used to extract experience data from the i-th buffer samples, the random seed is set to i to ensure the randomness of the sampling process; with the increase in the number of training times, the proportion of high-priority samples will also increase. In order to increase the sampling ratio of high-priority experience, the u value will increase by 1 after every 100 training rounds; at the same time, in order to maintain the exploratory nature of low-priority samples, it is necessary to set an upper limit for the u value until it reaches the hyperparameter Y, the default value of which is 10.

[0013] Furthermore, in step 4, a time difference error and an immediate reward are combined to calculate the sample priority of the experience data in the experience buffer pool, so as to evaluate the importance of the experience data and classify the experience data; The time difference error calculation formula is as follows: , ; The instant reward calculation formula is as follows: ; The formula for calculating the priority of experience data by combining the time difference error and the immediate reward is as follows: ; in, represents the time difference error, which updates the value function of the current state based on the estimated values ​​of other states obtained. Indicates the offset, which is used to ensure that the priority value of the experience data will not be zero; parameter and Used to adjust the relative weight of the temporal difference error and the immediate reward.

[0014] The present invention also claims a robot motion control device based on an improved deep deterministic policy gradient algorithm, comprising: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the above method.

[0015] The present invention also claims protection for a computer-readable medium having computer instructions stored thereon, which can implement the above method when executed by a processor.

[0016] Compared with the prior art, the robot motion control method based on the improved deep deterministic policy gradient algorithm of the present invention has the following beneficial effects: The present invention improves the DDPG algorithm by introducing an adaptive experience replay mechanism, and conducts experiments in the MuJoCo robot motion simulation environment. By comparing the performance of the method of the present invention with that of the DDPG algorithm in robot motion control tasks, the effectiveness and stability of the present method are verified. The present invention significantly improves sample utilization efficiency and training speed, and provides strong support for ensuring the stability and performance of the robot control strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the overall design of a robot motion control method based on an improved deep deterministic policy gradient algorithm provided by an embodiment of the present invention; Figure 2 The robot motion simulation environment diagram provided by the embodiment of the present invention is Figure 3 It is a schematic diagram of the average return comparison experimental results of the DDPG algorithm provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The present invention will be further described below in conjunction with specific embodiments.

[0019] The embodiment of the present invention provides a robot motion control method based on an improved deep deterministic policy gradient algorithm, which solves the problems of long training time, low learning efficiency and unstable control strategy in the initial training stage of the deep deterministic policy gradient algorithm in robot motion control, including: screening the experience data in the experience buffer pool through an adaptive experience playback mechanism; improving the utilization efficiency of the experience data through a priority sampling strategy of the experience data. Thereby significantly accelerating the training time and learning rate of the robot and improving the stability of the robot motion control.

[0020] The adaptive experience replay mechanism evaluates the importance of the experience data and selects the experience data that is more helpful to the robot motion training process according to a certain sampling strategy. During the training process, the sample priority of the updated experience data can be dynamically adjusted to adapt to the needs of the environment and the task.

[0021] The adaptive experience replay mechanism designs multiple experience buffers based on the dimensions of the robot's state space and action space, and designs a calculation method for the classification thresholds between the buffers.

[0022] In the early stage of robot model training, experience data is mainly concentrated in the initial accumulation stage, and the model will not enable the adaptive experience replay mechanism; the action is selected by combining OU noise and policy network, and the obtained experience data is stored in the experience buffer pool; when the amount of experience data in the experience buffer pool reaches 5000, the adaptive experience replay mechanism is enabled.

[0023] The priority sampling strategy of the experience data adopts a combination of time difference error and immediate reward to calculate the sample priority of each experience data in the experience buffer pool and distribute the experience data to different experience buffers.

[0024] The priority sampling strategy for the experience data selects as much high-priority experience data as possible according to a certain strategy, while also ensuring that low-priority experience data is not ignored, so as to enhance the robot's environmental exploration ability and learning efficiency and improve the stability of the robot's motion control strategy.

[0025] The specific implementation steps of this method include: Step 1: Initialize the robot motion simulation environment and DDPG model: Step 1.1: Initialize the MuJoCo robot motion simulation environment and its related components. The robot motion simulation environment is as follows: Figure 2 As shown, (a) Swimmer-v2 environment is a swimming robot model with 2-dimensional action space and 8-dimensional state space; (b) Hopper-v2 environment is a single-leg robot model with 3-dimensional action dimension and 11-dimensional state dimension; (c) Walker2d-v2 environment is a two-leg robot model with 17-dimensional state space and 6-dimensional action space; (d) HalfCheetah-v2 environment is a half-cheetah robot with 17-dimensional state space and 6-dimensional action space. The task goal of the four robot motion simulation environments is to make the robot walk forward by controlling the movement of the joints; Step 1.2: Initialize the value network of the DDPG model , Strategy Network , target value network and target strategy network Related parameters, set the number of batches N and the maximum time step of the empirical data .

[0026] Step 2: Design multiple experience buffers to store the experience data in the experience buffer pool, such as Figure 2 As shown. According to the dimensions of the robot's state space and action space, determine the number of experience buffers required. As the dimensions increase, the number of buffers also increases accordingly to cope with more complex robot motion control tasks; initialize the experience buffers and number the experience buffers in sequence as 1, 2, 3, ..., k, where the experience data of buffer No. 1 has the highest priority, and the experience data of buffer No. k has the lowest priority. The number of experience buffers k is calculated by the following formula: , Among them, m and n are hyperparameters. In order to prevent the number k of experience buffers from being set too large, the hyperparameters m and n are used to control the upper limit of k, thereby avoiding excessive computational and storage overhead caused by too many experience buffers. Through multiple experiments and verifications, when m and n are set to 5 and 12 respectively, the present invention can show good versatility and stability in a variety of robot motion control environments.

[0027] Step 3: In the early stage of robot model training, experience data is mainly concentrated in the initial accumulation stage. In this stage, in order to fully collect and utilize early experience samples, the model will not enable the adaptive experience replay mechanism, which helps to quickly collect enough experience data in the initial stage of robot motion training. Initialize OU noise and select actions based on the policy network and OU noise ,in, represents the time-dependent OU noise, represents the policy network, Represents the parameters of the policy network and performs actions Get rewarded , next state And the robot movement end flag done, the experience data is stored in the experience buffer.

[0028] Step 4: When the number of experience data in the experience buffer reaches 5000, the adaptive experience replay mechanism is enabled, and the experience data is distributed to different experience buffers according to the proposed sampling strategy. The sample priority of the experience data in the experience buffer pool is calculated by combining the time difference error and the immediate reward to evaluate the importance of the experience data and classify the experience data.

[0029] The time difference error calculation formula is as follows: , ; The instant reward calculation formula is as follows: ; The formula for calculating the priority of experience data by combining the time difference error and the immediate reward is as follows: ; in, represents the time difference error, which updates the value function of the current state based on the estimated values ​​of other states obtained. Indicates the offset, which is used to ensure that the priority value of the experience data will not be zero; parameter and Used to adjust the relative weight of the temporal difference error and the immediate reward.

[0030] Step 5: The classification threshold calculation formula for k-1 experience buffers of k experience buffers is as follows: ; in, Represents the amount of experience data in the jth experience buffer.

[0031] According to the classification threshold and the priority of the experience data, the new experience data generated during the interaction between the robot and the environment are allocated to different experience buffers; the experience data in the same buffer have similar priorities to avoid additional sorting overhead.

[0032] Step 6: After the experience data is classified and distributed to different buffers, batches of experience data need to be selected for model training. During the model training process, high-frequency selection of high-priority experience data can speed up model training, but relying solely on these experience data will limit the robot's ability to explore the environment, thus falling into local optimality or overfitting.

[0033] Therefore, during the model training process, it is necessary to ensure that the model fully utilizes high-priority data while also taking into account low-priority experience data to improve exploration and generalization capabilities and accelerate the optimization and convergence speed of the robot motion control strategy. The sampling strategy for experience data is designed as follows: The number of samples in a Batch is N, which is specifically expressed as: ; in, represents the number of experience buffers sampled from the i-th buffer. The calculation formula for the number of experience data sampled from k experience buffers is as follows: ; in, represents the total number of experience data in the kth buffer, u and v are dynamic adjustment factors used to adjust the probability of selecting experience data from buffers of different priorities; since the samples in the buffer are not sorted by priority, uniform random sampling is used to extract experience data from the i-th buffer samples, the random seed is set to i to ensure the randomness of the sampling process; with the increase in the number of training times, the proportion of high-priority samples will also increase. In order to increase the sampling ratio of high-priority experience, the u value will increase by 1 after every 100 training rounds; at the same time, in order to maintain the exploratory nature of low-priority samples, it is necessary to set an upper limit for the u value until it reaches the hyperparameter Y, the default value of which is 10.

[0034] Step 7: Pass Calculate the target Q value and update the policy network, value network parameters, target policy network, and target value network parameters. represents the target Q value, Indicated in Execute actions when Rewards received, is the discount factor, is Time After resetting the robot's motion environment state, go to step 4 to enter a new round of exploration.

[0035] Figure 3 This is a schematic diagram of the average return of this method embodiment and the classic DDPG algorithm. During the robot motion training process, the algorithm effect is evaluated every 3000 time steps. The average return calculated through 10 tests is used to evaluate the motion control effect of the strategy. The average return of the algorithm with better performance converges faster and has a larger value. The experimental results show that compared with the DDPG model, this method has a good convergence effect, can obtain more returns in fewer time steps, and the average return has been significantly improved. It has stronger stability and faster learning speed in robot motion control.

[0036] This method solves the problems of long training time and low learning efficiency in robot motion control algorithms based on the DDPG algorithm. These problems restrict the stability and performance of robot motion control training, thus limiting the practical application of the algorithm in robot motion control. This method improves the DDPG algorithm and improves the efficiency of training data utilization, so that the robot can explore the action space more effectively during training, thereby accelerating the convergence speed and improving the performance of the final motion control strategy, enhancing the robot's ability to adapt to complex environmental dynamics and the stability of the behavior control strategy.

[0037] An embodiment of the present invention also provides a robot motion control device based on an improved deep deterministic policy gradient algorithm, comprising: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the robot motion control method based on the improved deep deterministic policy gradient algorithm described in the above embodiment.

[0038] The embodiment of the present invention further provides a computer-readable medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, the robot motion control method based on the improved deep deterministic policy gradient algorithm described in the above embodiment is implemented. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code for implementing the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device is enabled to read and execute the program code stored in the storage medium.

[0039] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0040] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0041] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0042] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0043] The present invention is shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A robot motion control method based on an improved deep deterministic policy gradient algorithm, characterized in that: The experience data in the experience buffer pool is screened through the adaptive experience playback mechanism; the utilization efficiency of experience data is improved through the priority sampling strategy of experience data; The adaptive experience replay mechanism designs multiple experience buffers based on the dimensions of the robot's state space and action space, and designs a calculation method for the classification threshold between each buffer; The priority sampling strategy of the experience data adopts a combination of time difference error and immediate reward to calculate the sample priority of each experience data in the experience buffer pool and distribute the experience data to different experience buffers.

2. The robot motion control method based on the improved deep deterministic policy gradient algorithm according to claim 1 is characterized in that: The adaptive experience replay mechanism evaluates the importance of the experience data and selects experience data that is helpful for the robot motion training process according to the set sampling strategy; during the training process, the sample priority of the experience data is dynamically adjusted to adapt to the needs of the environment and tasks; In the early stage of robot model training, a combination of OU noise and policy network is used to select actions, and the obtained experience data is stored in the experience buffer pool; when the amount of experience data in the experience buffer pool reaches 5000, the adaptive experience replay mechanism is enabled.

3. The robot motion control method based on the improved deep deterministic policy gradient algorithm according to claim 1 is characterized in that: The priority sampling strategy for the experience data selects as much high-priority experience data as possible according to the strategy, while also ensuring that low-priority experience data is not ignored.

4. The robot motion control method based on the improved deep deterministic policy gradient algorithm according to claim 1 is characterized in that: The specific implementation steps of this method include: Step 1: Initialize the robot motion simulation environment and DDPG model: Step 1.1: Initialize the MuJoCo robot motion simulation environment and its related components; Step 1.2: Initialize the value network of the DDPG model , Strategy Network , target value network and target strategy network Related parameters, set the number of batches N and the maximum time step of the empirical data ; Step 2: Design multiple experience buffers to store the experience data in the experience buffer pool, and determine the number of experience buffers required based on the dimensions of the robot's state space and action space; Step 3: Initialize OU noise at the beginning of robot model training and select actions based on the policy network and OU noise ,in, represents the time-dependent OU noise, represents the policy network, Represents the parameters of the policy network and performs actions Get rewarded , next state And the robot movement end mark done, the experience data is stored in the experience buffer; Step 4: When the number of experience data in the experience buffer reaches 5000, the adaptive experience replay mechanism is enabled, and the experience data are distributed to different experience buffers according to the proposed sampling strategy; Step 5: According to the classification threshold and the priority of the experience data, the new experience data generated during the interaction between the robot and the environment are allocated to different experience buffers; the experience data in the same buffer have similar priorities; Step 6: During model training, ensure that the model fully utilizes high-priority data while also taking into account low-priority experience data to improve exploration and generalization capabilities and accelerate the optimization and convergence speed of the robot motion control strategy; Step 7: Pass Calculate the target Q value, update the policy network, value network parameters, target policy network, and target value network parameters; represents the target Q value, Indicated in Execute action when Rewards received, is the discount factor, is Time After resetting the robot's motion environment state, a new cycle is entered by entering step 4.

5. The robot motion control method based on the improved deep deterministic policy gradient algorithm according to claim 4 is characterized in that: The step 1.1 is to initialize the MuJoCo robot motion simulation environment and its related components. The robot motion simulation environment includes: A swimming robot model simulation environment with a 2-dimensional action space and an 8-dimensional state space; A single-leg robot model simulation environment with 3-dimensional action dimensions and 11-dimensional state dimensions; A two-legged robot model simulation environment with a 17-dimensional state space and a 6-dimensional action space; A semi-cheetah robot simulation environment with a 17-dimensional state space and a 6-dimensional action space; The simulation environment enables the robot to walk forward by controlling the movement of joints.

6. The robot motion control method based on the improved deep deterministic policy gradient algorithm according to claim 4 or 5, characterized in that: In step 2, as the dimension increases, the number of buffers also increases accordingly; the experience buffer is initialized, and the experience buffers are numbered in sequence as 1, 2, 3, ..., k, where the experience data of buffer No. 1 has the highest priority, and the experience data of buffer No. k has the lowest priority; the number k of the experience buffer is calculated by the following formula: , Among them, m and n are hyperparameters, and the upper limit of k is controlled by the hyperparameters m and n to prevent the number k of experience buffers from being set too large; In step 5, the calculation formula for the classification threshold of k-1 experience buffers of k experience buffers is as follows: ; in, Represents the amount of experience data in the jth experience buffer.

7. The robot motion control method based on the improved deep deterministic policy gradient algorithm according to claim 6 is characterized in that: In step 6, the sampling strategy of the empirical data is designed as follows: The number of samples in a Batch is N, which is specifically expressed as: ; in, represents the number of experience buffers sampled from the i-th buffer. The calculation formula for the number of experience data sampled from k experience buffers is as follows: ; in, represents the total number of experience data in the kth buffer, u and v are dynamic adjustment factors used to adjust the probability of selecting experience data from buffers of different priorities; since the samples in the buffer are not sorted by priority, uniform random sampling is used to extract experience data from the i-th buffer samples, the random seed is set to i to ensure the randomness of the sampling process; with the increase in the number of training times, the proportion of high-priority samples will also increase. In order to increase the sampling ratio of high-priority experience, the u value will increase by 1 after every 100 training rounds; at the same time, in order to maintain the exploratory nature of low-priority samples, it is necessary to set an upper limit for the u value until it reaches the hyperparameter Y, the default value of which is 10.

8. The robot motion control method based on the improved deep deterministic policy gradient algorithm according to claim 4 is characterized in that: The step 4 uses a combination of time difference error and instant reward to calculate the sample priority of the experience data in the experience buffer pool, so as to evaluate the importance of the experience data and classify the experience data; The time difference error calculation formula is as follows: , ; The instant reward calculation formula is as follows: ; The formula for calculating the priority of experience data by combining the time difference error and the immediate reward is as follows: ; in, represents the time difference error, which updates the value function of the current state based on the estimated values ​​of other states obtained. Indicates the offset, which is used to ensure that the priority value of the experience data will not be zero; parameter and Used to adjust the relative weight of the temporal difference error and the immediate reward.

9. A robot motion control device based on an improved deep deterministic policy gradient algorithm, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the method described in any one of claims 1 to 8.

10. A computer-readable medium, characterized in that The computer readable medium stores computer instructions, which, when executed by a processor, can implement the method according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • Deep reinforcement learning robot control method based on priority experience playback

    CN111421538A

  • Autonomous vehicle simulation using machine learning

    CN112034730A

  • Robot path planning method and system based on priority experience playback mechanism

    CN115509233A

  • Fuel cell vehicle deep reinforcement learning energy management method based on experience pool migration

    CN116001655A

  • Autonomous vehicle simulation using machine learning

    US20200368906A1

Cited By

  • Robot real-time trajectory smoothing method, device and equipment and storage medium

    CN122231922A

  • A method, apparatus, device, and storage medium for real-time trajectory smoothing of a robot.

    CN122231922B