Robot control method based on decoupled multi-scheme prioritized experience replay reinforcement learning

Abstract

This invention discloses a robot control method based on decoupled multi-scheme priority experience replay reinforcement learning. The method includes: establishing a decoupled multi-scheme priority experience replay framework as a training sampling mechanism for a deep reinforcement learning algorithm; acquiring transfer samples of agent-environment interactions and storing them in an experience replay buffer; constructing a set of priority schemes and obtaining priority values, then independently sampling batch sub-sample sets and concatenating them to form a final sample set; training and updating the network parameters of the deep reinforcement learning algorithm and deploying it on the agent; inputting the robot's current state into the agent to obtain decision-making action instructions to control the robot. This invention integrates multiple priority schemes within a model-based reinforcement learning framework, significantly improving sample efficiency, stability, and anti-forgetting ability. Furthermore, it designs a batch allocation mechanism based on a preset sub-sample capacity parameter, mapping weights to the number of samples allocated, reducing parameter sensitivity and improving versatility.

Description

Technical Field

[0001] This invention relates to a robot control method, which falls under the field of artificial intelligence technology, and specifically to a robot control method based on decoupled multi-scheme priority experience replay reinforcement learning. Background Technology

[0002] In recent years, Deep Reinforcement Learning (DRL) has made significant progress in sequential decision-making problems. As an interdisciplinary field combining classical reinforcement learning methods with deep learning techniques, DRL has demonstrated strong application potential in multiple areas such as robot control, game artificial intelligence, and large language models. For example, the DQN algorithm was the first to combine neural networks with Q-learning, achieving breakthrough performance in Atari games; algorithms such as DDPG, TRPO, and PPO have solved continuous control problems; the SAC algorithm achieves excellent sample efficiency by integrating entropy maximization theory; and model-based reinforcement learning (MbRL) methods such as Dyna, Dreamer, and TD-MPC further improve sample efficiency by learning dynamic models and / or reward models.

[0003] Prioritized Experience Replay (PER) is a widely adopted technique to improve the efficiency and adaptability of DRL samples. Compared to uniform sampling, PER samples samples according to their priority, ensuring that more important samples are sampled more frequently. The core idea of ​​PER is that the samples in the replay set are not equally important for algorithm training. Therefore, researchers often need to design various prioritization sampling schemes to evaluate the importance of transfer samples from different perspectives and collect more important samples more frequently. Common prioritization scheme design choices include temporal difference (TD) error, model learning error, or recency, each targeting a specific scenario: TD error focuses on Q-function learning, model learning error is beneficial for learning dynamic and reward models, sampling recent transfer samples can improve sample efficiency, and sampling older samples can mitigate the catastrophic forgetting of old behaviors.

[0004] Naturally, by employing multiple prioritization schemes simultaneously, their positive effects (such as sample efficiency, adaptability, and resistance to forgetting) can be preserved and combined to further improve RL methods. However, WS-PER has the following drawbacks that prevent it from becoming a good candidate for multi-scheme PER: 1) High-parameter requirements: The scale and variance of different prioritization schemes vary greatly and change continuously during online training, leading to changes in the weight vector. Parameter tuning is extremely difficult. It requires first running experiments and then manually determining the weights corresponding to each loss, a tedious process; moreover, each loss changes significantly during training, and fixed weights are difficult to balance the losses at all times, causing some prioritization schemes to fail at specific moments.

[0005] 2) Limited applicability: Integrating new prioritization schemes through WS-PER can be difficult. For example, policy losses in DRLs may have a significantly varying value range, making it difficult to balance with the typically non-negative TD error.

[0006] 3) Coupled computation: The prioritization schemes are coupled together through a summation operation: Considering the integration of the three schemes, one with a higher-than-average value... But below average and The transferred samples may still have below-average quality. Therefore, it was ignored by WS-PER. But for the second priority sampling scheme The transfer sample is beneficial. Furthermore, this effect becomes more pronounced as the number of priority options increases.

[0007] Furthermore, simple techniques such as batch normalization or tracking normalizers are insufficient to fully address these issues: batch normalization only uses the normalization priority value of the current batch statistics, leading to locality problems, and the normalized values ​​are not comparable between training steps; while tracking normalizers track the global scale, they still rely on additional hyperparameters and tuning overhead.

[0008] Therefore, the core challenge facing current technology is how to design a multi-scheme PER method that can effectively integrate multiple prioritization schemes while ensuring training stability and reducing parameter tuning complexity. Existing technologies lack an effective mechanism that can make a principled trade-off between "preserving the advantages of multiple schemes" and "reducing parameter tuning complexity". Summary of the Invention

[0009] To address the problems existing in the background technology, this invention provides a robot control method based on decoupled multi-scheme priority experience replay reinforcement learning. This invention addresses the problems of coupled computation and high parameter tuning complexity caused by weighted summation of multiple schemes in traditional priority experience replay. It proposes to independently sample each priority scheme and concatenate the resulting sub-batches to form the final training batch, thereby achieving decoupled computation. Furthermore, a batch allocation mechanism based on a preset sub-sample capacity parameter is designed, mapping weights to the number of samples allocated, reducing parameter sensitivity and improving versatility. Multiple priority strategies can be integrated into a model-based reinforcement learning framework, including but not limited to sampling schemes based on model consistency, value, and recency. By decoupling the contributions of each scheme, this invention significantly improves sample efficiency, stability, and anti-forgetting ability, overcoming the problems of poor applicability and robustness of traditional methods.

[0010] The technical solution adopted in this invention is: The robot control method based on decoupled multi-scheme priority experience replay reinforcement learning of the present invention includes: Step 1: Establish the Decoupled Prioritized Experience Replay (DePER) framework as the training sampling mechanism for the Deep Reinforcement Learning (DRL) algorithm. The DePER framework uses batch-level ensemble instead of the traditional priority-level weighted summation ensemble.

[0011] Step 2: Obtain several transfer samples of the robot generated by the interaction between the agent and the environment in the deep reinforcement learning (DRL) algorithm and store them in the experience replay buffer; construct a set of schemes containing several priority schemes in the decoupled multi-scheme priority experience replay DePER framework, obtain the priority values ​​under the scheme set, and then independently sample batch sub-sample sets based on each priority value in the experience replay buffer. Finally, concatenate the batch sub-sample sets in the batch dimension to form the final training sample set.

[0012] Step 3: Input the final training sample set into the Deep Reinforcement Learning (DRL) algorithm to train and update the network parameters until the preset number of iterations is reached, and obtain the trained Deep Reinforcement Learning (DRL) algorithm.

[0013] Step 4: Deploy the trained Deep Reinforcement Learning (DRL) algorithm on the agent to be controlled, input the robot's current state into the agent, process it, and output decision-making action instructions, thereby controlling the robot to execute decision-making actions through the agent.

[0014] In the first step, the DePER framework for decoupling multiple-scheme priority experience replay is specifically as follows:

[0015]

[0016]

[0017] in, They respectively represent the 1st, 2nd, ..., ...th solutions in the set of solutions. The set of batch subsamples formed by sampling several transition samples obtained from the experience replay buffer for the I priority schemes, i = 1, 2, ..., I, where I is the total number of priority schemes; Indicates independent and identically distributed sampling; Indicates the sampling temperature parameter; Natural exponential function; For the first The priority scheme for the first Prioritization value for each transferred sample; For the final training sample set, This indicates vector concatenation; Total batch size; The modulus of the set.

[0018] In the first step, the deep reinforcement learning (DRL) algorithm adopts the model-based reinforcement learning algorithm MbRL, which is a simplified version of tdmpc, including an encoder network, a dynamic model network, a reward model network, a value function network, and a policy network.

[0019] In the second step, the transferred sample includes the robot's current state, actions, reward, and next state.

[0020] In the second step, the priority schemes in the scheme set include, but are not limited to, the temporal difference error scheme, the model consistency loss scheme, the recent sampling scheme, the access counting scheme, and the uniform sampling scheme.

[0021] In the second step, the batch subsample sets are obtained by sampling based on their respective preset subsample capacity parameters.

[0022] This invention is applicable to the multi-scheme priority experience replay objective in deep reinforcement learning, employing an efficient and feasible Decoupled Priority Experience Replay (DePER) algorithm. By integrating multiple schemes at the batch level rather than the priority level, a general budget allocation mechanism is constructed to optimally control the impact of each priority scheme on training. The DePER method adaptively balances the contribution of each priority scheme with training stability, thereby achieving a more stable and efficient training process. Extensive simulation results demonstrate that the DePER method possesses good sample efficiency, adaptability, anti-interference performance, and scalability.

[0023] The beneficial effects of this invention are: 1. Highly efficient and stable training process: By converting the weight vector into a preset subsample capacity parameter, the method of this invention significantly reduces the workload of parameter tuning and effectively avoids the training instability caused by drastic changes in sample weights, making the training process of multi-scheme PER more stable and efficient.

[0024] 2. Significantly improved model performance: This method can more effectively utilize the characteristics of each prioritization scheme, enabling the model to benefit from multiple signals such as TD error, model learning error, and recency. As a result, it achieves a success rate and sample efficiency far exceeding the baseline method in scenarios such as environmental change adaptation and anti-forgetting.

[0025] 3. Enhanced robustness and adaptability: DRL models trained using this method exhibit stronger robustness when faced with perturbations in the preset subsample size parameter. Furthermore, in continuous learning tasks, this method effectively mitigates the "catastrophic forgetting" problem, demonstrating superior lifelong learning potential.

[0026] 4. The method possesses universality and scalability: The decoupled ensemble theory analysis and algorithm proposed in this invention are not only applicable to specific model-based reinforcement learning frameworks, but can also be extended to other deep reinforcement learning methods based on priority experience replay. Furthermore, the computation time increases approximately linearly with the number of schemes, demonstrating good scalability. Attached Figure Description

[0027] Figure 1 This is a flowchart comparing the DePER method of this invention with existing methods; Figure 2 This is a schematic diagram illustrating the challenging environment design for the method of this invention; Figure 3 The figure shows the main experimental results of the method of this invention; Figure 4 This is a comparison diagram of the DePER method and variants of the WS-PER method of this invention; Figure 5 This is a statistical comparison chart of the method of the present invention based on the Rliable tool; Figure 6 The experimental results of perturbation of the pre-set subsample capacity parameter in the method of the present invention are shown in the figure. Figure 7 The figure shows the ablation experiment results of the method of the present invention; Figure 8 This is a graph showing the computational efficiency of the method of the present invention. Detailed Implementation

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] like Figure 1 As shown, existing methods use weight vectors The sampling mechanism is described, where L1, L2, L3, and L4 are the temporal difference objective function, model consistency objective function, recentity objective function, and visit count objective function, respectively, and P is the total priority value. This invention studies a multi-scheme priority experience replay mechanism in deep reinforcement learning, where the model input includes the environment state, action, reward, and next state. Theoretically, a decoupled sampling mechanism based on a preset subsample capacity parameter is constructed to balance the contribution of each priority scheme and training stability. Finally, a solvable optimization framework is established, achieving multi-scheme integration through independent sampling and batch concatenation.

[0030] The robot control method based on decoupled multi-scheme priority experience replay reinforcement learning of the present invention is as follows: Step 1: Establish the Decoupled Multi-Solution Priority Experience Replay (DePER) framework as the training sampling mechanism for the Deep Reinforcement Learning (DRL) algorithm. The DePER framework uses batch-level ensemble instead of the traditional priority-level weighted summation ensemble.

[0031] Prioritized Experience Replay (PER) is a widely adopted technique to improve the efficiency and adaptability of DRL samples. Compared to uniform sampling, PER samples samples according to their priority, ensuring that more important samples are sampled more frequently. In its general form, sampling... The PER of each prioritization scheme is sampled using the following formula:

[0032] in, For the playback buffer, the first The final priority value for each transferred sample, For the first Prioritization scheme The weight, For sampling temperature, For the first The first transferred sample The above formula is the commonly used weighted sum-percentage PER (WS-PER) priority sampling method.

[0033] The Decoupled Multi-Solution Priority Experience Replay (DePER) framework of this invention is as follows:

[0034]

[0035]

[0036] in, They respectively represent the 1st, 2nd, ..., ...th solutions in the set of solutions. The set of batch subsamples formed by sampling several transition samples obtained from the experience replay buffer for the I priority schemes, i = 1, 2, ..., I, where I is the total number of priority schemes; Indicates independent and identically distributed sampling; Indicates the sampling temperature parameter; Natural exponential function; For the first The priority scheme for the first Prioritization value for each transferred sample; For the final training sample set, This indicates vector concatenation; Total batch size; The modulus of the set.

[0037] Unlike the weighted sum-first experience replay (WS-PER), which integrates at the prioritization level, DePER integrates at the batch level: it first samples sub-batches independently using each prioritization scheme, and then concatenates these samples at the batch level to form the final batch. This mechanism integrates the original weight vector... Transform into a new sampling vector It has the following advantages: 1) Low parameter requirements: The parameter tuning workload of the budget vector is significantly lower than that of the weight vector, and even the uniform distribution of batch size is already a good starting point; 2) Wide applicability: Since each prioritization scheme performs sampling independently, the scale and variance of different schemes will not conflict with each other; 3) Decoupled computation: Each prioritization scheme is processed independently, avoiding the coupling problem introduced by the summation operation.

[0038] The Deep Reinforcement Learning (DRL) algorithm employs the model-based reinforcement learning algorithm MbRL, a simplified version of tdmpc, which includes an encoder network, a dynamic model network, a reward model network, a value function network, and a policy network, as detailed below: Encoder Network : State Encoding as hidden state representation .

[0039] Dynamic Model Network Predicting the hidden state at the next moment , for t The action at each moment, with the optimization objective being consistency loss: ,in, To find the expected operator, To predict the time domain, for t The decay factor at time, To stop the gradient operator, This represents the L2 norm.

[0040] Reward Model Network Predicting instant rewards The optimization objective is to reward the loss: , For cross-entropy loss, for t Moment Rewards The estimated value.

[0041] Value function network Predicting State-Action Value The optimization objective is value loss: , State-Action Value The estimated value, , To use parameters that are updated slowly Value function network, For policy networks.

[0042] Policy Network Generate actions The optimization objective is the policy loss: , Here is the entropy regularization coefficient. Entropy represents the random policy.

[0043] Step 2: Obtain several transition samples of the robot generated by the interaction between the agent and the environment in the deep reinforcement learning (DRL) algorithm and store them in the experience replay buffer; construct a scheme set containing several priority schemes in the decoupled multi-scheme priority experience replay DePER framework, obtain the priority values ​​under the scheme set, and then independently sample batch sub-sample sets based on each priority value in the experience replay buffer. Then, concatenate the batch sub-sample sets in the batch dimension to form the final training sample set; the transition samples include the robot's current state, action, reward, and next state.

[0044] The priority schemes in the scheme set include, but are not limited to, the temporal difference error scheme, the model consistency loss scheme, the recent sampling scheme, the access counting scheme, and the uniform sampling scheme, as detailed below: Timing differential error scheme: Prioritization value is calculated based on timing differential error. ,in, This is the priority value for timing difference error. Let s be the timing difference error. These are the current state, action, reward, and next state. The discount factor is used to focus on the Q-function.

[0045] Model consistency loss scheme: based on the consistency loss between the encoder output and the hidden state representation at the next time step. ,in, This is a priority value for model consistency loss, focusing on dynamic model learning.

[0046] Recent sampling scheme: Prioritize sampling of recently transferred samples stored in the buffer. ,in, This is the priority value for recent sampling. This provides a recent value for the sample, improving sample efficiency.

[0047] Access counting scheme: Prioritization values ​​are calculated based on the access frequency of state-action pairs. ,in, This is a priority value for the access count. The frequency of access to state-action pairs promotes exploration.

[0048] Uniform sampling scheme: uniform random sampling as a baseline.

[0049] The batch subsample set is obtained by sampling based on its respective preset subsample size parameter. The advantages of tuning the preset subsample size parameter are: 1) good performance can be obtained even with equal distribution; 2) each prioritization scheme is guaranteed to contribute to the final batch in a predictable and controllable manner, regardless of how it changes during training; 3) deviation from the preset subsample size parameter only causes a linear deviation in the sampled batch, while deviation from the weight vector may have an exponential impact.

[0050] Step 3: Input the final training sample set into the Deep Reinforcement Learning (DRL) algorithm to train and update the network parameters. The network parameters include encoder parameters, environment transfer model parameters, reward model parameters, value function parameters, and policy network parameters, until the preset number of iterations is reached, and the trained Deep Reinforcement Learning (DRL) algorithm is obtained.

[0051] Step 4: Deploy the trained Deep Reinforcement Learning (DRL) algorithm onto the agent to be controlled. Input the robot's current state into the agent, process it, and output decision-making action instructions. The agent then controls the robot to execute these actions. The specific control process of the agent is as follows: Input: Observations of the current environmental state are input into the algorithm in real time while the agent is performing a task. .

[0052] Processing: The algorithm observes the current environmental state through an encoder network. Encoded as hidden state The learned dynamic model is used for multi-step planning, and action sequences are generated through a policy network.

[0053] Output: The model outputs actions. It defines the decision-making behavior of the agent at the current moment.

[0054] Objective: The agent performs this action and interacts with the environment to obtain the next state. and rewards Transfer samples The data is stored in an experience replay buffer for subsequent training, thereby enabling adaptive decision control for complex environments.

[0055] The method of this invention utilizes a decoupled multi-scheme priority sampling mechanism to independently leverage the characteristics of each priority scheme in each training step. This effectively improves sample efficiency while maintaining the stability of the training process, ultimately enhancing the overall performance of the deep reinforcement learning model and enabling the agent to make efficient, adaptive, and robust decisions and controls in complex dynamic environments.

[0056] Specific embodiments of the present invention are as follows: 1) Verify the generalization ability and sample efficiency of the proposed DePER method in a multi-task learning environment. This embodiment's verification was conducted in four classic continuous control environments of DeepMind Control Suite: Hopper, Walker, Cheetah, and the more challenging 3D Humanoid. Figure 2 As shown, this embodiment features a specially designed challenging environment: in the first stage (the first 0.5M frames), the task requires the agent to move only backward; in the second stage (the last 0.5M frames), the agent needs to move backward for the first 1 / 3 of each round and forward for the remaining 2 / 3, representing a sharp change in the reward function and requiring the agent to have rapid adaptability. Simultaneously, in this stage, only the transition samples corresponding to forward movement are added to the replay buffer to further challenge the PER method's ability to handle catastrophic forgetting.

[0057] The method of this invention was applied to train a model-based reinforcement learning framework. For performance comparison, several baseline methods (i.e., algos (algorithms) in the figure) were set, including Uniform sampling, the Near method using only recent sampling, and the WS-PER method after extensive parameter tuning (where WS-PER-best is the WS-PER parameter tuning variant that achieved the best performance, and WS-PER-median is the WS-PER parameter tuning variant that achieved the median performance). Experimental results are as follows. Figure 3 and Figure 4 As shown, the reward value is the sum of the trajectory rewards, and the number of training samples is 1 million. The method of this invention achieves excellent performance in all environments. In the first stage, DePER performs similarly to most other algorithms because the differences between various sampling strategies are not significant when the replay buffer is small. In the second stage, DePER brings a significant improvement in sample efficiency (the difference in reward value with the same number of samples) compared to other baselines: it surpasses WS-PER-median in all environments and surpasses WS-PER-best in 3 out of 4 environments. In particular, in the Humanoid environment, DePER's convergence performance is slightly better than other baselines, while the sample efficiency gain obtained by WS-PER-best in the first stage fails to be maintained in the second stage.

[0058] like Figure 5 The image shows the comparison results using Rliable statistical tools, where median is the median, IQM is the quartile mean, Mean is the mean, and Optimality Gap is defined as:

[0059] in, It is the performance metric (normalized performance or normalized fitness) of the k-th algorithm.

[0060] As can be seen, DePER significantly outperforms various WS-PER variants, including WSPER-best, WSPER-75%, WSPER-median, WSPER-25%, and WSPER-worst, in both normalized performance (average return) and normalized fitness (number of rounds required for performance recovery after environmental changes). (Where best refers to the WS-PER variant achieving optimal performance, 75% refers to the variant achieving 75th percentile performance, median refers to the variant achieving median performance, 25% refers to the variant achieving 25th percentile performance, and worst refers to the variant achieving the worst performance.) While WS-PER-max achieves performance comparable to DePER, it exhibits greater variability. In terms of fitness, all WS-PER variants are significantly inferior to DePER, indicating that DePER can benefit from integrating multiple prioritization schemes.

[0061] Preliminary results demonstrate that the decoupling integration mechanism of this invention can more effectively perform multi-scheme priority experience replay, thereby achieving better generalization performance and sample efficiency.

[0062] 2) Evaluate the advantages of the method of the present invention in terms of parameter tuning robustness, especially its performance compared with WS-PER after heavy parameter tuning.

[0063] To achieve a fair comparison between DePER and WS-PER, the weight vector for WS-PER is adjusted in each environment. Thirteen different budget configurations were tuned (the preset subsample size parameter in DePER). (No parameter tuning was performed, same as above). The results of WS-PER after these parameter tunings were analyzed, and the performance of DePER was compared with them.

[0064] Experimental results are as follows Figure 4 and Figure 5 As shown. In a relatively simple environment such as Cheetah, parameter tuning... No significant performance differences were shown; almost all WS-PER variants performed similarly (to The parameter tuning was also not significant in other environments. It can indeed bring significant performance differences in the first and second stages, but the difference is not large. Even the performance and adaptability of WS-PER-best are significantly inferior to DePER.

[0065] Rliable's statistical results show that DePER significantly outperforms the WS-PER variant in both performance and adaptability. This verifies that DePER's decoupling mechanism has a fundamental advantage over traditional weighted summation methods, rather than being achieved solely through fine-tuning.

[0066] The above results show that the method of the present invention achieves better performance than heavily tuned WS-PER with low parameter cost through a decoupling integration mechanism, which significantly reduces the practical application threshold of multi-scheme PER.

[0067] 3) Verify the robustness of the method of the present invention under the perturbation of the preset subsample size parameter, and test the preset subsample size parameter. The impact of certain disturbances on DePER performance. Regarding the original DePER configuration... (These correspond to model consistency loss, reward loss, value loss, policy loss, value estimation, uniform sampling, recency, old samples, and reward signal, respectively, totaling 256.) Adding a perturbation of approximately 25% yields four perturbation versions: DePER-pt1, pt2, pt3, and pt4. The specific perturbation configurations are as follows: , , and .

[0068] Experimental results are as follows Figure 6 As shown. In the Walker and Cheetah environments, The perturbation did not significantly affect the performance of DePER; in the Hopper and Humanoid environments, the perturbation did cause some performance variation, but the variation in DePER was still smaller than that in WS-PER. Furthermore, all DePER variants still outperformed the WS-PER variants, indicating that... compared to It exhibits better robustness. The above results demonstrate that the preset subsample capacity parameter mechanism of the method of the present invention has stronger robustness to environmental changes and parameter perturbations, which is crucial in practical applications because precise parameter tuning is often costly.

[0069] 4) Verify the specific contribution of each prioritization scheme in the method of the present invention, and analyze the impact of removing a specific scheme on performance through ablation experiments.

[0070] This embodiment employs a "remove one" experiment design: one priority scheme is removed each time (its original capacity is allocated to the uniform sampling scheme, and the total capacity remains unchanged), and the impact on DePER performance is tested. Experiments were conducted in Walker, Hopper, Cheetah, and Humanoid environments. The schemes removed included recent sampling (-near), old sample sampling (-old), value estimation (-value), and value loss (-val_loss). Experimental results are as follows... Figure 7 As shown, 1) in relatively simple environments such as Walker and Cheetah, the performance degradation caused by removing recent samples is significantly greater than that caused by removing other schemes; 2) on Hopper, each prioritization scheme contributes to some extent, and removing any one of them will lead to a performance degradation; 3) in the second stage of the Humanoid environment, the adverse effects of removing old sample samples are similar to those of removing recent samples, because only newer samples are added to the replay buffer, so old sample samples are crucial for dealing with forgetting.

[0071] The above results demonstrate that the method of the present invention can effectively integrate the unique contributions of each prioritization scheme, and that different schemes play a complementary role in different environments and training stages, thus verifying the necessity of multi-scheme integration.

[0072] 5) Evaluate the computational efficiency of the method of this invention, particularly the relationship between computation time and the number of prioritization schemes. The total training time (clock time) for each variant is recorded by removing prioritization schemes one by one from the original DePER. Experimental results are as follows: Figure 8 As shown. The results indicate that the total training time of DePER increases approximately linearly with the number of schemes, and the fitted line of the real line segment is... (h) is obtained based on linear regression. This shows that DePER has good predictability and controllability when extended to more prioritization schemes, and the computational cost is within an acceptable range.

[0073] The above results verify the computational feasibility of the method in practical applications and provide a basis for extending it to more prioritization schemes. Future work can further reduce the runtime overhead of DePER by employing multithreading techniques.

[0074] The above content is merely a technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the core technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A robot control method based on decoupled multi-scheme prioritized experience replay reinforcement learning, characterized in that, include: Step 1: Establish the Decoupled Multi-Solution Priority Experience Replay (DePER) framework as the training sampling mechanism for the Deep Reinforcement Learning (DRL) algorithm; Step 2: Obtain several transfer samples of the robot generated by the interaction between the agent and the environment in the deep reinforcement learning (DRL) algorithm and store them in the experience replay buffer; construct a set of schemes containing several priority schemes in the decoupled multi-scheme priority experience replay DePER framework, obtain the priority values ​​under the scheme set, and then independently sample batch sub-sample sets based on each priority value in the experience replay buffer. Then, concatenate the batch sub-sample sets in the batch dimension to form the final training sample set. Step 3: Input the final training sample set into the Deep Reinforcement Learning (DRL) algorithm to train and update the network parameters until the preset number of iterations is reached, and obtain the trained Deep Reinforcement Learning (DRL) algorithm. Step 4: Deploy the trained Deep Reinforcement Learning (DRL) algorithm on the agent to be controlled, input the robot's current state into the agent, process it, and output decision-making action instructions, thereby controlling the robot to execute decision-making actions through the agent.

2. The robot control method based on decoupled multi-scheme prioritized experience replay reinforcement learning according to claim 1, characterized in that: In the first step, the DePER framework for decoupling multiple-scheme priority experience replay is specifically as follows: in, They respectively represent the 1st, 2nd, ..., ...th solutions in the set of solutions. The set of batch subsamples formed by sampling several transition samples obtained from the experience replay buffer for the I priority schemes, i = 1, 2, ..., I, where I is the total number of priority schemes; Indicates independent and identically distributed sampling; Indicates the sampling temperature parameter; Natural exponential function; For the first The priority scheme for the first Prioritization value for each transferred sample; For the final training sample set, This indicates vector concatenation; Total batch size; The modulus of the set.

3. The robot control method based on decoupled multi-scheme priority experience playback reinforcement learning according to claim 1, characterized in that: In the first step, the Deep Reinforcement Learning (DRL) algorithm employs a model-based reinforcement learning algorithm, including an encoder network, a dynamic model network, a reward model network, a value function network, and a policy network.

4. The robot control method based on decoupled multi-scheme priority experience playback reinforcement learning according to claim 1, characterized in that: In the second step, the transferred sample includes the robot's current state, actions, reward, and next state.

5. The robot control method based on decoupled multi-scheme priority experience playback reinforcement learning according to claim 1, characterized in that: In the second step, the priority schemes in the scheme set include the temporal difference error scheme, the model consistency loss scheme, the recent sampling scheme, the access counting scheme, and the uniform sampling scheme.

6. The robot control method based on decoupled multi-scheme priority experience playback reinforcement learning according to claim 1, characterized in that: In the second step, the batch subsample sets are obtained by sampling based on their respective preset subsample capacity parameters.

Images