Robot dexterous operation method based on improved residual reinforcement learning and electronic equipment

Abstract

The invention relates to the technical field of robot dexterous operation, in particular to a robot dexterous operation method and electronic equipment based on improved residual reinforcement learning, and the method comprises the steps: constructing a basic strategy and a reinforcement learning agent, the reinforcement learning agent comprises a strategy network Actor and a value network Critic, outputting a basic action through the basic strategy, and outputting the basic action through the value network Critic; and outputting a residual action through the strategy network Actor, superposing the residual action on the basic action, generating a final dexterous operation of the robot, and introducing a residual amplitude modulation function in the training process of the reinforcement learning agent to adaptively scale the amplitude of the residual action. According to the method, a training mechanism based on residual amplitude adaptive modulation, off-line residual data optimization guidance and distributed value function enhanced reward signal utilization is provided, so that rapid, stable and efficient convergence of a residual reinforcement learning algorithm is realized.

Description

Technical Field

[0001] This invention relates to the field of robot dexterity manipulation technology, and in particular to a robot dexterity manipulation method and electronic device based on improved residual reinforcement learning. Background Technology

[0002] Existing imitation learning methods typically experience performance saturation after acquiring a certain number of expert demonstration samples, with further increases in demonstration data leading to significantly reduced gains and low cost-effectiveness. To further overcome this performance bottleneck, residual reinforcement learning proposes learning and adding a residual term to the output of the trained imitation learning policy (also known as the base policy) to compensate for the difference between the base policy and the optimal policy, thereby generating better action outputs. The algorithm framework can be found in [link to algorithm]. Figure 1 .

[0003] However, in challenging tasks such as robot vision grasping, reinforcement learning needs to learn the optimal policy from high-dimensional visual observations (e.g., 480×640×3 RGB images) relying solely on sparse binary reward signals (1 for success, 0 for failure). These tasks require algorithms not only to learn visual representations of the environment but also to reason and learn decision-making strategies to complete the task, making the learning process extremely difficult. Without a stable and efficient training mechanism, reinforcement learning often struggles to converge or even fails to complete the task in this scenario. To improve sample utilization, the field of robot intelligent control generally uses off-policy-based algorithms, such as td3, which employs an actor-critic architecture. (See [link to relevant documentation]). Figure 2 .

[0004] Currently, the main technical solutions for dexterous robot manipulation based on residual reinforcement learning include:

[0005] CN117863152B: A residual reinforcement learning method that constructs a reward function based on a state machine to improve learning efficiency. This method requires designing a state machine and reward function for a specific task and is not applicable to more general 0-1 reward tasks.

[0006] CN114454160A: This method belongs to the maximum entropy reinforcement learning framework. Its core innovation lies in using kernel methods and least squares techniques to directly solve the soft Bellman residual problem. However, the combination of the maximum entropy framework and kernel least squares makes the overall algorithm theory quite complex, and the threshold for practical engineering implementation and debugging is high.

[0007] CN115390439A: This method targets the typical task of robot assembly, explicitly decomposing the complex operation process into three discrete stages: "approach," "search," and "insertion." However, this method relies on threshold tuning, the switching is unstable, and it requires manual stage design based on the task itself, thus failing to achieve true end-to-end training.

[0008] In summary, existing technologies generally suffer from insufficient versatility and reliance on extensive manual prior design and engineering debugging. Whether it's reward construction based on state machines, kernel method solving, or explicit phased division of operational tasks, all are highly sensitive to task domain knowledge and hyperparameter settings, making it difficult to adapt to diverse robot operation tasks. Especially in complex, high-precision, and vision-driven scenarios, this reliance on prior knowledge and manual tuning has become a core bottleneck hindering residual reinforcement learning from laboratory research to large-scale industrial applications.

[0009] To address this problem, this invention proposes a robot dexterity manipulation method and electronic device based on improved residual reinforcement learning. Summary of the Invention

[0010] This invention addresses the aforementioned technical problems, specifically the issues of unstable convergence, low learning efficiency, and insufficient utilization of sparse rewards in existing residual reinforcement learning training processes. It proposes a training mechanism based on adaptive residual amplitude modulation, offline residual data optimization guidance, and distributed value function-enhanced reward signal utilization. This mechanism achieves fast, stable, and efficient convergence of the residual reinforcement learning algorithm. Specifically, it provides a robot dexterity manipulation method and electronic device based on improved residual reinforcement learning.

[0011] Firstly, this invention provides a robot dexterity manipulation method based on improved residual reinforcement learning. Specifically, it proposes an improved residual reinforcement learning method oriented towards visual input and suitable for 0 / 1 sparse reward scenarios. Without requiring manual design of reward functions and task stages, it achieves accelerated convergence and stable enhancement during the training process, exhibiting stronger generalization ability and practical deployment value. It is suitable for robot dexterity manipulation and includes the following steps:

[0012] Step A: Construct the base policy and reinforcement learning agent, which includes a policy network (Actor) and a value network (Critic);

[0013] Step B: Output basic actions through the basic strategy, output residual actions through the policy network Actor, and superimpose the residual actions onto the basic actions to generate the robot's final dexterous operation;

[0014] Step C: During the training process of the reinforcement learning agent, a residual amplitude modulation function that dynamically changes with training time or training steps is introduced to adaptively scale the amplitude of the residual actions. Initially, the modulation function outputs a first scaling factor; as the training progresses, this scaling factor gradually increases to a second scaling factor. This residual amplitude adaptive modulation mechanism dynamically adjusts the residual action amplitude, reducing the instability caused by excessively large random residuals in the early stages of policy exploration, thus ensuring the rationality and controllability of the policy output from the initial training phase.

[0015] Furthermore, the residual amplitude modulation function is selected from one or a combination of linear growth functions, piecewise functions, exponential growth functions, cosine rising functions, and sigmoid growth functions.

[0016] Furthermore, the scaling factor of the modulation function output in step C is gradually increased to the second scaling factor, specifically: using 0.01 as the first scaling factor, and within a predetermined training step of 5 × 10... 4 Within a single step, the scaling factor is linearly increased to 1.0 using a modulation function; when the number of training steps exceeds 5 × 10... 4 During the step, keep the scaling factor at 1.0.

[0017] Furthermore, it also includes steps for guiding network updates based on offline expert data, specifically including:

[0018] Step D1: Obtain an offline dataset containing multiple expert demonstration trajectories, where each demonstration trajectory includes at least state data and corresponding expert action data;

[0019] Step D2: Construct a residual action sample set based on the expert action data. The residual action sample set is used to provide optimization directions for the training of the policy network.

[0020] Furthermore, the step of constructing the residual action sample set in step D2 specifically involves: superimposing Gaussian noise of different amplitudes onto the expert action data to generate perturbed basic actions; calculating the difference between the expert action data and the perturbed basic actions as residual action samples; wherein the mean of the Gaussian noise is 0, and the standard deviation ranges from 0.1 to 1.0. This scheme guides the optimization strategy through offline expert data, generating residual examples using a small amount of expert data, providing clear and directional initial optimization guidance for the strategy, and significantly improving training efficiency and convergence speed without increasing model complexity.

[0021] Furthermore, a hierarchical sampling strategy is adopted to generate Gaussian noise of different amplitudes: five noise amplitude levels are preset, and the standard deviations corresponding to each level are 0.1, 0.3, 0.5, 0.7, and 1.0, respectively; during the sample generation process, noise is sampled from each level in a ratio of 1:1:1:1:1 to ensure that the sample set covers multi-scale residual scenarios.

[0022] Furthermore, step D2 includes: adding an additional behavior cloning loss when updating the policy network Actor.

[0023] in Output residual actions for the policy network Actor. For expert residual motion samples;

[0024] The overall loss function of the actor is:

[0025]

[0026] in The weight coefficient ranges from 0.1 to 1.0. This method provides a clear optimization direction for the Actor in the early stage of training, accelerating the convergence speed in the exploration phase.

[0027] Furthermore, the value network Critic adopts a distributed value function form, outputting a distributional representation of the random variable Z(s,a). This distributed value function provides a stable optimization signal to the policy network by preserving reward uncertainty and sparse reward characteristics. This scheme's distributed value function modeling mechanism utilizes a distributed value function instead of a traditional scalar value function to provide richer, more stable, and more discriminative feedback signals for policy optimization, improving learning efficiency and policy performance in sparse reward scenarios.

[0028] Furthermore, the distributed value function employs the Quantile Regression (QR) method to output an approximate distribution of 64 quantiles.

[0029] In a second aspect, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the robot dexterity operation method as described in any of the embodiments of the first aspect.

[0030] The beneficial effects of this invention are:

[0031] (1) Significantly improved training stability: The residual amplitude adaptive modulation mechanism proposed in this invention enables the residual policy to smoothly connect to the basic policy in the early stage of training, avoiding the problem of training collapse caused by task failure and lack of reward signal due to random residual actions in the early stage of traditional residual reinforcement learning, thereby ensuring the stability of policy training.

[0032] (2) Faster convergence speed: By using offline expert data to generate multi-scale residual action samples and adding behavior cloning regularization terms in Actor update, this invention provides a clear optimization direction for the policy, enabling the agent to quickly obtain effective gradients in sparse reward environment and significantly accelerate the policy convergence speed.

[0033] (3) Improved upper limit of strategy performance: The phased improvement of residual amplitude and the design of multi-scale residual samples enable the strategy to exert maximum compensation ability while maintaining training stability, thereby achieving an effective performance improvement of the basic strategy. Compared with the prior art, the present invention enables the strategy success rate and task completion rate to exceed the upper limit of the basic strategy;

[0034] (4) Enhanced value function expressive power: The introduction of distributed value function modeling method can directly learn the complete probability distribution of the reward, rather than just estimating the expected value, thus preserving sparse reward information and uncertainty features, improving the quality of policy update and learning efficiency in sparse reward environment;

[0035] (5) Enhanced generalization and robustness: Through multi-scale residual data and distributed value modeling, the strategy trained by this invention has stronger robustness to different task changes, environmental disturbances and motion noise, and can be widely applied to various complex tasks such as fine operation of robotic arms, control of embodied intelligent agents, industrial automation and service robots.

[0036] (6) Outstanding technical application value: This invention comprehensively improves the applicability of residual reinforcement learning in sparse reward, high uncertainty and complex control tasks, and provides efficient, stable and high-performance solutions for robot precision operation, complex environment control and continuous motion optimization tasks, which have significant technical progress and practical value. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the residual reinforcement learning framework in a specific embodiment of the present invention;

[0038] Figure 2 This is a diagram of the Actor-Critic architecture in a specific embodiment of the present invention;

[0039] Figure 3 This is a comparison curve of residual reinforcement learning in this application with other forms in a specific embodiment of the present invention. Detailed Implementation

[0040] This invention proposes an improved solution to address the problems of unstable training, difficulty in convergence, and limited performance improvement of Residual Reinforcement Learning (Residual RL) in robot control tasks. Existing Residual RL often suffers from early policy collapse, difficulty in exploring effective behaviors, and long-term performance stagnation when dealing with high-dimensional visual input and sparse reward scenarios.

[0041] Therefore, this application proposes at least the following core innovative ideas:

[0042] 1. Residual Amplitude Adaptive Modulation Mechanism: By dynamically adjusting the amplitude of residual actions during training, the intensity of residual intervention is controlled, avoiding a sharp drop in the performance of the base policy due to excessively large random residuals in the early stage of the policy. This improves training stability from the source and enables a smooth transition to a high-performance policy.

[0043] 2. Residual generation and optimization guidance method based on offline expert data: Generate reasonable residual action samples using a small amount of expert demonstration data, and use them as guidance signals for policy learning, so that Residual RL can obtain a clear optimization direction in the early stage of training, significantly improving exploration efficiency and convergence speed.

[0044] 3. Introduce a distributed value function modeling mechanism: By using a distributed value function to replace the traditional scalar value function, the algorithm can capture the characteristics of the reward distribution, thereby making fuller use of sparse reward signals and enhancing the stability and effectiveness of the policy optimization process.

[0045] Through the above-mentioned innovative concept, this invention enables Residual RL to achieve fast and stable convergence in tasks such as robot dexterity operation, complex planning and control, and embodied agent policy enhancement, and significantly improves the performance ceiling, demonstrating good versatility and promotional value.

[0046] Regarding the adaptive modulation mechanism for residual scale, the core idea of ​​the residual policy is to superimpose a learnable residual policy on top of the existing base policy to compensate for the shortcomings of the base policy. However, the setting of the residual scale directly affects whether training can start stably and eventually converge.

[0047] If the residual magnitude is too large: In the early stage of training, the random residual actions output by the Actor will severely disrupt the output of the basic policy, making it impossible for the agent to complete the task. The success rate drops to zero instantly, resulting in the inability to obtain effective reward signals, failure of Critic learning, and direct collapse of training.

[0048] If the residual magnitude is too small: the adjustability of the Actor is limited, the agent will find it difficult to break through the performance limit of the original Base Policy, and the residual strategy will be ineffective.

[0049] To address this, the present invention proposes an adaptive residual amplitude modulation mechanism based on the training phase. By gradually scaling the proportion of residual policies in the early stages of policy training, the mechanism smoothly integrates with the Base Policy, avoiding training instability and preserving room for performance improvement in the later stages.

[0050] Its implementation methods include, but are not limited to:

[0051] 1. Staged linear modulation:

[0052] At the initial training 50,000 (5×10) 4 Within a step, the residual magnitude is gradually increased from 0.01 to 1.0 using a linear or piecewise function, so that the residual strategy only needs to be fine-tuned in the early stage, reducing the destructive disturbance to the environment.

[0053] 2. Optional nonlinear or adaptive modulation methods:

[0054] Based on changes in the number of steps completed and the success rate, exponential, cosine, sigmoid, or performance-feedback-based adaptive modulation functions can be introduced to adjust the residual amplitude in conjunction with the learning effect.

[0055] As shown in Figure 3, in a certain robotic arm operation task, the success rate of the basic policy is about 60%~70%. If Residual RL is directly introduced, the initial success rate will drop sharply to close to 0%; however, by using the residual amplitude modulation method of this invention, the training collapse can be significantly alleviated, allowing Residual RL to recover smoothly from low success rate to high performance, and after stabilization, it can surpass BasePolicy.

[0056] For precision manipulation tasks performed by robotic arms, such as grasping, inserting / removing, assembling, rotating, and pushing / pulling operations, relying solely on random exploration is almost insufficient to complete the objective, thus preventing the agent from obtaining any positive reward signals. To improve the learnability of Residual RL in sparse reward scenarios, this invention proposes an optimization method based on offline expert data-guided residual learning. The core idea is to utilize existing expert demonstration data to generate perturbed residual action samples, providing the Actor with a reference optimization direction, thereby significantly improving early training efficiency and policy stability. Its implementation includes:

[0057] Expert residual motion construction method: assuming expert motion The ideal behavior of the robot in performing the task is represented by superimposing Gaussian noise of a certain amplitude to generate the actual action. This process can be viewed as a data augmentation method. By applying noise perturbations of different intensities to the ideal action, diverse residual action samples can be obtained, thereby improving the generalization and robustness of the model under different perturbation scales.

[0058] Layered residual amplitude sampling strategy: To avoid training bias caused by a single residual amplitude, this invention introduces a layered residual amplitude sampling method: for example, setting multiple amplitude layers (such as 0.1, 0.3, 0.5, 0.7, 1.0), each occupying about 10% of the data proportion, so that the strategy can take into account both small fine-tuning and large compensation capabilities during the training process.

[0059] Behavioral Cloning Regularization Based on Expert Data: Under sparse 0-1 rewards, the Critic struggles to provide effective gradients in the early stages. Therefore, this invention incorporates behavioral cloning loss as a regularization constraint into the Actor update, enabling the policy to simultaneously learn "how the residual should approximate the expert action." in, This regularization term can significantly enhance the early learning signals of Actors and prevent policy direction loss.

[0060] Standard Critics typically employ Clipped Double Q-learning to mitigate the overestimation of Q-values. However, in sparse reward scenarios, this method often leads to an excessive underestimation of the value function, thereby weakening the driving signal for policy updates and causing policy learning to stagnate or converge slowly. To alleviate this problem, this invention introduces a distributed value function for value estimation.

[0061] Unlike traditional Critic, which only models the expected value of rewards, the distributed value function directly learns the complete probability distribution of rewards, thus preserving more information about reward uncertainty, tail probability quality, and the frequency of sparse rewards. This method not only improves the expressive power of value estimation but also more effectively captures learning signals in sparse, highly volatile reward environments, making policy updates more stable and efficient.

[0062] Furthermore, by modeling the value distribution rather than a single scalar, this method can significantly improve the training performance of Residual RL in the following ways:

[0063] 1. Improve the utilization of sparse reward signals: Rare positive reward events can still be reflected in the distribution and will not be compressed by the mean.

[0064] 2. Reduce policy update direction distortion: Policy gradient updates can be optimized based on distribution information for risk-sensitive or risk-neutral purposes, which is more in line with the needs of real tasks.

[0065] 3. Mitigating the value function collapse problem: In Residual RL, residual actions introduce higher uncertainty. Distributed value functions can better model this uncertainty, thereby avoiding value function degradation.

[0066] Building upon this, we can further enhance the Critic's ability to model sparse rewards and long-tail distributions by combining it with implementations such as Quantile Regression, Implicit Quantile Networks (IQN), or Expectile-based Distributional Critic, thereby providing the policy with richer, more stable, and more discriminative optimization signals.

[0067] It should be noted that, Figure 1 This diagram illustrates the residual reinforcement learning framework, which uses online reinforcement learning to improve upon the two-stage imitation learning approach. First, a base policy is trained on an expert dataset using imitation learning and then frozen (without gradient updates). Next, a residual policy is trained using reinforcement learning to correct the actions output by the base policy. The reinforcement learning stage employs an offline policy actor-critic scheme, which leverages both demonstration data and environmental interaction data, and achieves more stable and safer exploration in the real world by directly controlling the residual magnitude.

[0068] like Figure 2 As shown, the Actor-Critic framework is a commonly used policy optimization framework in reinforcement learning, consisting of a policy network (Actor) and a value network (Critic). The Actor is responsible for outputting actions or action distributions based on the current observed state, used for direct decision-making and control; the Critic evaluates the value of the actions performed by the Actor, outputting state value or state-action value to guide the Actor's policy updates. The Actor performs gradient optimization based on the value estimation signal provided by the Critic, thereby continuously improving policy quality. By separating "decision-making" from "evaluation," the Actor-Critic architecture effectively reduces policy gradient variance, improves learning efficiency and training stability, and is a mainstream structure in continuous control and robot reinforcement learning.

[0069] As an optional implementation, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the robot dexterity operation method as described above.

[0070] The following uses a robotic arm grasping task as an example to illustrate the specific implementation steps of the residual amplitude adaptive modulation mechanism proposed in this invention.

[0071] 1. Basic Strategy Preparation

[0072] (1) The base policy π_b is obtained by training using imitation learning or offline reinforcement learning methods, so that the robotic arm can achieve a task success rate of about 60%~70% under the residual policy.

[0073] (2) Freeze the basic policy or update it only with a low learning rate, and add residual actions on top of its output actions.

[0074] 2. Residual Strategy Network Structure

[0075] The residual strategy (Actor) adopts a multilayer perceptron structure, including 3 fully connected layers, each with 256 hidden units, and the activation function is ReLU. The output dimension is consistent with the robot arm's motion dimension.

[0076] 3. Setting the residual amplitude modulation function (1) Set the initial residual amplitude scale_0=0.01 and the maximum residual amplitude scale_max=1.0. (2) Use the linear modulation function for the first 50,000 training steps:

[0077] ;

[0078] When the number of training steps t ≥ 50000, scale(t) = scale_max. (3) The actual actions generated during the training process are:

[0079]

[0080] Where a_b is the output action of the base policy and a_r is the output action of the residual policy.

[0081] 4. Training Process

[0082] Each interaction step includes: state acquisition → basic action generation → superimposing residual actions → action execution → collecting state transition samples → training the Actor and Critic. This modulation method ensures that the residual policy's perturbation of the basic policy progresses from weak to strong, achieving smooth integration.

[0083] The residual generation and guided training method based on offline expert data provided in this application includes expert data construction and residual sample generation: (1) Prepare a dataset D containing N expert demonstration trajectories, each trajectory containing state s and expert action a_expert. (2) Add random Gaussian noise of different amplitudes. Generate the base action corresponding to the state

[0084] .

[0085] (3) Generate residual motion samples according to the following formula:

[0086]

[0087] in, The noise is Gaussian noise with a mean of 0, and the standard deviation is sampled in five levels from 0.1 to 1.0, each accounting for 10% of the data volume, in order to form multi-scale residual samples.

[0088] This application's residual generation and guided training method based on offline expert data also includes: the introduction of a behavior cloning regularization term; and the addition of behavior cloning loss during Actor updates.

[0089]

[0090] The overall loss function of the actor is:

[0091]

[0092] Where λ is the weighting coefficient, and the preferred value range is 0.1 to 1.0.

[0093] This method provides Actor with a clear optimization direction in the early stages of training, accelerating the convergence speed during the exploration phase.

[0094] As one of the optional implementation methods of this application, the distributed value function Critic is specifically implemented through the following steps:

[0095] 1. Critic Network Structure

[0096] Critic uses a distributed value function to output the distribution representation of Z(s,a). In this embodiment, the Quantile Regression (QR) method is used to output an approximate distribution of 64 quantiles.

[0097] 2. Loss Function Design

[0098] The Critic is trained using Quantile Huber Loss to fit the full distribution of returns rather than the scalar expectation, thereby enhancing its ability to preserve value signals under sparse rewards.

[0099] 3. Compared with Clipped Double Q-learning, this distributed Critic can avoid underestimation of the value function, improve the quality of policy updates, and enable Residual RL to effectively improve the success rate in sparse reward scenarios.

[0100] Figure 3The graph shows a comparison between the residual reinforcement learning training method of this application and other conventional learning methods. The horizontal axis represents the number of steps, and the vertical axis represents the success rate. It includes four training curves: one without residual amplitude modulation, one using the method of this application, one without offline data, and one without distributed value functions. It can be seen that the method of this application (blue solid line) performs the best, with a final success rate of nearly 90%. It also exhibits the fastest convergence speed and the highest stability throughout the training process. Therefore, it is evident that the residual amplitude adaptive modulation mechanism, the residual generation and optimization guidance method using offline expert data, and the distributed value function designed in this application all have a significant positive impact on the algorithm performance.

[0101] In addition to the embodiments described above, the present invention may have other implementations. All technical solutions formed by equivalent substitution or equivalent transformation fall within the protection scope claimed by the present invention.

Claims

1. A robot dexterity manipulation method based on improved residual reinforcement learning, characterized in that, Includes the following steps: Step A: Construct the basic policy and reinforcement learning agent, wherein the reinforcement learning agent includes a policy network Actor and a value network Critic; Step B: Output basic actions through the basic strategy, output residual actions through the strategy network Actor, and superimpose the residual actions onto the basic actions to generate the robot's final dexterous operation; Step C: In the training process of the reinforcement learning agent, a residual amplitude modulation function is introduced to adaptively scale the amplitude of the residual action; wherein, in the early stage of training, the modulation function outputs a first scaling factor, and as the training process progresses, the scaling factor output by the modulation function gradually increases to a second scaling factor.

2. The robot dexterity operation method according to claim 1, characterized in that, The residual amplitude modulation function is selected from one or a combination of linear growth function, piecewise function, exponential growth function, cosine rising function, and S-shaped growth function.

3. The robot dexterity operation method according to claim 1, characterized in that, In step C, the scaling factor of the modulation function output is gradually increased to the second scaling factor. Specifically, the scaling factor is 0.01 as the first scaling factor, and the scaling factor is increased to the second scaling factor within a predetermined training step of 5 × 10. 4 Within a single step, the scaling factor is linearly increased to 1.0 using a modulation function; when the number of training steps exceeds 5 × 10... 4 During the step, keep the scaling factor at 1.

0.

4. The robot dexterity operation method according to claim 1, characterized in that, It also includes steps for guiding network updates based on offline expert data, specifically including: Step D1: Obtain an offline dataset containing multiple expert demonstration trajectories, where each demonstration trajectory includes at least state data and corresponding expert action data; Step D2: Construct a residual action sample set based on the expert action data.

5. The robot dexterity operation method according to claim 4, characterized in that, The construction of the residual action sample set in step D2 specifically includes: superimposing Gaussian noise of different amplitudes on the expert action data to generate perturbed basic actions; calculating the difference between the expert action data and the perturbed basic actions as residual action samples; wherein the mean of the Gaussian noise is 0 and the standard deviation ranges from 0.1 to 1.

0.

6. The robot dexterity operation method according to claim 5, characterized in that, The Gaussian noise of different amplitudes is generated using a hierarchical sampling strategy: five noise amplitude levels are preset, and the standard deviations of each level are 0.1, 0.3, 0.5, 0.7, and 1.0, respectively; during the sample generation process, noise is sampled from each level in a ratio of 1:1:1:1:1 to ensure that the sample set covers multi-scale residual scenarios.

7. The robot dexterity operation method according to claim 4, characterized in that, Step D2 is followed by: adding an additional behavior cloning loss during the update of the policy network Actor. in Output residual actions for the policy network Actor. For expert residual motion samples; The overall loss function of the Actor is: in This is the weighting coefficient, with a value ranging from 0.1 to 1.

0.

8. The robot dexterity operation method according to claim 1, characterized in that, The value network Critic adopts a distributed value function form, which outputs a distributed representation of random variables. The distributed value function provides a stable optimization signal for the policy network by preserving the uncertainty and sparse reward characteristics of rewards.

9. The robot dexterity operation method according to claim 8, characterized in that, The distributed value function uses the Quantile Regression method to output an approximate distribution of 64 quantiles.

10. An electronic device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the robot dexterity operation method as described in any one of claims 1 to 9.

Images