Residual error correction robot skill synthesis method and system

Abstract

The invention discloses a residual error correction robot skill synthesis method and system, and the method comprises the steps: obtaining the data of a perception sensor; encoding the sensing sensor data into a current potential state; randomly selecting a candidate action of a robot body, inputting the current potential state and the candidate action into a generation engine, and generating probability distribution of a future potential state at the next moment so as to form a virtual track; in the virtual track generated by the generation engine, an imagination strategy is synthesized through reinforcement learning; the knowledge of the imagination strategy is compressed and migrated to a lightweight self-owned strategy, and the self-owned strategy is deployed on the robot body; a residual error strategy is deployed in a robot entity, and a final execution action of the robot entity is generated by linear superposition of the body strategy and the residual error strategy. According to the method, imagination strategy knowledge compression migration and double-strategy execution are fused, so that the robustness and adaptability of the robot in the environment are improved; after deployment, errors caused by model inaccuracy or environment sudden change are rapidly corrected.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and robotics technology, specifically relating to a method and system for synthesizing robot skills with residual correction. Background Technology

[0002] As robotics technology permeates broader industrial and service sectors, enabling it to learn and adapt autonomously in open, unstructured environments has become a core challenge. One of these core challenges is endowing robots with the ability to operate robustly and adaptively in unstructured, dynamically changing real-world environments. To achieve this goal, researchers have explored various learning paradigms; however, existing technological approaches all face inherent and insurmountable limitations, which collectively constitute the bottleneck in the current development of robot intelligence.

[0003] Current mainstream robot learning paradigms can be categorized as follows: Supervised learning based on massive amounts of data, which uses motion capture to record tens of thousands of videos or motion trajectories of humans performing tasks, allowing the model to learn the mapping from observation to action. However, this method can only generalize within the range of the training data distribution. It may fail if an unfamiliar object appears or is observed from a completely new angle. Reinforcement learning, on the other hand, involves receiving a reward or penalty signal after performing an action, and using a strategy to maximize the long-term cumulative reward. Current robots heavily rely on known or structured environments and cannot handle unknown scenarios. Summary of the Invention

[0004] To address the problems mentioned in the background section, this invention provides a method and system for robot skill synthesis with residual correction.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a robot skill synthesis method with residual correction, comprising the following steps: S1. Acquire sensor data; S2. Encode the sensor data into the current latent state; S3. Randomly select a candidate action of the robot body, input the current latent state and the candidate action into the generation engine, generate the probability distribution of the future latent state at the next moment, so as to form a virtual trajectory. S4. In the virtual trajectory generated by the generation engine, an imaginary strategy is synthesized through reinforcement learning; S5. Compress and transfer the knowledge of the imagination strategy into a lightweight embodied strategy, and deploy the embodied strategy on the robot body. S6. Deploy residual strategies on robot entities. The final execution action of the robot entity is generated by the linear superposition of the embodied strategy and the residual strategy. S7. During the robot's execution of the task, an online correction step is performed, including: keeping the network parameters of the embodied policy frozen; and using real-time interactive data, updating only the residual policy through an online reinforcement learning algorithm to compensate for the execution deviation of the embodied policy in the virtual network and the real operation.

[0006] In a preferred embodiment of the present invention, it further includes: The robot identifies areas with limited prior experience as regions of curiosity and samples data generated within these regions to guide the synthesis of imaginative strategies.

[0007] In a preferred embodiment of the present invention, in S1, the sensing sensor data includes: body sensing sensor data and external sensing sensor data; The body sensing sensor data includes at least one of the following: joint encoder data, torque sensor data, and inertial measurement unit data; the external sensing sensor data includes at least one of the following: camera data and lidar data.

[0008] In a preferred embodiment of the present invention, in step S2, encoding the sensing sensor data into the current latent state specifically involves compressing the high-dimensional, multimodal raw sensing sensor data into a low-dimensional, information-dense latent state representation.

[0009] In a preferred embodiment of the present invention, in step S3, the generation engine is a learning model based on a generative adversarial network, comprising: a generator for receiving the current latent state and candidate actions, and outputting the predicted latent state at the next time step; A discriminator is used to assess the physical plausibility of the future potential state; The learning model learns a reverse denoising process to generate a probability distribution of future latent states based on the current latent state and candidate actions.

[0010] In a preferred embodiment of the present invention, in step S5, the knowledge of the imagination strategy is compressed and transferred, specifically through a strategy distillation method, namely: The imagination strategy provides a soft target action probability distribution, and the embodied strategy is trained through supervised learning to mimic the soft target action probability distribution provided by the imagination strategy. The output layer of the imagined strategy uses a temperature scaling parameter to smooth the probability distribution of the soft target's actions.

[0011] In a preferred embodiment of the present invention, in step S7, the online reinforcement learning algorithm is a policy-agnostic reinforcement learning algorithm, specifically a soft actor-critic algorithm.

[0012] In a preferred embodiment of the present invention, it further includes: Collect the actual state transition sequences generated during the physical execution of the robot and feed them back to the generation engine. Based on the feedback, calibrate and optimize the generation engine.

[0013] A residual-corrected robot skill synthesis system, comprising: The data acquisition unit is used to acquire sensor data from the robot itself. A latent state encoding unit is used to encode the sensing sensor data into the current latent state; The generation engine unit is configured to generate the probability distribution of the future latent state at the next moment based on the current latent state and candidate actions, so as to form a variety of virtual trajectories. A policy synthesis unit is used to synthesize an imagined policy in the virtual trajectory through reinforcement learning. The strategy distillation unit is used to compress and transfer the knowledge of the imagined strategy into a lightweight embodied strategy; The system also includes a dual-policy execution and correction unit configured to deploy the embodied policy and a residual policy on the robot entity, wherein the network parameters of the embodied policy are kept frozen during execution and the residual policy is updated only using an online reinforcement learning algorithm with real-time interactive data.

[0014] In a preferred embodiment of the present invention, the latent state encoding unit includes a cyclic variational autoencoder for processing sensing sensor data; The dual-strategy execution and correction unit is configured to update the residual strategy using a soft actor-critic algorithm.

[0015] This invention addresses the shortcomings of the prior art and has the following beneficial effects: This invention employs a technical solution that integrates imagined policy knowledge compression and transfer with dual-policy execution. During robot execution, the network parameters of the embodied policy are frozen, and only a lightweight residual policy is updated through an online reinforcement learning algorithm. The final action is generated by the linear superposition of the two. The robot can retain the generalized prior knowledge learned from large-scale virtual training, and can also compensate for the execution deviation between the virtual network and real-world operations in real time through the residual policy. Compared with the problem that existing technologies struggle to handle unknown scenarios or rely on structured environments, this invention avoids forgetting caused by online learning by freezing the embodied policy. At the same time, it utilizes the residual policy to achieve rapid adaptation to dynamic, unstructured real-world environments, significantly improving the robot's robustness and adaptability in open environments. This allows it to quickly correct errors caused by model inaccuracies or sudden environmental changes after deployment.

[0016] This invention, by setting up a generation engine unit, synthesizes imagined strategies within virtual trajectories. It can generate physically plausible probability distributions of future latent states based on the current latent state and candidate actions, forming a massive number of virtual trajectories. The strategy synthesis unit can utilize massively parallel computing to efficiently train reinforcement learning strategies in a virtual environment without interacting with the expensive and slow physical world. This improves the data efficiency and training speed of robot skill learning, making the synthesis of complex strategies possible. Compared to the high cost and low efficiency of existing technologies that rely on large amounts of real-world data, this invention significantly reduces the dependence on real physical data. The robot can acquire a high-performance basic strategy before deployment, providing a solid starting point for subsequent online correction and shortening the cycle from robot training to task execution.

[0017] This invention also designs a closed-loop optimization mechanism that integrates online calibration steps and real data feedback to the generation engine. The dual-strategy execution and calibration unit utilizes real-time interactive data to update only the residual strategy through an online reinforcement learning algorithm. The real state transition sequences generated during physical execution are collected and fed back to the generation engine for calibration and optimization. The direct effect of this design is the realization of dual calibration: the residual strategy provides immediate correction of current deviations, while the feedback calibration of the generation engine fundamentally reduces the difference between future virtual trajectories and the real world. Compared to the shortcomings of existing technologies where models become fixed and unchanging once trained, unable to adapt to environmental changes, this invention constructs a continuously learning and self-improving system, enabling the robot to possess lifelong learning capabilities. It can not only adapt to current tasks but also continuously improve the accuracy of its imagination, making it more efficient and precise when learning new skills in the future. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the process of the present invention; Figure 2 This is a system framework diagram of the present invention; Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. These drawings are simplified schematic diagrams, which are only used to illustrate the basic structure of the present invention and therefore only show the components relevant to the present invention.

[0020] The purpose of this invention is to overcome the aforementioned deficiencies of the prior art and provide a method, system, and computer storage medium for robot skill synthesis with residual correction. This invention aims to address the problem of robot skill learning's heavy reliance on known or structured environments and its inability to handle unknown scenarios.

[0021] To achieve the above objectives, the technical solution adopted by the present invention is as follows: like Figure 1 As shown, a method for synthesizing robot skills with residual correction includes the following steps: S1. Acquire sensor data; S2. Encode the sensor data into the current latent state; S3. Randomly select a candidate action of the robot body, input the current latent state and the candidate action into the generation engine, generate the probability distribution of the future latent state at the next moment, so as to form a virtual trajectory. S4. In the virtual trajectory generated by the generation engine, an imaginary strategy is synthesized through reinforcement learning; S5. Compress and transfer the knowledge of the imagination strategy into a lightweight embodied strategy, and deploy the embodied strategy on the robot body. S6. Deploy residual strategies on robot entities. The final execution action of the robot entity is generated by the linear superposition of the embodied strategy and the residual strategy. S7. During the robot's execution of the task, an online correction step is performed, including: keeping the network parameters of the embodied policy frozen; and using real-time interactive data, updating only the residual policy through an online reinforcement learning algorithm to compensate for the execution deviation of the embodied policy in the virtual network and the real operation.

[0022] Each step will be explained in detail below.

[0023] S1. Acquire sensor data; Specifically, the sensing sensor data includes: body sensing sensor data and external sensing sensor data; Body perception sensor data provides intrinsic information about the physical state of the robot body. Body perception sensor data includes at least one of the following: joint encoder data, torque sensor data, and inertial measurement unit data; external perception sensor data includes at least one of the following: camera data and lidar data.

[0024] Joint encoder data: Each physical joint of the robot is equipped with a high-precision joint encoder that provides real-time accurate angle or position readings of all joints of the robot. Through time differentiation, the joint velocity and joint acceleration can be further calculated.

[0025] Torque sensor data: To enable precise interaction with the environment, robots integrate torque sensors. For example, a torque sensor is based on a piezoresistive strain gauge. When a force is applied, the piezoresistive strain gauge deforms, causing a change in resistance. This change is converted into a voltage signal via a Wheatstone bridge, and ultimately digitized into precise force (Fx, Fy, Fz) and torque (Tx, Ty, Tz) readings.

[0026] The inertial measurement unit data, through sensor fusion algorithms, outputs the robot's absolute attitude, such as pitch angle, roll angle, and yaw angle.

[0027] Camera data, including but not limited to: An RGB camera provides 2D color images rich in texture and color information for object recognition and classification.

[0028] A depth camera provides depth information for each pixel, generating an RGB-D (color + depth) image.

[0029] Stereo cameras calculate depth using binocular parallax.

[0030] LiDAR data calculates distance by actively emitting laser beams and measuring their flight time, providing high-precision 3D point cloud data. Its advantage lies in its unaffectedness by changes in ambient lighting, enabling accurate geometric measurements.

[0031] S2. Encode the sensor data into the current latent state; The raw perceptual data stream acquired in step S1 is high-dimensional (e.g., a 640x480 depth image has over 300,000 dimensions), multimodal (e.g., image pixels, torque in Newton-meters, angle in radians, scales, and units vary), and temporally dependent (e.g., the motion state of an object depends not only on the current frame but also on historical frames; whether a contact force is abnormal depends not only on its current value but also on its trend). Directly performing reinforcement learning or generative model training in this high-dimensional, heterogeneous raw data space will result in extremely low learning efficiency and almost no chance of convergence.

[0032] The goal of this step is to compress the raw, high-dimensional, multimodal sensor data into a low-dimensional, information-dense latent state representation, capturing key information useful for decision-making while filtering out irrelevant noise.

[0033] The raw sensor data acquired in step S1 is fed into a mode-specific preprocessing network.

[0034] For example, camera data is processed through a convolutional neural network to extract spatial visual features; LiDAR data is processed through a PointNet network to extract geometric features; and joint encoder data, torque sensor data, and inertial measurement unit data are processed through a multilayer perceptron for feature transformation and dimension alignment. Features from all modalities are concatenated into a unified fusion feature vector.

[0035] The fused feature vector is output to a recurrent variational autoencoder (ROC). The recurrent units within the ROC receive the fused feature vector and the previous time step's hidden state, and compute the current time step's hidden state. The current time step's hidden state captures all relevant historical information up to the current time step (e.g., the object's velocity, acceleration, and the trend of contact force). The ROC's output layer receives the hidden state and outputs the Gaussian distribution parameters of the latent state, namely the mean and log-variance.

[0036] The system uses reparameterization techniques to sample the latent state. : ,in It is noise sampled from a standard normal distribution. For the current time step, Represents the mean. This represents the logarithmic variance.

[0037] This is a low-dimensional, information-dense representation of the current latent state.

[0038] S3. Randomly select a candidate action of the robot body, input the current latent state and the candidate action into the generation engine, generate the probability distribution of the future latent state at the next moment, so as to form a virtual trajectory. Specifically, the generation engine is implemented through two advanced generation models: A learning model based on generative adversarial networks (GANs) consists of a generator whose input is the current latent state and candidate actions, and whose output is a predicted latent state for the next time step.

[0039] The discriminator takes a triple containing state, action, and next state as input and outputs a scalar indicating whether the state transition comes from the robot entity’s real experience or is generated by the generator.

[0040] The training process involves adversarial training between the generator and the discriminator. The discriminator aims to distinguish between genuine and fake transitions as accurately as possible. The generator's goal is to generate the next latent state so that the transition triple formed by the generator, the current latent state, and the candidate action can deceive the discriminator, causing it to mistakenly believe it to be genuine.

[0041] The discriminator acts as a physical plausibility evaluator. Because it is trained on real-world physical interaction data, it will identify a generator as a forgery if the generator produces a physically impossible transition. This adversarial loss forces the generator to learn a model that is physically plausible.

[0042] Inverse denoising process based on diffusion model: Sampling a real transfer ,right Step by step (e.g.) Step 1) Add Gaussian noise to obtain a series of noisy versions. ,in Training a noise prediction network is almost entirely based on noise. The input to this network is a noisy state. Noise steps and conditions The goal of the network is to predict the... Noise added step .

[0043] Given As a condition, a pure noise vector is sampled from a standard normal distribution. .from Iteration to ,use Predict the noise in this step , , and then from Subtracting the predicted noise and adding a small amount of randomness, we get... The final result It's just a basis The generated, physically plausible next dive state.

[0044] By using different initial noises By repeatedly executing the reverse denoising process, the model can naturally generate multiple different but all reasonable noise reduction methods. It realizes the probability distribution of future latent states. The sampling provides a virtual trajectory for step S4.

[0045] S4. In the virtual trajectory generated by the generation engine, an imaginary strategy is synthesized through reinforcement learning; In synthesizing an imaginary policy within a virtual trajectory, a reinforcement learning algorithm is used to train the imaginary policy. The training process is as follows: Training in a virtual environment eliminates the need for the system to interact with the expensive and slow physical world. It can leverage massively parallel computing to generate millions of frames of imagined experiences per second across thousands of parallel generative engine instances, with the sole purpose of finding the optimal solution in the imagination.

[0046] Trajectory generation specifically involves the system simulating the robot's execution process in a virtual network, starting from an initial state. Beginning, in the Step, Strategy Based on the current imagined state Choose an action Generation Engine Predicting the imagined state of the next moment A predefined reward function Evaluate this transfer: For example, repeat Step, generate a virtual trace .

[0047] The invention also includes a step of designating a region of little prior experience of the robot entity as a region of curiosity, and sampling and generating data in the region of curiosity to guide the synthesis of an imagination strategy.

[0048] Identify regions of curiosity and train a system that includes The ensemble of learning models, for the same , A learning model will predict A different .

[0049] Curiosity Zone: The regions predicted by the learning models have high variance. If the variance is small, it means that all the learning models are confident in this dynamic, and this is a known region; if the variance is large, it means that the models are unsure of what will happen.

[0050] Treating uncertainty or variance as an intrinsic reward, the total reward... It becomes: in It's a task reward. It is a curiosity reward (e.g., the variance predicted by the learning model E). It is a hyperparameter.

[0051] By maximizing The imaginative strategy learns how to complete tasks and is guided to actively explore areas of curiosity where the learning model is uncertain. This allows the imaginative strategy to anticipate possible unexpected situations, thereby achieving extremely high robustness and generalization ability.

[0052] S5. Compress and transfer the knowledge of the imagination strategy into a lightweight embodied strategy, and deploy the embodied strategy on the robot body. This involves compressing and transferring knowledge about imaginative strategies, specifically through strategy distillation. Imaginary strategies provide a probability distribution of soft-target actions, while embodied strategies are trained through supervised learning to mimic the probability distribution of soft-target actions provided by imaginary strategies. The output layer of the imagination strategy uses a temperature scaling parameter to smooth the probability distribution of soft target actions.

[0053] This step is implemented using the strategy distillation method, with the roles defined as follows: teacher It is the imagination strategy trained in step S4.

[0054] student It is a lightweight, shallow neural network whose input is the latent state. The output is an action. The forward propagation speed is fast, sufficient to meet real-time control requirements.

[0055] Distillation dataset generation: A distillation dataset is generated within the system by sampling a massive number of latent states from the learned model. For each Using teacher strategies Calculate its teaching labels .

[0056] Teaching tags It is the probability distribution of actions on soft targets, and the teacher's strategy. The decision (e.g., in a certain state, there is a 70% probability of going left and a 30% probability of going forward) contains richer information than just that the best action is to go left. It suggests to students that going left is the best strategy, and going forward is a second-best option, thus helping students develop their strategies. Learn to make the decision boundaries smoother.

[0057] It is a random policy that outputs a probability distribution.

[0058] when If it is based on the Q value, then its Q function can be used. .

[0059] Temperature scaling, use the "temperature scaling parameters" "To smooth the probability distribution."

[0060] in It is temperature, when At that time, it was a soft target. As it approaches 0, the distribution approaches a hard target. In this case, the probability distribution is such that the probability is more evenly distributed to the suboptimal action.

[0061] Use a temperature Distillation can force students To learn the best movements, one must also learn how teachers evaluate suboptimal movements, thus more completely mimicking the teacher's decision-making logic.

[0062] Supervised learning training, using distillation datasets Used for training students .

[0063] Minimize students Output distribution and teacher The difference between them. The commonly used loss function is KL divergence.

[0064] After training, It is deployed on the control system of the robot entity as a basic strategy for performing tasks.

[0065] S6. Deploy residual strategies on robot entities. The final execution action of the robot entity is generated by the linear superposition of the embodied strategy and the residual strategy. Two policy networks are deployed on the robot entity: Embodied Strategy This is the strategy obtained from distillation in step S5.

[0066] Residual strategy It is a similarly lightweight policy network, with the same output dimension as the action space, and can be initialized to zero output or a random small value during deployment.

[0067] In action synthesis, at each control cycle (e.g., every 20 milliseconds), the system obtains the current latent state from step S2. .

[0068] Calculate basic movements:

[0069] Calculate residual actions:

[0070] The final action performed by the robot entity It is generated by the linear superposition of the embodied policy and the residual policy:

[0071] It provides the basic actions to solve 99% of the problems in a task. The task is greatly simplified to: learning only tiny corrections or residuals to this basic movement to compensate for a 1% gap.

[0072] S7. During the robot's execution of a task, an online correction step is performed, including: keeping the network parameters of the embodied policy frozen; and using real-time interactive data, updating only the residual policy through an online reinforcement learning algorithm to compensate for the execution deviation of the embodied policy in the virtual network and real-world operations. Throughout the online execution and learning process, the network parameters of the embodied policy—namely, the weights and biases—remain frozen and unupdated. The embodied policy contains prior knowledge learned from the model's imagination; freezing the network parameters ensures that this prior knowledge is not forgotten during execution.

[0073] Using real-world interaction data, an online reinforcement learning algorithm is used to update only the residual policy. Network parameters.

[0074] The specific learning objective is residual strategy. The learning objective is to maximize the expected total of real rewards obtained in the real world. The task is to learn an optimal correction function. , making To become the optimal action in the real physical world.

[0075] Online reinforcement learning algorithms are used to update The algorithm uses a policy-dependent approach, where the training data consists of continuously changing policies. In the (Collected in the last update). The policy algorithm can learn from data from historical policies.

[0076] Soft actor-critic algorithm: Soft actors, i.e., residual strategy .

[0077] The critic is one or more Q networks .critic Its purpose is to assess the state in the real world. Next, execute the overall action. The expected future reward.

[0078] The invention also includes a step of collecting the real state transition sequence generated during the physical execution of the robot entity and feeding it back to the generation engine, and calibrating and optimizing the generation engine based on the feedback.

[0079] Specifically, during the online calibration process in step S7, the dual-strategy execution and calibration unit will continuously and automatically collect the actual state transition sequences generated during the physical execution of the robot entity. The real state transition sequence data is stored in the real experience replay buffer.

[0080] When enough real-world state transition sequence data accumulates in the real-world experience replay buffer, the real-world state transition sequence data is fed back to the generation engine, which then uses the real-world state transition sequence data for retraining, fine-tuning, or calibration.

[0081] like Figure 2 As shown, the present invention also provides a robot skill synthesis system with residual correction, comprising: Data Acquisition Unit: This unit serves as the interface between the system and the physical sensor array of the robot body 201. It is configured to acquire real-time sensor data from the robot body, providing a high-fidelity raw data source for all subsequent processing and decision-making.

[0082] Latent state encoding unit: This unit is connected to the output of the data acquisition unit 210. Its core function is to perform dimensionality compression and information fusion, which encodes the high-dimensional, multimodal and noisy raw sensor data stream received from the data acquisition unit into a low-dimensional, information-dense current latent state that can characterize the dynamics of the system.

[0083] The generation engine unit is connected to the latent state encoding unit and the policy synthesis unit. Essentially, it is a learned model that operates within the latent state space constructed by the latent state encoding unit. Its function is to receive the current latent state and candidate actions from the policy synthesis unit, and generate the probability distribution of the future latent state at the next time step. Through iterative sampling on this engine, the system can form diverse virtual trajectories in the latent space.

[0084] Policy Synthesis Unit: This unit is tightly coupled with the generation engine unit. Its function is to synthesize a complex and powerful imaginary policy by performing large-scale, parallel policy training through reinforcement learning algorithms in the virtual environment provided by the generation engine.

[0085] Policy distillation unit: This unit connects the output of the policy synthesis unit and the input of the dual policy execution and correction unit. Its function is to solve the problem of high computational cost of embodied policies and inability to run in real time. Through knowledge compression techniques, such as policy distillation, the complex decision knowledge contained in the embodied policy is transferred to a lightweight, feedforward embodied policy.

[0086] Dual-strategy execution and correction unit: This unit is the core execution and adaptation module of this invention. It is deployed on the real-time controller of the robot body. Its function is to simultaneously deploy an embodied strategy and an initialized residual strategy. During the robot body's task execution, this unit maintains the network parameters of the embodied strategy frozen to preserve its generalization knowledge and only updates the residual strategy. Using real-time interactive data from the data acquisition unit, it quickly learns a compensation signal to fill the deviation through an efficient online reinforcement learning algorithm.

[0087] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0088] In the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example that are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example.

[0089] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for synthesizing robot skills with residual correction, characterized in that, Includes the following steps: S1. Acquire sensor data; S2. Encode the sensor data into the current latent state; S3. Randomly select a candidate action of the robot body, input the current latent state and the candidate action into the generation engine, generate the probability distribution of the future latent state at the next moment, so as to form a virtual trajectory. S4. In the virtual trajectory generated by the generation engine, an imaginary strategy is synthesized through reinforcement learning; S5. Compress and transfer the knowledge of the imagination strategy into a lightweight embodied strategy, and deploy the embodied strategy on the robot body. S6. Deploy residual strategies on robot entities. The final execution action of the robot entity is generated by the linear superposition of the embodied strategy and the residual strategy. S7. During the robot's execution of a task, an online correction step is performed, including: keeping the network parameters of the embodied strategy frozen; Furthermore, by utilizing real-time interactive data, online reinforcement learning algorithms are used to update only the residual policy to compensate for the execution deviation of the embodied policy in the virtual network and real-world operations.

2. The robot skill synthesis method and system with residual correction according to claim 1, characterized in that, Also includes: The robot identifies areas with limited prior experience as regions of curiosity and samples data generated within these regions to guide the synthesis of imaginative strategies.

3. The robot skill synthesis method and system with residual correction according to claim 1, characterized in that: In S1, the sensing sensor data includes: body sensing sensor data and external sensing sensor data; The body sensing sensor data includes at least one of the following: joint encoder data, torque sensor data, and inertial measurement unit data; the external sensing sensor data includes at least one of the following: camera data and lidar data.

4. The robot skill synthesis method and system with residual correction according to claim 1, characterized in that: In S2, encoding the sensing sensor data into the current latent state specifically involves compressing the high-dimensional, multimodal raw sensing sensor data into a low-dimensional, information-dense latent state representation.

5. The robot skill synthesis method and system with residual correction according to claim 1, characterized in that: In S3, the generation engine is a learning model based on generative adversarial networks, including: a generator, which receives the current latent state and candidate actions, and outputs the predicted latent state at the next time step; A discriminator is used to assess the physical plausibility of the future potential state; The learning model learns a reverse denoising process to generate a probability distribution of future latent states based on the current latent state and candidate actions.

6. The robot skill synthesis method and system with residual correction according to claim 1, characterized in that: In S5, the knowledge of the imagined strategy is compressed and transferred, specifically through a strategy distillation method, as follows: The imagination strategy provides a soft target action probability distribution, and the embodied strategy is trained through supervised learning to mimic the soft target action probability distribution provided by the imagination strategy. The output layer of the imagined strategy uses a temperature scaling parameter to smooth the probability distribution of the soft target's actions.

7. The robot skill synthesis method and system with residual correction according to claim 1, characterized in that: In S7, the online reinforcement learning algorithm is a policy-agnostic reinforcement learning algorithm, specifically the soft actor-critic algorithm.

8. The robot skill synthesis method and system with residual correction according to claim 1, characterized in that, Also includes: Collect the actual state transition sequences generated during the physical execution of the robot and feed them back to the generation engine. Based on the feedback, calibrate and optimize the generation engine.

9. A robot skill synthesis system with residual correction, characterized in that, include: The data acquisition unit is used to acquire sensor data from the robot itself. A latent state encoding unit is used to encode the sensing sensor data into the current latent state; The generation engine unit is configured to generate the probability distribution of the future latent state at the next moment based on the current latent state and candidate actions, so as to form a variety of virtual trajectories; A policy synthesis unit is used to synthesize an imagined policy in the virtual trajectory through reinforcement learning. The strategy distillation unit is used to compress and transfer the knowledge of the imagined strategy into a lightweight embodied strategy; The system also includes a dual-policy execution and correction unit configured to deploy the embodied policy and a residual policy on the robot entity, wherein the network parameters of the embodied policy are kept frozen during execution and the residual policy is updated only using an online reinforcement learning algorithm with real-time interactive data.

10. The robot skill synthesis method and system with residual correction according to claim 9, characterized in that: The latent state coding unit includes a cyclic variational autoencoder for processing sensing sensor data; The dual-strategy execution and correction unit is configured to update the residual strategy using a soft actor-critic algorithm.

Images