Robot operation skill learning method and device based on sample generation

Abstract

The invention provides a robot operation skill learning method and device based on sample generation, and the method comprises the steps: enabling a robot to interact with an environment, collecting a real sample, and storing the real sample in a real experience playback buffer area; training a self-verification sample generator by using the real sample; the adaptive imagination module dynamically determines a target generation length of an imagination track and a target sampling proportion of an imagination sample in strategy training according to training loss; the generator generates an imaginary trajectory of a target generation length based on the current strategy and the selected initial state; in strategy training iteration, sampling imaginary samples and real samples from the two buffer areas according to a target sampling proportion to jointly form a training batch for strategy training; and repeating all the steps until the adaptive imagination module determines that a preset condition is reached according to the training loss sequence of the self-verification sample generator, and reducing the frequency of interaction with the environment. The robot skill learning efficiency can be improved while actual interaction with the environment is reduced.

Description

Technical Field

[0001] This invention relates to the field of robot operation skill learning technology, and in particular to a robot operation skill learning method and apparatus based on sample generation. Background Technology

[0002] Reinforcement learning has achieved great success in learning robot operational skills, especially in off-policy reinforcement learning, where learning policies from historical samples allows for more robust adaptation to environmental changes. However, off-policy reinforcement learning faces the challenge of low sample efficiency, particularly in sparse reward environments. Sparse rewards refer to situations where robots only receive rewards upon completing a task, lacking intermediate guidance signals. This necessitates extensive interaction with the environment for the robot to learn operational skills, resulting in low sample efficiency. Summary of the Invention

[0003] This invention provides a method and apparatus for learning robot operational skills based on sample generation, to solve the problems of low sample efficiency and high number of interactions with the environment in existing robot reinforcement learning technologies under sparse reward environments. The technical solution proposed by this invention is as follows: In a first aspect, the present invention provides a robot operation skill learning method based on sample generation, comprising: The robot interacts with the environment, collects real samples, and stores them in a real-world experience playback buffer. The self-validating sample generator is trained using the real samples, and the training loss of the self-validating sample generator is obtained; the self-validating sample generator is used to learn environmental dynamics and generate imagined samples. The adaptive imagination module dynamically determines the target generation length of the imagination trajectory and the target sampling ratio of the imagination samples in the policy training based on the training loss. The self-verifying sample generator generates an imagined trajectory of the target generation length based on the current strategy and the selected starting state, and stores it in the imagined experience playback buffer. In the strategy training iteration, imagined samples are sampled from the imagined experience playback buffer at the target sampling ratio, and real samples are sampled from the real experience playback buffer, which together form a training batch for strategy training. Repeat all the above steps until the adaptive imagination module determines that the preset conditions have been met based on the training loss sequence of the self-verifying sample generator, then reduce the frequency of interaction with the environment.

[0004] Optionally, the self-verifying sample generator includes a state visualization module, an action verification module, and a reward visualization module; the real sample includes the current state, the action being performed, the environmental reward, and the next state; The step of training a self-validating sample generator using the real samples and obtaining the training loss of the self-validating sample generator includes: The current state and the action performed in the real sample are input into the state imagination module, the imagined next state is output, and the first loss is determined based on the imagined next state and the next state in the real sample. The imagined next state and the current state in the real sample are input into the action verification module, the predicted action is output, and the second loss is determined based on the predicted action and the executed action in the real sample. The current state in the real sample is input into the reward imagination module, the imagined reward is output, and the third loss is determined based on the imagined reward and the environmental reward in the real sample. The training loss of the self-verifying sample generator is determined based on the first loss, the second loss, and the third loss.

[0005] Optionally, the target generation length of the imagined trajectory is determined as follows: Determine the scaling factor between the training loss and the initial loss of the self-validating sample generator, and determine the target generation length based on the scaling factor and the maximum step size of the task.

[0006] Optionally, the target sampling proportion of imagined samples in policy training is determined in the following way: Determine the ratio of the training loss to the initial loss of the self-validating sample generator. The target sampling ratio is determined based on the maximum value of the set imagined sample sampling ratio, the initial loss, and the scaling factor.

[0007] Optionally, the method for determining the preset conditions is as follows: Calculate the expected value and standard deviation of the training loss of the self-validating sample generator in the same training round; If the expected value is less than the first threshold and the standard deviation is less than the second threshold for multiple consecutive training rounds, it is determined that the time to reduce interaction with the environment has been reached, and the interaction frequency is switched.

[0008] Optionally, the sampling ratio of real samples from the real experience replay buffer is 1 minus the target sampling ratio.

[0009] Secondly, the present invention also provides a robot operation skill learning device based on sample generation, comprising the following modules: The interaction module is used for the robot to interact with the environment, collect real samples and store them in the real experience playback buffer. The training module is used to train a self-validating sample generator using the real samples and obtain the training loss of the self-validating sample generator; the self-validating sample generator is used to learn environmental dynamics and generate imagined samples. The calculation module is used to dynamically determine the target generation length of the imagination trajectory and the target sampling ratio of the imagination samples in the policy training based on the training loss of the adaptive imagination module. The generation module is used by the self-verifying sample generator to generate an imagined trajectory of the target generation length based on the current strategy and the selected starting state, and store it in the imagined experience playback buffer. The sampling module is used to sample imagined samples from the imagined experience playback buffer and sample real samples from the real experience playback buffer at the target sampling ratio during the policy training iteration, and to form a training batch together for policy training. The judgment module is used to repeat all the above steps until the adaptive imagination module determines that the preset conditions have been met based on the training loss sequence of the self-verifying sample generator, and then reduces the frequency of interaction with the environment.

[0010] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the sample-based robot operation skill learning method described in the first aspect above.

[0011] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the sample-based robot operation skill learning method described in the first aspect above.

[0012] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the sample-based robot operation skill learning method described in the first aspect above.

[0013] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows: This invention provides a robot operation skill learning method and apparatus based on sample generation. The robot collects real samples only through interaction with the environment, uses these real samples to train a self-validating sample generator, which learns environmental dynamics and generates imagined samples, reducing interaction with the environment and overcoming dependence on environmental interaction. An adaptive imagination module dynamically determines the target generation length of the imagined trajectory and the target sampling ratio of imagined samples in policy training based on the training loss of the self-validating sample generator. This approach determines key parameters through an adaptive mechanism, avoiding the need for manually designed hyperparameters, making the method more universal and adaptable to different tasks, and solving the problem of existing technologies being limited in application to diverse tasks due to hyperparameter design limitations. During the repetition of the above steps, the adaptive imagination module determines whether a preset condition is met based on the training loss sequence of the self-validating sample generator. When the preset condition is met, the frequency of interaction with the environment is reduced. This means that after meeting certain conditions, the robot can reduce interaction with the environment and rely on generated imagined samples for policy training, thereby actively reducing the number of actual interactions and improving learning efficiency.

[0014] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained through the structures particularly pointed out in the description and the drawings.

[0015] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is one of the flowcharts of the robot operation skill learning method based on sample generation provided by the present invention.

[0018] Figure 2 This is the second flowchart of the robot operation skill learning method based on sample generation provided by the present invention.

[0019] Figure 3 This is a schematic diagram of the structure of the self-verification sample generator provided by the present invention.

[0020] Figure 4 This is a schematic diagram of the adaptive imagination module provided by the present invention.

[0021] Figure 5 This is the third flowchart of the robot operation skill learning method based on sample generation provided by the present invention.

[0022] Figure 6 This is a schematic diagram of a continuous robot control task under four sparse rewards in an embodiment of the present invention.

[0023] Figure 7 This is a schematic diagram of the robot operation skill learning device based on sample generation provided by the present invention.

[0024] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] In existing technologies, sample efficiency is improved by enhancing the target update or value estimation methods of algorithms to encourage exploration. For example, this can be achieved by maximizing the entropy target or adding random noise to the updated target value to incentivize the agent to explore more extensively; improving value estimation methods by incorporating Monte Carlo estimation; or providing the agent with expert data, such as demonstration data, or conducting offline reinforcement learning on pre-collected large-scale datasets. While these methods improve sample efficiency to some extent, they require the introduction of additional demonstration data and carefully designed hyperparameters, limiting their applicability to diverse tasks. Furthermore, these methods still require significant interaction with the environment to complete the learning process and cannot proactively reduce the number of actual interactions.

[0027] Therefore, in the field of robot skill learning, there is an urgent need for a method that can improve sample efficiency and reduce the number of actual interactions with the environment in sparse reward environments, enabling robots to efficiently acquire skills with as little environmental interaction as possible. This invention provides a sample-based robot skill learning method and apparatus that can improve robot skill learning efficiency while reducing actual interactions with the environment.

[0028] The robot operation skill learning method based on sample generation provided by this invention utilizes a self-verifying sample generator to learn from historical samples and continuously generate near-realistic imagined samples for policy learning. Simultaneously, an adaptive imagination module adaptively adjusts the trajectory length and sampling ratio of the imagined samples and determines when to reduce interaction with the environment, thereby significantly improving sample efficiency while ensuring policy convergence. (Refer to...) Figure 1 As shown, the method includes: S110: The robot interacts with the environment, collects real samples, and stores them in a real experience playback buffer.

[0029] The robot begins to physically interact with its environment. During this process, the robot performs various actions, such as grasping objects and moving them to designated locations. Through sensors (such as vision sensors and force sensors), the robot acquires environmental state information, including its own position and orientation, as well as the positions and states of surrounding objects.

[0030] After each interaction, the state, actions, and subsequent states (i.e., state transition information) during the interaction are collected as a real sample. These real samples are stored in the real experience replay buffer. The real experience replay buffer is a data structure used to store and manage these real interaction samples so that they can be randomly sampled from them during subsequent policy training, improving the stability and efficiency of training.

[0031] S120. Use the real samples to train the self-verifying sample generator and obtain the training loss of the self-verifying sample generator; the self-verifying sample generator is used to learn the dynamics of the environment and generate imagined samples.

[0032] A certain number of real samples are drawn from the real-world experience replay buffer and input into the self-validating sample generator for training. During training, the predictions generated by the self-validating sample generator are compared with the real samples, and the difference between the two is calculated; this difference is the training loss. The training loss reflects the accuracy of the self-validating sample generator in dynamically learning the environment; the smaller the loss, the closer the samples generated by the generator are to the state transitions in the environment.

[0033] S130. The adaptive imagination module dynamically determines the target generation length of the imagination trajectory and the target sampling ratio of the imagination samples in the policy training based on the training loss.

[0034] The adaptive imagination module dynamically adjusts the target generation length of the imagination trajectory based on the training loss of the self-validating sample generator. When the training loss is large, it indicates that the self-validating sample generator's learning of environmental dynamics is not accurate enough. In this case, the generation length of the imagination trajectory can be appropriately shortened to avoid generating overly inaccurate imagination samples that affect policy training. Conversely, when the training loss is small, the generation length of the imagination trajectory can be gradually increased to fully utilize the self-validating sample generator's good learning ability of environmental dynamics and generate more valuable imagination samples.

[0035] Meanwhile, the adaptive imagination module also determines the target sampling ratio of imagined samples in policy training based on the training loss. If the training loss is large, it means that real samples are more important for accurately learning the policy. In this case, the sampling ratio of imagined samples is reduced and the sampling ratio of real samples is increased. When the training loss is small, the sampling ratio of imagined samples can be appropriately increased to enrich the training data and improve the generalization ability of the policy.

[0036] S140. The self-verifying sample generator generates an imagined trajectory of the target generation length based on the current strategy and the selected starting state, and stores it in the imagined experience playback buffer.

[0037] Based on the current strategy and the selected starting state (which can be randomly selected from the real-world experience replay buffer or selected according to specific rules), the self-verifying sample generator generates an imagined trajectory according to the previously determined target length. The imagined trajectory consists of a series of state-action-next-state pairs. These samples are dynamically simulated by the self-verifying sample generator based on the learned environment and do not come from real robot-environment interactions.

[0038] The generated imagined trajectories are stored in the imagined experience replay buffer. Similar to the real experience replay buffer, the imagined experience replay buffer is used to store and manage imagined samples, providing data support for subsequent policy training.

[0039] S150. In the policy training iteration, imagined samples are sampled from the imagined experience playback buffer at the target sampling ratio, and real samples are sampled from the real experience playback buffer to form a training batch for policy training.

[0040] During the policy training iteration, imaginary samples are sampled from the imaginary experience playback buffer and real samples are sampled from the real experience playback buffer, according to the target sampling ratio determined by the adaptive imagination module. The real and imaginary samples are then mixed to form a training batch. This training batch is used to train the robot's operational policy. Reinforcement learning algorithms (such as deep deterministic policy gradient, proximal policy optimization, etc.) are employed to update the parameters of the policy network by minimizing the policy loss function, enabling the robot to choose better actions in a given state to maximize long-term cumulative rewards. Interacting with the environment typically consumes significant resources; using imaginary samples generated by the adaptive imagination module for training can significantly reduce the number of interactions with the environment, thereby reducing resource consumption and costs.

[0041] S160. Repeat all the above steps until the adaptive imagination module determines that the preset conditions have been met based on the training loss sequence of the self-verifying sample generator, then reduce the frequency of interaction with the environment.

[0042] After completing one policy training iteration, the robot interacts with the environment again, repeating steps S110-S150. This involves continuously collecting real samples, training the self-validating sample generator, adjusting the parameters of the adaptive imagination module, generating imagined trajectories, and training the policy, forming a cyclical iterative learning process. All steps S110-S150 are repeated until the loop termination condition is met. Among them, reference Figure 2 As shown, the loop termination condition is one of the following conditions: (a) The number of training steps reaches the maximum training stride length; or, (b) The adaptive imagination module determines that the preset conditions have been met. At this time, it reduces the frequency of interaction with the environment and continues to loop until the maximum training step size is reached.

[0043] Before each repeated loop, it also includes: Determine whether the time for strategy evaluation has arrived; If the time for strategy evaluation is reached, then strategy evaluation will be conducted.

[0044] This invention enables the adaptive imagination module to continuously generate diverse imagination samples by repeatedly executing steps such as generating imagined samples, training strategies, and evaluating strategies. These samples can simulate various possible environmental states and action outcomes, providing a rich data source for policy learning. Compared to relying solely on environmental interactions to obtain data, the generation of imagined samples is not limited by the environment and can generate a large amount of data in a short time, thereby accelerating the policy learning process and enabling robots to master operational skills more quickly.

[0045] The adaptive imagination module generates imagined samples covering a wide range of possible scenarios in the environment. By training on these diverse samples, the policy learns more general features and patterns. This enables the robot to make reasonable decisions even when faced with similar situations it has never encountered before, improving the policy's generalization ability. During the iteration process, the generation of imagined samples encourages the policy to explore states that are difficult to reach or have not yet been explored in the environment. By training on these unknown states, the policy can continuously expand its applicability, enhance its adaptability to complex environments, and further improve its generalization performance.

[0046] When the adaptive imagination module determines that preset conditions have been met, it reduces the frequency of interaction with the environment and continues the loop. This gradual reduction in real interaction avoids instability caused by frequent environmental interaction, such as noise and interference. Simultaneously, continuing training with imagined samples ensures the policy's continuous learning and optimization, guaranteeing the stability of the learning process. Before each repeated loop, it is determined whether the policy evaluation time has arrived; if so, evaluation is performed. Regularly evaluating the policy's performance allows for the timely detection of problems during the learning process, such as overfitting and underfitting.

[0047] The adaptive decision mechanism in the loop termination condition enables the learning process to dynamically adjust according to the actual situation. When the environment changes or policy learning encounters difficulties, the adaptive imagination module can adjust the generation strategy of imagined samples in a timely manner according to preset conditions, providing more targeted data for policy learning and enabling the robot to better adapt to changes in complex and dynamic environments. By continuously repeating the loop steps, the robot can continuously learn and optimize its operating strategies. Even if the environment is constantly changing, the robot can continuously update and improve its strategies through interaction with the imagined environment and a small number of environments, maintaining its adaptability and competitiveness.

[0048] Existing techniques for encouraging exploration and improving sample efficiency, such as entropy maximization and adding random noise to update target values, often require carefully designed hyperparameters. In this invention, the adaptive imagination module dynamically determines the target generation length of the imagination trajectory and the target sampling ratio of the imagination samples during policy training based on the training loss of the self-validating sample generator. This approach determines key parameters through an adaptive mechanism, avoiding the need for manually designed hyperparameters. This makes the method more versatile and adaptable across different tasks, solving the problem of existing techniques being limited in application to diverse tasks due to hyperparameter design constraints.

[0049] While existing technologies improve sample efficiency to some extent, they still require significant interaction with the environment to complete learning and cannot proactively reduce the number of actual interactions. In this invention, during the repetition of the above steps, the adaptive imagination module determines whether a preset condition has been met based on the training loss sequence of the self-verifying sample generator. When the preset condition is met, the frequency of interaction with the environment is reduced. This means that, under certain conditions, the robot can reduce its interaction with the environment and rely on generated imagined samples for policy training, thereby proactively reducing the number of actual interactions and improving learning efficiency.

[0050] In one embodiment, the self-verifying sample generator includes a state imagination module, an action verification module, and a reward imagination module. Through the collaborative work of these three modules, the self-verifying sample generator can learn environmental dynamics, verify the rationality of imagined states, and accurately imagine environmental rewards, thereby generating high-quality imagined samples and providing rich data support for the robot's strategy training. The real samples include the current state. Execution of actions Environmental rewards and the next state .

[0051] An environmental dynamic model is a mathematical abstraction and simulation of the laws governing environmental operation. It is used to simulate and predict the state changes of a robot during interaction with its environment. It aims to capture the intrinsic relationships between states, actions, and rewards in the environment, enabling the prediction of the next possible state and corresponding reward given the current state and the action performed. By learning from the environmental dynamic model, a self-validating sample generator can break free from its dependence on a large number of environmental interactions, internally generating imagined samples and providing rich data support for policy learning. The learning method is reinforcement learning. In this approach, the agent interacts with the environment, continuously tries different actions, and adjusts the model parameters based on the rewards obtained.

[0052] The components of an environmental dynamics model include state, action, and reward. State is a comprehensive description of the environment at a given moment, containing all relevant information influencing the robot's decisions. For example, state can include the robot's position and orientation. Action is the operation the robot can perform in the environment. Different tasks have different action spaces; for example, in a robot arm control task, an action could be the rotation angle of a joint. The environmental dynamics model needs to be able to handle various possible actions and predict the next state based on the current state and the performed action. Reward is a feedback signal from the environment to the robot's actions, used to guide the robot in learning the optimal policy. Rewards are usually defined according to the task's objective; for example, in a robot grasping task, successfully grasping an object yields a high positive reward, while colliding with an obstacle may result in a negative reward. The environmental dynamics model needs to be able to predict the possible reward given the current state and the performed action to provide an accurate optimization objective for policy learning. The self-validating sample generator uses the learned environmental dynamics model to predict the next state and reward based on the current state and randomly or policy-generated actions, thereby generating imagined samples. These imagined samples share similar statistical properties with real samples and can be used to expand the training dataset, improving the generalization ability of policy learning. For example, in reinforcement learning, generating a large number of imagined samples allows agents to explore and experiment more in virtual environments, accelerating the learning process.

[0053] The above-described S120 method of training a self-validating sample generator using the real samples and obtaining the training loss of the self-validating sample generator includes: S1201. Input the current state and the action performed in the real sample into the state imagination module, output the imagined next state, and determine the first loss based on the imagined next state and the next state in the real sample.

[0054] The state visualization module is a neural network whose input is the current state. and perform actions The output is the imagined next state. Its main purpose is to learn a dynamic model of the environment, that is, to predict the next possible state based on the current state and the actions performed. This is achieved by minimizing the predicted state (i.e., the imagined next state mentioned above). ) and the true state (i.e., the next state in the true sample) By learning from the differences between the network parameters, we can improve the accuracy of predicting environmental state transitions by continuously optimizing the network parameters.

[0055] First loss The loss function is designed as follows: in, This represents the L2 norm.

[0056] S1202. Input the imagined next state and the current state in the real sample into the action verification module, output the predicted action, and determine the second loss based on the predicted action and the executed action in the real sample.

[0057] The action verification module is a neural network, with the current state as its input. And the imagined next state The output is the predicted action. This module's function is to verify the rationality of the imagined states generated by the state imagination module by minimizing the predicted actions. With the execution of actions The differences between them are used to ensure that imagined states can be used to deduce actions that are similar to real situations, thereby improving the credibility of imagined samples.

[0058] Second loss Its loss function is designed as follows: S1203. Input the current state in the real sample into the reward imagination module, output the imagined reward, and determine the third loss based on the imagined reward and the environmental reward in the real sample.

[0059] The reward imagination module is a neural network that takes the current state as its input. The output is an imagined reward. Simultaneously, a reward filtering mechanism ensures the accuracy of imagined rewards. This is achieved by minimizing imagined rewards. With real rewards The differences between them enable the module to learn the reward mechanism given by the environment in different states, providing accurate reward signals for policy training.

[0060] Third loss Its loss function is designed as follows: S1204. Determine the training loss of the self-verifying sample generator based on the first loss, the second loss, and the third loss.

[0061] Loss function of self-validating sample generator The loss functions of the state imagination module, action verification module, and reward imagination module are jointly composed. By minimizing the training loss, the parameters of the three modules can be optimized simultaneously, enabling the state imagination module to generate more accurate imagined states, the action verification module to ensure the rationality of the imagined states, and the reward imagination module to provide more accurate reward signals. This collaborative optimization approach improves the overall performance of the self-verification sample generator, thereby generating high-quality imagined samples and providing strong support for the robot's policy training.

[0062] The first loss, second loss, and third loss are combined by weighted summation or direct addition to obtain the training loss of the self-validating sample generator.

[0063] This invention inputs the current state and action from a real sample into a state imagination module, and determines the first loss based on the imagined next state and the next state in the real sample. This process enables the state imagination module to continuously learn the rules of state transitions in the environment. As training progresses, the module can more accurately predict the next state based on the current state and action, making the generated imagined states closer to the real situation, thus providing high-quality state samples for subsequent policy learning and verification. This invention inputs the imagined next state and the current state from a real sample into an action verification module, outputs a predicted action, and determines the second loss based on the predicted action and the action executed in the real sample. This enables the action verification module to learn the ability to take appropriate actions in different states. After training, the module can more reliably predict reasonable actions based on the input state information, ensuring that the generated action samples are consistent with the action logic in the environment.

[0064] This invention inputs the current state from a real sample into a reward imagination module, outputs an imagined reward, and determines a third loss based on the imagined reward and the environmental reward in the real sample. The reward imagination module learns the reward allocation mechanism in the environment by continuously adjusting parameters to minimize the third loss. When generating imagined rewards subsequently, it can more reasonably estimate reward values ​​under different states, providing accurate reward signals for policy learning and improving the quality of reward samples. The first, second, and third losses constrain and optimize the self-validating sample generator from the perspectives of state, action, and reward, respectively. By continuously reducing these losses, the samples generated by the generator can better guide policy learning in the right direction. For example, accurate state imagination and action prediction help the policy understand the relationship between states and actions, and reasonable reward estimation provides a clear optimization objective for the policy, enabling it to converge to the optimal solution more quickly.

[0065] After training, the self-validating sample generator can generate high-quality imagined samples, including imagined next states, predicted actions, and imagined rewards. These imagined samples complement real samples, providing a rich data source for policy learning. Policies can be trained on more samples, learning more comprehensive knowledge and skills, thereby improving policy performance and effectiveness. Because the self-validating sample generator can generate high-quality imagined samples, policy learning can rely on these imagined samples for training to a certain extent, without requiring excessive interaction with the environment. By making reasonable use of imagined samples, the number of times the robot explores and experiments in the environment can be reduced, lowering the costs and risks associated with real interaction, such as equipment wear and tear and time consumption. Moreover, the self-validating sample generator, trained on real samples, learns general rules and features of the environment. When generating imagined samples, these rules and features are applied to different combinations of states and actions, giving the generated samples a certain degree of generalization. When the policy is trained using these imagined samples, it can also learn more general knowledge and skills, thus better adapting and coping with unseen states and tasks, improving the policy's generalization ability.

[0066] By comprehensively considering the first, second, and third losses to determine the training loss of the self-validating sample generator, the generator can be optimized from multiple aspects, avoiding training instability caused by a single loss. For example, focusing only on the loss of state imagination may lead to inaccurate action prediction and reward estimation, thus affecting the stability of policy learning. The comprehensive loss function can balance the optimization objectives of various aspects, making the training process more stable and improving the generator's performance.

[0067] Figure 3 This invention illustrates a self-verifying sample generator for generating high-quality imagined samples. In a specific implementation, the state imagination module, action verification module, and reward imagination module all employ a three-layer fully connected neural network with 256 hidden units per layer, trained using the Adam optimizer. The state imagination module learns a dynamic model of the environment, the action verification module ensures the rationality of the imagined state through an action verification mechanism, and the reward imagination module simulates environmental reward feedback. In a sparse reward environment setting, the robot receives a reward of -1 when the task is not completed and a reward of 0 when the task is completed. To simulate this realistic reward feedback, the reward imagination module predicts the imagined reward. Then, reward filtering is performed. When When the value is within the range [-0.05, 0.05], it indicates that the task is complete. =0. When If the value is within the range [-1.05, -0.95], it indicates that the task has failed. = 1. If In other intervals, it means that the output of the reward imagination module is uncertain. The sample generator will skip this round of imagination and not generate any imagination samples, thereby ensuring the quality of the imagination samples to ensure stable learning.

[0068] In one embodiment, the adaptive imagination module adaptively adjusts the generated length of the imagination trajectory based on the training loss of the self-verifying sample generator, generating an imagination trajectory of the most suitable length to provide imagination samples for policy training. The target generated length of the imagination trajectory is determined as follows: S1301. Determine the ratio coefficient between the training loss and the initial loss of the self-verifying sample generator, and determine the target generation length based on the ratio coefficient and the maximum step size of the task.

[0069] First, obtain the initial loss. Before training the self-validating sample generator, it needs to be initialized. After initialization, perform a forward propagation and loss calculation on the generator using an initial set of real samples (which can be a small number of pre-collected samples) to obtain the initial loss value. This initial loss reflects the generator's performance before any training; it represents the degree of difference between the generated samples and real samples in the initial state.

[0070] During training, as the self-validating sample generator is iteratively trained using real samples, the current training loss is calculated in each iteration. The training loss is calculated by comparing the imagined samples generated by the generator with real samples, based on a predefined loss function (i.e., the combination of the first, second, and third losses mentioned earlier). It reflects the accuracy and quality of the samples generated by the generator in the current training phase. Dividing the current training loss by the initial loss yields a ratio coefficient between the training loss and the initial loss. This ratio coefficient is a relative value that intuitively represents the degree of performance improvement of the generator during training. A smaller ratio coefficient indicates a significant reduction in the current training loss relative to the initial loss, suggesting that the generator has learned useful information during training and its performance has improved; conversely, a larger ratio coefficient indicates that the generator's performance improvement is not significant or that problems have occurred during training.

[0071] The maximum step size for a task refers to the maximum number of interactions or state transitions a robot can take to complete a specific task. It is determined by the nature and complexity of the task. For example, in a simple robot grasping task, the maximum step size might be the total number of steps the robot takes to move from its initial position to the object's position, grasp the object, and place it at the target location; while in a complex maze exploration task, the maximum step size might be the maximum number of steps the robot can take in the maze.

[0072] A mapping function between the scaling factor and the target generation length can be preset based on the actual situation. This mapping function can be linear or non-linear, and its specific form needs to be adjusted according to the characteristics of the task and experimental results. For example, a linear function can be used: Target generation length = Maximum step size of the task × Adjustment factor of the scaling factor. Here, the adjustment factor is an empirical value used to amplify or reduce the influence of the scaling factor according to the actual situation. Substituting the calculated scaling factor into the preset mapping function, the target generation length of the imagined trajectory is calculated. This target generation length represents the length of the imagined trajectory generated by the self-validating sample generator in the current training phase. It can be dynamically adjusted according to the training progress of the generator; as training progresses, the scaling factor gradually decreases, and the target generation length will change accordingly.

[0073] As a specific example, the target generation length can be calculated using the following formula: In the formula, Generate length for the target. The maximum step size of the task. For training loss, The initial loss for the self-validating sample generator. This is the ratio of training loss to initial loss.

[0074] This invention determines the target generation length based on the ratio of training loss to initial loss, allowing the self-validating sample generator to dynamically adjust the length of the generated imagined trajectories according to its training progress. In the early stages of training, the generator's performance is poor, the training loss is high, the ratio is large, and the target generation length may be short. This avoids generating too many low-quality imagined samples, reducing ineffective computation and resource waste. As training progresses, the generator's performance gradually improves, the training loss decreases, the ratio decreases, and the target generation length increases accordingly. The generator can generate longer imagined trajectories, providing more sample information for policy learning and promoting policy optimization. This dynamic adjustment mechanism allows the generator to focus on generating imagined trajectories of different lengths at different training stages, avoiding the problems that may arise from a fixed-length generation method. For example, if a long target generation length is always used, the generator may not be able to generate high-quality long trajectory samples in the early stages of training, leading to low training efficiency; conversely, if a short target generation length is always used, the generator's capabilities may not be fully utilized in the later stages of training, limiting the effectiveness of policy learning. Dynamically adjusting the target generation length can be optimized according to the actual training situation, improving overall training efficiency.

[0075] Longer imagined trajectories offer more combinations of states and actions, enabling the policy to explore a wider range of possibilities. By dynamically increasing the target generation length based on training progress, the generator can produce longer imagined trajectories in the later stages of training, helping the policy discover more potential states and actions and explore better strategies. For example, in a complex robot navigation task, a longer imagined trajectory allows the robot to consider obstacles and target locations at greater distances, thus planning a better path. Shorter target generation lengths focus more on local exploration and utilization of the current state. In the early stages of training, the shorter imagined trajectories generated by the generator can help the policy quickly learn some basic action and state transition rules, using existing knowledge for preliminary policy optimization. As training progresses, gradually increasing the target generation length allows for deeper exploration based on existing knowledge, achieving a balance between exploration and utilization.

[0076] The maximum step size of a task reflects its complexity. By combining a scaling factor with the maximum step size to determine the target generation length, the generator can adaptively adjust to different task complexities. For complex tasks, a longer maximum step size requires the generator to produce longer imagined trajectories to fully cover the task's state space; conversely, for simple tasks, a shorter maximum step size allows the generator to produce shorter imagined trajectories to meet the needs of policy learning. This adaptability makes this technique widely applicable to various types of robot manipulation and decision-making tasks. Different tasks have different characteristics and requirements. By dynamically adjusting the target generation length, the generator can produce imagined samples more suitable for the current task, enabling policy learning to better adapt to the environments and rules of different tasks. This helps improve the generalization ability of the policy, allowing the trained policy to quickly adapt and exhibit good performance when facing new or similar tasks.

[0077] In one embodiment, the target sampling ratio of imagined samples in policy training is determined in the following manner: S1302. Determine the ratio coefficient between the training loss and the initial loss of the self-validating sample generator, and determine the target sampling ratio based on the maximum value of the set imaginary sample sampling ratio, the initial loss, and the ratio coefficient.

[0078] Before the self-validating sample generator can be used to generate samples, it needs to be initialized. After initialization, a set of pre-prepared, representative environmental interaction data is used to perform a forward propagation calculation to obtain the initial loss. This initial loss reflects the generator's performance level without any training optimization; it embodies the degree of difference between the generator's generated samples and real samples in the initial state and serves as an important benchmark for measuring subsequent training effectiveness. During the training process of the self-validating sample generator, it is iteratively updated using environmental data, and the current training loss is calculated in each iteration. The training loss is calculated by comparing the imagined samples generated by the generator with real samples, based on a predefined loss function (i.e., the one mentioned above). The training loss is calculated using the following method. It intuitively reflects the quality and accuracy of the samples generated by the generator in the current training phase. The smaller the training loss, the closer the generated samples are to the real samples. Dividing the current training loss by the initial loss yields the ratio of training loss to initial loss. This ratio is a relative value that clearly shows the performance improvement of the generator during training. If the ratio is less than 1, it indicates that the current training loss is less than the initial loss, and the generator's performance has improved; the smaller the ratio, the more significant the performance improvement. Conversely, if the ratio is greater than 1, it means that the generator's performance may not have improved, or may even have degraded.

[0079] Based on the nature, complexity, and training resources of the specific task, a maximum value for the imagined sample sampling ratio is pre-set. This maximum value represents the highest proportion of imagined samples among all training samples during policy training. For example, in tasks with high real-time requirements and limited computing resources, the maximum value might be set to 0.5, meaning imagined samples account for at most half of the training samples; while in tasks with high complexity and requiring extensive exploration, the maximum value might be set higher, such as 0.8. A mapping function from the scaling factor to the target sampling ratio is constructed. This mapping function can be designed according to the actual situation, commonly using linear or non-linear mapping. For example, when using a linear mapping, the target sampling ratio can be set as: Target Sampling Ratio = Maximum Value of Imagined Sample Sampling Ratio × (1... (Proportion coefficient × adjustment factor), where the adjustment factor is an empirical value used to fine-tune the influence of the proportion coefficient to ensure that the target sampling ratio is within a reasonable range.

[0080] The calculated scaling factor is substituted into a preset mapping function to calculate the target sampling ratio of the imagined samples during policy training. This target sampling ratio is dynamically adjusted according to the generator's training progress. In the early stages of training, the generator's performance is poor, the scaling factor is large, and the target sampling ratio may be low. As training progresses, the generator's performance improves, the scaling factor decreases, and the target sampling ratio will gradually increase, but it will not exceed the preset maximum value.

[0081] As a specific example, the target sampling ratio is determined by the following formula: in, For the target sampling ratio, The maximum value of the set imaginary sample sampling ratio. The initial loss for the self-validating sample generator. This is the ratio of training loss to initial loss.

[0082] In the early stages of training, the imagined samples generated by the self-validating sample generator may be of low quality and differ significantly from real samples. At this point, a large scaling factor keeps the target sampling ratio low, ensuring that policy training relies primarily on real samples. This guarantees the stability and accuracy of training and prevents policy learning from deviating from the correct direction due to the use of too many low-quality imagined samples. As training progresses, generator performance improves, the quality of imagined samples increases, the scaling factor decreases, and the target sampling ratio gradually increases. The policy can then fully utilize imagined samples for broader exploration, discovering more potential optimal policies and thus improving the policy's generalization ability. This invention dynamically adjusts the target sampling ratio to rationally allocate computational resources according to the actual training needs. When the quality of imagined samples is low, reducing their usage saves computational resources and avoids unnecessary computational waste; conversely, when the quality of imagined samples is high, increasing their usage fully utilizes the generator's capabilities, improves training efficiency, and makes policy learning more efficient.

[0083] If a consistently high proportion of imagined samples is used for training, the policy may overfit the imagined data generated by the generator, failing to adapt well to the environment. This invention dynamically adjusts the target sampling ratio based on a scaling factor. By limiting the use of imagined samples in the early stages of training, it prevents the policy from prematurely overfitting to imagined data, ensuring the policy's generalization performance in the environment. In the later stages of training, when the generator can generate high-quality imagined samples, appropriately increasing the sampling ratio of imagined samples can provide more information for policy learning, helping the policy find the optimal solution faster and accelerating the convergence of the training process. Simultaneously, dynamically adjusting the sampling ratio avoids the training getting stuck in local optima due to a fixed sampling ratio, improving the stability and convergence of policy training.

[0084] In practical applications, the environment may change, leading to alterations in the distribution of real samples. This invention dynamically adjusts the target sampling ratio, enabling policy training to better adapt to environmental changes. When the environment changes significantly, the imagined samples generated by the generator may not match the new environment. In this case, reducing the target sampling ratio and increasing the use of real samples helps the policy quickly adapt to the new environment. When the environment is relatively stable, the sampling ratio of imagined samples can be gradually increased to improve the policy's exploratory capabilities.

[0085] In one embodiment, the sampling ratio for sampling real samples from the real experience playback buffer is 1 minus the target sampling ratio. The adaptive imagination module, based on... Adaptively adjust the sampling ratio of imagined samples, using the target sampling ratio during policy training. Sampling imagined samples, in ( Sampling real samples.

[0086] When the target sampling ratio is set reasonably, the actual sample sampling ratio can be set to 1. The target sampling ratio balances the impact of real and imagined samples on model training. Imagined samples are generated based on a dynamic environmental model; while they can be produced quickly and in large quantities, they may deviate from the actual environment. Real samples, on the other hand, come directly from interactions with the environment and accurately reflect real-world situations. By setting this sampling ratio, the model can leverage the efficiency of imagined samples for rapid learning while using the accuracy of real samples to correct for biases introduced by imagined samples. This makes the training process more stable and prevents performance fluctuations caused by over-reliance on any one type of sample.

[0087] If the target sampling ratio is too high, meaning too many imagined samples are used, the model may gradually deviate from the data distribution of the environment. This is because imagined samples are generated based on model assumptions and may not cover all the details and complexities of the environment. Setting the sampling ratio of real samples to 1... The target sampling ratio ensures that a certain proportion of real samples participate in training, which can promptly pull the model back to a trajectory close to the real distribution, preventing the model from learning features and patterns that are inconsistent with the environment, thereby improving the stability and reliability of training.

[0088] Figure 4 The adaptive imagination module of this invention is illustrated, responsible for adaptively adjusting the use of imagined samples and reducing interaction with the environment. This is based on the training loss of the self-validating sample generator. Adaptively adjust the target generation length of the imagined trajectory Imagine the target sampling ratio of the sample And the timing for starting to reduce interaction with the environment Specifically, in the early stages of training, the prediction accuracy of the self-validating sample generator is low, and the adaptive imagination module generates shorter imagination trajectories and uses fewer imagination samples. As training progresses, the prediction accuracy of the self-validating sample generator improves, and the adaptive imagination module gradually increases the length of the imagination trajectory and the sampling ratio of imagination samples, while reducing actual interaction with the environment at appropriate times.

[0089] Figure 5 The flowchart of this sample-based robot operation skill learning method is illustrated. The robot interacts with the environment through a policy network, storing real samples in a replay buffer. A self-validating sample generator uses real samples to train and generate imagined samples. An adaptive imagination module controls the use of imagined samples, ensuring high-quality samples for the policy network's learning while reducing actual interactions with the environment at appropriate times.

[0090] Specifically, refer to Figure 5 As shown, the adaptive imagination module is based on Adjust the target generation length of the imagined trajectory and the target sampling ratio of the imagined sample From the replay buffer (i.e., the aforementioned real-world experience replay buffer). A random initial state is selected, and a self-validating sample generator is used to predict... and ,in Provided by the policy network, it gradually generates lengths of... The imagined trajectory is stored in the imagined replay buffer (i.e., the aforementioned imagined experience replay buffer). During policy training, The proportion from Sampling imagined samples, in ( The proportion of ) from Real samples are sampled and combined to form training batches for training. Figure 5 In This represents each random moment in the process. Simultaneously, the adaptive imagination module... Determine when to start reducing interaction. When arriving Then, the robot began to operate at a lower frequency. It interacts with the environment, thereby significantly reducing the number of interactions with the environment and improving sample efficiency. This is the initial interaction frequency.

[0091] To verify the effectiveness of this method, Figure 6 Four types of continuous robot control tasks under sparse rewards are shown. Figure 6 In the middle, (a), (b), (c), and (d) represent picking up, opening the door, pulling out, and pushing, respectively.

[0092] In the "Pick Up" task, the robot is placed in front of a table with a randomly placed block on it. The task requires the robot to lift the block to a specified height. Failure to lift the block will result in a reward of -1, while successfully lifting the block will result in a reward of 0.

[0093] In the "Open the Door" task, the robot is positioned on the floor, and the door's location on the table is randomly generated. The task requires the robot to turn the doorknob to open the door. Failure to open the door will result in a reward of -1, while successfully opening the door will result in a reward of 0.

[0094] In the "Draw" task, the robot is positioned in front of a table with three cubes randomly placed on it. The task requires the robot to draw a specific red cube from the three cubes. Failure to draw the red cube results in a reward of -1 at each time step, while successfully drawing the red cube results in a reward of 0.

[0095] In the "Push" task, the robot is placed in front of a table with three blocks randomly placed on it. The task requires the robot to push the blocks forward in sequence. Each successful push increases the reward by 1. Failing to push any block results in a -3 reward, while completing the task yields a reward of 0.

[0096] This invention constructs a self-verifying sample generator comprising a state imagination module, an action verification module, and a reward imagination module. The state imagination module learns a dynamic model of the environment, the action verification module ensures the rationality of the imagined state through an action verification mechanism, and the reward imagination module simulates environmental reward feedback. An adaptive imagination module is constructed, which adaptively adjusts the generated length of the imagined trajectory, the sampling ratio of the imagined samples, and the timing of reducing interaction with the environment based on the prediction accuracy of the self-verifying sample generator. Simultaneously, the self-verifying sample generator generates imagined trajectories. During policy training, the self-verifying sample generator uses real samples to train and generate imagined samples for the policy network. The adaptive imagination module controls the length and sampling ratio of the imagined trajectory to ensure high-quality samples for policy learning, while reducing actual interaction with the environment at appropriate times. By constructing a self-verifying sample generator, this invention provides the robot with high-quality imagined samples as experience supplementation, improving the sample efficiency of continuous robot control tasks under sparse rewards and proactively and significantly reducing the robot's actual interaction with the environment.

[0097] In one embodiment, the method for determining the preset condition in S160 above is as follows: S1601. Calculate the expected value and standard deviation of the training loss of the self-validating sample generator in the same training round.

[0098] S1602. If the expected value of multiple consecutive training rounds is less than the first threshold and the standard deviation is less than the second threshold, it is determined that the time to reduce interaction with the environment has been reached, and the interaction frequency switching is triggered.

[0099] Set a number of consecutive training epochs. This number can be determined by taking into account factors such as training stability, computational resources, and model convergence speed. For example, 5-10 epochs can be chosen.

[0100] The self-validating sample generator is trained using real samples and calculates the training loss for each sample using a predefined loss function (such as mean squared error, cross-entropy loss, etc.). The expected value (i.e., the mean) is an important metric for measuring the average level of the training loss. Based on the training losses of all collected samples, their expected values ​​are calculated. The smaller the expected value, the closer the self-validating sample generator produces samples that are generally closer to the real samples, meaning the generator performs better. The training loss is calculated based on all collected samples. Calculate the standard deviation The standard deviation is used to measure the volatility of training loss. It reflects the stability of generator performance. The smaller the standard deviation, the less volatile the training loss, and the more stable the generator performance.

[0101] Two thresholds are preset, the first threshold Second threshold The first threshold is used to determine if the expected value is small enough, i.e., whether the generator's average performance reaches a high level; the second threshold is used to determine if the standard deviation is small enough, i.e., whether the generator's performance is stable enough. The setting of these two thresholds needs to be determined based on the specific task, model complexity, and experience. The calculated expected value and standard deviation are compared with the first and second thresholds, respectively. If multiple consecutive rounds... and If the interaction with the environment is reduced, the interaction frequency is switched. This means that the self-verifying sample generator has achieved good performance and stability, and can reduce interaction with the environment, relying more on the imagined samples generated by the generator for policy training.

[0102] The adaptive imagination module is based on Expected value and standard deviation Determine when to begin reducing interaction with the environment. When the training stride length reaches At that time, the robot began to reduce its interaction with the environment, among which for: in, It's a point in time. It is an index within the time window. It is a set of integers. `min` represents the minimum value, indicating the search for and determination of the earliest time point. t . K It is the number of consecutive rounds that satisfy the condition, that is, when K consecutive rounds... When both the expected value and standard deviation meet the requirements, interaction with the environment begins to decrease.

[0103] This invention measures the overall quality of samples generated by the generator in each training epoch by calculating the expected value of the training loss of the self-validating sample generator. A smaller expected value indicates a higher degree of fit between the generated imaginary samples and the environmental dynamics, meaning better sample quality. By ensuring that the expected value remains below a first threshold for multiple consecutive epochs, the generator can consistently and stably generate high-quality imaginary samples over a period of time, providing a reliable data foundation for reducing subsequent interactions with the environment. The standard deviation measures the dispersion of the training loss, reflecting the fluctuation in the quality of samples generated by the generator under different states. A smaller standard deviation indicates more stable sample quality and less fluctuation. When the standard deviation remains below a second threshold for multiple consecutive epochs, it indicates that the generator has entered a relatively stable operating state, capable of consistently and reliably generating imaginary samples that conform to environmental dynamics.

[0104] Deciding to reduce interaction with the environment based solely on the training loss of a single round may lead to erroneous decisions due to unforeseen factors (such as data anomalies in a particular round). This invention effectively avoids this unpredictability by requiring multiple consecutive rounds to meet the expected value and standard deviation. It ensures that the interaction frequency switch is only triggered when the generator truly reaches a stable and high-quality generation state, thus preventing a decline in policy training quality due to premature reduction in interaction. Reducing interaction with the environment while ensuring the generator can stably generate high-quality imagined samples allows for continued policy training using these imagined samples, maintaining training continuity and stability, while also reducing dependence on the environment and minimizing the costs and risks associated with actual interaction (such as equipment wear and tear, and time consumption). This reasonable switching mechanism effectively improves the efficiency of the entire robot's operational skill learning process while ensuring the effectiveness of policy training.

[0105] Different robot operation tasks may have varying environmental complexities and dynamic characteristics. This pre-defined condition judgment method does not rely on a specific task scenario but is based on the general metric of the training loss of the self-validating sample generator. Therefore, it is applicable to various types of robot operation skill learning tasks, exhibiting strong adaptability and versatility. In practical applications, the environment may change, such as alterations in the position or state of objects. By continuously monitoring the expected value and standard deviation of the training loss, the system can promptly detect changes in the quality of the generator-generated samples. If environmental changes lead to a decrease in the quality of generated samples, the expected value and standard deviation of the training loss may exceed a threshold range. In this case, the system will not trigger an interaction frequency switch but will continue to interact with the environment to ensure that the policy training can adapt to environmental changes and improve the system's robustness.

[0106] The robot operation skill learning device based on sample generation provided by the present invention is described below. The robot operation skill learning device based on sample generation described below and the robot operation skill learning method based on sample generation described above can be referred to in correspondence.

[0107] The robot operation skill learning device based on sample generation provided by this invention refers to... Figure 7 As shown, it includes: The interaction module 310 is used for the robot to interact with the environment, collect real samples and store them in the real experience playback buffer. Training module 320 is used to train a self-validating sample generator using the real samples and obtain the training loss of the self-validating sample generator; the self-validating sample generator is used to learn environmental dynamics and generate imagined samples. The calculation module 330 is used to dynamically determine the target generation length of the imagination trajectory and the target sampling ratio of the imagination sample in the policy training based on the training loss of the adaptive imagination module. The generation module 340 is used by the self-verifying sample generator to generate an imagined trajectory of the target generation length based on the current strategy and the selected starting state, and store it in the imagined experience playback buffer. The sampling module 350 is used to sample imagined samples from the imagined experience playback buffer at the target sampling ratio and sample real samples from the real experience playback buffer during the policy training iteration, and to form a training batch together for policy training. The judgment module 360 ​​is used to repeat all the above steps until the adaptive imagination module determines that the preset conditions have been met based on the training loss sequence of the self-verifying sample generator, and then reduces the frequency of interaction with the environment.

[0108] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as...Figure 8 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a sample-based robot operation skill learning method.

[0109] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0110] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the sample-based robot operation skill learning method provided by the above methods.

[0111] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the sample-based robot operation skill learning methods provided by the above methods.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot operation skill learning method based on sample generation, characterized in that, include: The robot interacts with the environment, collects real samples, and stores them in a real-world experience playback buffer. The self-validating sample generator is trained using the real samples, and the training loss of the self-validating sample generator is obtained; the self-validating sample generator is used to learn environmental dynamics and generate imagined samples. The adaptive imagination module dynamically determines the target generation length of the imagination trajectory and the target sampling ratio of the imagination samples in the policy training based on the training loss. The self-verifying sample generator generates an imagined trajectory of the target generation length based on the current strategy and the selected starting state, and stores it in the imagined experience playback buffer. In the strategy training iteration, imagined samples are sampled from the imagined experience playback buffer at the target sampling ratio, and real samples are sampled from the real experience playback buffer, which together form a training batch for strategy training. Repeat all the above steps until the adaptive imagination module determines that the preset conditions have been met based on the training loss sequence of the self-verifying sample generator, then reduce the frequency of interaction with the environment.

2. The robot operation skill learning method based on sample generation according to claim 1, characterized in that, The self-verifying sample generator includes a state visualization module, an action verification module, and a reward visualization module; the real sample includes the current state, the action being performed, the environmental reward, and the next state. The step of training a self-validating sample generator using the real samples and obtaining the training loss of the self-validating sample generator includes: The current state and the action performed in the real sample are input into the state imagination module, the imagined next state is output, and the first loss is determined based on the imagined next state and the next state in the real sample. The imagined next state and the current state in the real sample are input into the action verification module, the predicted action is output, and the second loss is determined based on the predicted action and the executed action in the real sample. The current state in the real sample is input into the reward imagination module, the imagined reward is output, and the third loss is determined based on the imagined reward and the environmental reward in the real sample. The training loss of the self-verifying sample generator is determined based on the first loss, the second loss, and the third loss.

3. The robot operation skill learning method based on sample generation according to claim 1, characterized in that, The method for determining the target generation length of the imagined trajectory is as follows: Determine the scaling factor between the training loss and the initial loss of the self-validating sample generator, and determine the target generation length based on the scaling factor and the maximum step size of the task.

4. The robot operation skill learning method based on sample generation according to claim 1, characterized in that, The target sampling ratio of imagined samples in policy training is determined in the following way: Determine the ratio of the training loss to the initial loss of the self-validating sample generator. The target sampling ratio is determined based on the maximum value of the set imagined sample sampling ratio, the initial loss, and the scaling factor.

5. The robot operation skill learning method based on sample generation according to claim 1, characterized in that, The method for determining the preset conditions is as follows: Calculate the expected value and standard deviation of the training loss of the self-validating sample generator in the same training round; If the expected value is less than the first threshold and the standard deviation is less than the second threshold for multiple consecutive training rounds, it is determined that the time to reduce interaction with the environment has been reached, and the interaction frequency is switched.

6. The robot operation skill learning method based on sample generation according to claim 1, characterized in that, The sampling ratio of real samples from the real experience replay buffer is 1 minus the target sampling ratio.

7. A robot operation skill learning device based on sample generation, characterized in that, include: The interaction module is used for the robot to interact with the environment, collect real samples and store them in the real experience playback buffer. The training module is used to train a self-validating sample generator using the real samples and obtain the training loss of the self-validating sample generator; the self-validating sample generator is used to learn environmental dynamics and generate imagined samples. The calculation module is used to dynamically determine the target generation length of the imagination trajectory and the target sampling ratio of the imagination samples in the policy training based on the training loss of the adaptive imagination module. The generation module is used by the self-verifying sample generator to generate an imagined trajectory of the target generation length based on the current strategy and the selected starting state, and store it in the imagined experience playback buffer. The sampling module is used to sample imagined samples from the imagined experience playback buffer and sample real samples from the real experience playback buffer at the target sampling ratio during the policy training iteration, and to form a training batch together for policy training. The judgment module is used to repeat all the above steps until the adaptive imagination module determines that the preset conditions have been met based on the training loss sequence of the self-verifying sample generator, and then reduces the frequency of interaction with the environment.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the robot operation skill learning method based on sample generation as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the robot operation skill learning method based on sample generation as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the robot operation skill learning method based on sample generation as described in any one of claims 1 to 6.

Images