A method for active learning, enhancement and virtual-real transfer of complex skills in a heterogeneous scene

Abstract

The application provides a complex skill active learning, enhancement and virtual-real migration method in a heterogeneous scene, which comprises the following steps: firstly, collecting state data through multi-actuator environment interaction, storing the data into a shared experience pool, alternately training the actuators and discriminators after sampling, and finally selecting the optimal actuator deployment to realize complex skill active learning and enhancement; then in the digital twin environment of 3C assembly, a skill knowledge base is constructed by collecting multi-modal data, a strategy sequence is generated, residual reinforcement learning and other technologies are used for optimization, and the skill strategy is migrated from virtual to real through an encoding-decoding model. Based on the scheme provided in the application, without a teacher model or demonstration data, the efficiency of skill learning is improved by combining reinforcement learning and knowledge distillation technology, which provides a new perspective and method for solving the problem of skill learning in a complex environment, and realizes the virtual-real migration of operation skills from a simulation environment to a real assembly environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep reinforcement learning, and in particular to a method for active learning, enhancement and virtual-real transfer of complex skills in a heterogeneous scene. BACKGROUND

[0002] Current skill learning methods based on reinforcement learning, especially deep reinforcement learning, have achieved great success in the field of robot-related applications. However, due to the trial-and-error exploration mechanism in the interactive environment during the training of intelligent agents, it is difficult for intelligent agents to efficiently train optimal or suboptimal strategies, which poses a great challenge to the efficiency and stability of skill policy training.

[0003] With the application of reinforcement learning, imitation learning and other methods to the 3C assembly field, skill learning usually requires a large amount of training data, and the cost of data collection in the real environment is high, so direct implementation of policy training is usually not feasible. SUMMARY

[0004] The present application aims to at least partially solve one of the technical problems in the related art.

[0005] To this end, the first object of the present application is to propose a method for active learning and enhancement of complex skills in a heterogeneous scene, which learns the performance and stability of skill learning by drawing on the mechanism of human mutual learning in the field of skill learning without a teacher model or demonstration data.

[0006] The second object of the present application is to propose a virtual-real transfer method to realize the virtual-real transfer of operation skills from a simulation environment to a real assembly environment.

[0007] To achieve the above-mentioned objects, the first aspect of the present application proposes a method for active learning and enhancement of complex skills in a heterogeneous scene, comprising:

[0008] Initializing the parameters of a plurality of executors and a plurality of discriminators, and collecting state transition pair data based on the interaction of each executor with the environment;

[0009] Storing the state transition pair data in a shared data experience replay pool, and sampling a small batch of training sample data sets from the data experience replay pool;

[0010] Performing alternating training of the discriminators and the executors in the skill policy learning process through the training sample data sets;

[0011] After the skill policy training is completed, selecting the best-performing executor to perform in the environment.

[0012] Optionally, the state transition pair data {(st ,a t ,r t ,s t+1 )},include:

[0013] At each interaction time step t, each executor π θi Receive state s t And output the state corresponding to s t The actions follow a normal distribution, where i represents the i-th policy function and θ represents the parameters of the policy function;

[0014] Sampling action a from the normal distribution t : t ~π θi (·| t ), and control actuator π θi Execute a in the interactive environment t ;

[0015] After performing action a t Afterwards, actuator π θi The state transitions to the new state s t+1 And receive reward r t .

[0016] Optional, also includes:

[0017] All actuators share a data experience replay pool. The state transition pairs obtained from the interaction between different actuators and the environment are stored in the same data experience replay pool. middle;

[0018] When the data experience replay pool When the maximum capacity is reached, the first-in, first-out (FIFO) principle is used to replace the earliest collected interactive collection state transition pair with the latest collected interactive collection state transition pair.

[0019] Optionally, the alternating training of the discriminator and the actuator during the skill strategy learning process includes:

[0020] Each discriminator Training begins with calculating the target value y, and all discriminators are optimized based on minimizing the mean square error.

[0021] Each actuator π θi Training begins with calculating the distillation loss and then using gradient ascent to optimize the parameters of each actuator.

[0022] To achieve the above objectives, a second aspect of this application proposes a virtual-to-real migration method, comprising:

[0023] constructing a digital twin environment based on a real 3C assembly scene, collecting multi-modal demonstration data in the digital twin environment;

[0024] establishing a skill knowledge base based on skill analysis of the multi-modal demonstration data, and generating a basic strategy sequence of an operation task according to the skill knowledge base;

[0025] learning each primitive strategy sequence using residual reinforcement learning and curriculum learning, and realizing migration of the skill strategy from the digital twin environment to the real environment through an encoder-decoder model, wherein the skill strategy is obtained through the active learning and enhancement method of the complex skill in the heterogeneous scene according to any one of the first aspect.

[0026] Optionally, the constructing a digital twin environment based on a real 3C assembly scene, collecting multi-modal demonstration data in the digital twin environment, comprises:

[0027] constructing the digital twin environment based on a Unity simulation engine, which is equivalent and consistent with the real 3C assembly scene;

[0028] realizing collection of the multi-modal demonstration data in the digital twin environment based on a virtual reality (VR) device.

[0029] Optionally, the skill analysis process of the multi-modal demonstration data comprises:

[0030] precisely analyzing the collected multi-modal demonstration data through a time-space-event correlation model to obtain RGB-D images and three-dimensional force data;

[0031] identifying and segmenting an assembly action sequence of the multi-modal demonstration data using a time-space-contact feature segmentation system.

[0032] Optionally, the learning each primitive strategy sequence using residual reinforcement learning and curriculum learning comprises:

[0033] At each time step t, for a given state-action pair, a target function optimization follows a skill strategy execution that maximizes the expected cumulative reward obtained, and the action is an instruction output by the skill strategy for controlling the motion of the robot.

[0034] Optionally, the realizing migration of the skill strategy from the digital twin environment to the real environment through an encoder-decoder model comprises:

[0035] using an encoder as a feature extractor of an image input to realize migration of the skill strategy from the digital twin environment to an assembly scene in the real environment.

[0036] Optionally, it further comprises:

[0037] The basic operation of the robot is represented using a multi-level knowledge structure, the action sequence derived from human action demonstration is stored through a dynamic layer, and the basic knowledge relatively fixed in the task scene is stored through a static layer.

[0038] The technical solutions provided by the embodiments of the present application at least bring the following beneficial effects:

[0039] In the active learning and enhancement of complex skills in a heterogeneous scene, by combining reinforcement learning and knowledge distillation technology, the efficiency of skill learning is improved, and the robustness and generalization ability of the strategy are enhanced, providing a new perspective and method for solving skill learning problems in complex environments; and the 3C assembly operation skill learning method based on digital twin environment assistance not only improves the efficiency and accuracy of skill learning, but also greatly enhances the application potential of skills in complex and uncertain environments, realizes the virtual-real migration of operation skills from the simulation environment to the real assembly environment, and then in the real assembly environment, the robot can complete the operation tasks in the virtual environment, reducing the error between the virtual and real spatial domains. For typical and common pick-and-place, jack, and other tasks in the assembly environment, the accuracy of the robot is improved, and the robot has robustness.

[0040] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0041] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0042] Figure 1 is a flowchart of a complex skill active learning and enhancement method in a heterogeneous scene according to an embodiment of the present application;

[0043] Figure 2 is a training principle diagram of a complex skill active learning and enhancement method in a heterogeneous scene according to an embodiment of the present application;

[0044] Figure 3 is a flowchart of a virtual-real migration method according to an embodiment of the present application;

[0045] Figure 4 is a construction schematic diagram of a multi-modal digital twin environment according to an embodiment of the present application;

[0046] Figure 5 is an action recognition framework diagram based on time-space-touch-sound features according to an embodiment of the present application;

[0047] Figure 6This is a schematic diagram illustrating a primitive policy learning method according to an embodiment of this application. Detailed Implementation

[0048] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0049] The following description, with reference to the accompanying drawings, illustrates an embodiment of this application of a method for active learning, enhancement, and virtual-real transfer of complex skills in heterogeneous scenarios.

[0050] Figure 1 This is a flowchart illustrating a method for active learning and enhancement of complex skills in heterogeneous scenarios, as shown in the embodiments of this application. Figure 1 As shown, the method includes the following steps:

[0051] Step 101: Initialize the parameters of multiple executors and multiple discriminators, and collect state transition pair data based on the interaction between each executor and the environment.

[0052] In the field of robot skill learning, there is often a lack of powerful teacher models or a large amount of high-quality demonstration data. Therefore, methods such as imitation learning or policy distillation are typically limited in their application in this area. To address these shortcomings, this embodiment aims to enhance the performance and stability of skill learning by drawing on mechanisms of human peer learning in the absence of teacher models or demonstration data.

[0053] In this embodiment of the application, a method for active learning and enhancement of complex skills in heterogeneous scenarios is essentially a skill enhancement learning algorithm based on knowledge distillation. The algorithm includes multiple executors and multiple discriminators, and establishes a mutual learning mechanism based on knowledge distillation among multiple executors. In order to introduce the diversity of policy training, the parameters of each executor and discriminator are randomly initialized to different values.

[0054] First, the embodiments of this application are based on each actuator π θi Interaction with the environment to collect state transition pairs of data {(s) t ,a t ,r t ,s t+1 )}, where i represents the i-th policy function and θ represents the parameters of the policy function.

[0055] Specifically, the process of collecting state transition data is as follows: at each interaction time step t, each executor π θi Receive state s tAnd output the state corresponding to s t The actions follow a normal distribution; sample action a from the normal distribution. t : t ~π θi (·| t ), and control actuator π θi Execute a in the interactive environment t After completing action a t Afterwards, actuator π θi The state transitions to the new state s t+1 And receive reward r t .

[0056] Step 102: Store the state transition pair data in a shared data experience replay pool, and sample mini-batch training sample datasets from the data experience replay pool.

[0057] In this embodiment of the application, after obtaining the state transition pair data {(s)} t ,a t ,r t ,s t+1 After that, it is stored in the shared data experience replay pool.

[0058] It should be noted that all actuators share the data experience replay pool. The state transition pairs obtained from the interaction between different actuators and the environment are stored in the same data experience replay pool. middle.

[0059] Additionally, in one embodiment of this application, a data experience replay pool is used. The size is set to 1.0 × 10⁵, when the data experience replay pool... When the maximum capacity is reached, the first-in, first-out (FIFO) principle is used to replace the earliest collected interactive collection state transition pair with the latest collected interactive collection state transition pair.

[0060] Then, a small batch of training sample datasets is sampled from the initial data experience replay pool and continuously updated. The data experience pool data recycling method described above ensures the diversity of training data and improves the sample utilization efficiency of the knowledge distillation-based skill reinforcement learning algorithm to a certain extent.

[0061] Step 103: Using the training sample dataset, perform alternating training of the discriminator and the executor during the skill policy learning process.

[0062] In the embodiments of this application, such as Figure 2 As shown, the discriminator and the executor are trained alternately throughout the entire skill strategy learning process.

[0063] Specifically, each discriminator The training starts with calculating the target value y, and all discriminators are optimized based on a form of minimizing mean square error; each executor θi The training starts with calculating the distillation loss, and the parameters of each executor are optimized using the gradient ascent method.

[0064] Step 104, after the skill policy training is completed, the best performing executor is selected to perform in the environment.

[0065] In an embodiment of the present application, after the skill policy training is completed, the algorithm converges, and the best performing executor is selected to perform in the environment.

[0066] Through the above algorithm, in the absence of a teacher model or demonstration data, the performance and stability of skill learning are enhanced by learning from the mechanism of human mutual learning in the field of skill learning.

[0067] According to the first aspect of the present application, the skill reinforcement learning algorithm based on knowledge distillation is used to improve the performance and stability of skill learning. In order to facilitate skill transfer, the present application also proposes a virtual-real transfer method.

[0068] Figure 3 is a flowchart of a virtual-real transfer method according to an embodiment of the present application, comprising:

[0069] Step 201, a digital twin environment is constructed based on a real 3C assembly scene, and multi-modal demonstration data is collected in the digital twin environment.

[0070] In order to facilitate the collection of data required for training and verify the transfer of training strategy from simulation environment to real 3C assembly scene, in the present application, as shown in Figure 4 , a twin environment equivalent and consistent with the real 3C assembly scene is constructed based on the Unity simulation engine to simulate the real assembly scene, and a perception model containing multi-modal information such as vision, touch, and depth is established in the constructed digital twin simulation environment.

[0071] Then, based on the virtual reality VR device, multi-modal demonstration data is collected in the digital twin environment.

[0072] Step 202, a skill knowledge base is established based on skill analysis of multi-modal demonstration data, and a basic strategy sequence of operation tasks is generated according to the skill instruction base.

[0073] In an embodiment of the present application, the process of skill analysis of multi-modal demonstration data is as shown in Figure 5As shown, the collected multimodal teaching data is accurately analyzed using a time-space-event correlation model to obtain RGB-D images and three-dimensional force data; and the assembly action sequence of the multimodal teaching data is identified and segmented using a time-space-contact feature segmentation system.

[0074] By performing feature extraction, contact force feature extraction, and temporal action segmentation on multimodal teaching data, accurate parsing of multimodal teaching data was achieved.

[0075] Step 203: Residual reinforcement learning and curriculum learning are used to learn each primitive policy sequence, and the skill policy is transferred from the digital twin environment to the real environment through the encoder-decoder model.

[0076] Based on a constructed skill knowledge base, the robot can infer primitive policy sequences for performing complex assembly tasks. However, in real-world 3C assembly scenarios, executing primitive policy sequences to manipulate 3C components involves contact dynamics and friction, and is also affected by unavoidable random perturbations. To address these issues, this application proposes a novel primitive policy learning method, such as... Figure 6 As shown, the domain randomization method and residual reinforcement learning are integrated into the training of primitive policies in common 3C assembly scenarios.

[0077] Specifically, to improve the generalization performance of the strategy, residual reinforcement learning and curriculum learning are used to learn each primitive policy sequence. At each time step t, for a given state-action pair, the objective function optimization follows the principle of maximizing the expected cumulative reward obtained when the skill policy is executed. The action is the instruction output by the skill policy to control the robot's movement.

[0078] It should be noted that the skill strategy in the embodiments of this disclosure is based on, for example, Figure 1 This is a method for actively learning and enhancing complex skills in a heterogeneous scenario.

[0079] Furthermore, a randomization method is employed to transfer learned skill strategies from the digital twin environment to the real environment. An encoder-decoder model is used, with the encoder serving as a feature extractor for image input, to facilitate the transfer of skill strategies from the digital twin environment to the assembly scene of the real environment.

[0080] Furthermore, in some embodiments of this application, a multi-level knowledge structure is used to represent the robot's basic operations, which can be divided into static and dynamic layers. These two knowledge representations not only ensure the stability of the skill knowledge base structure but also guarantee its flexibility.

[0081] As one possible implementation, the dynamic layer stores action sequences derived from demonstrations of human actions, while the static layer primarily stores relatively fixed basic knowledge within the task scenario. The knowledge provided by the static and dynamic layers serves as semantic guidance for subsequent skill learning.

[0082] By employing the aforementioned algorithm, the accuracy and robustness of robots in skill learning under dynamic scenarios with uncertainties are improved, enabling multi-task transfer of operational skills in complex assembly scenarios.

[0083] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0084] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for actively learning and enhancing complex skills in heterogeneous scenarios, characterized in that, include: Initialize the parameters of multiple executors and multiple discriminators, and collect state transition pair data based on the interaction between each executor and the environment; The state transition pairs are stored in a shared data experience replay pool, and mini-batch training sample datasets are sampled from the data experience replay pool. Using the training sample dataset, the discriminator and the executor are trained alternately during the skill policy learning process; Once the skill strategy training is complete, select the best-performing executor to execute it in the environment; The alternating training of the discriminator and the actuator during the skill strategy learning process includes: Each discriminator Training begins with calculating the target value y, and all discriminators are optimized based on minimizing the mean square error. Each actuator π θi Training begins with calculating the distillation loss and then using gradient ascent to optimize the parameters of each actuator.

2. The method according to claim 1, characterized in that, The state transition data {(s) collected based on the interaction between each actuator and the environment t ,a t ,r t ,s t+1 )},include: At each interaction time step t, each executor π θi Receive state s t And output the state corresponding to s t The actions follow a normal distribution, where i represents the i-th policy function and θ represents the parameters of the policy function; Sampling action a from the normal distribution t :a t ~π θi (·|s t ), and control actuator π θi Execute a in the interactive environment t ; After completing action a t Afterwards, actuator π θi The state transitions to the new state s t+1 And receive reward r t .

3. The method according to claim 1, characterized in that, Also includes: All actuators share a data experience replay pool. The state transition pairs obtained from the interaction between different actuators and the environment are stored in the same data experience replay pool. middle; When the data experience replay pool When the maximum capacity is reached, the first-in, first-out (FIFO) principle is used to replace the earliest collected interactive collection state transition pair with the latest collected interactive collection state transition pair.

4. A method for virtual-to-real migration, characterized in that, include: A digital twin environment is constructed based on a real 3C assembly scenario, and multimodal teaching data is collected in the digital twin environment; A skill knowledge base is established based on the skill analysis of the multimodal teaching data, and a basic strategy sequence for the operation task is generated based on the skill knowledge base. Each primitive policy sequence is learned by using residual reinforcement learning and curriculum learning. The skill policy is transferred from the digital twin environment to the real environment through an encoder-decoder model. The skill policy is obtained by the active learning and enhancement method for complex skills in heterogeneous scenarios as described in any one of claims 1-3.

5. The method according to claim 4, characterized in that, The digital twin environment, constructed based on a real 3C assembly scenario, collects multimodal teaching data within this environment, including: The twin environment is constructed based on the Unity simulation engine and is equivalent to and consistent with the real 3C assembly scene. The multimodal teaching data is collected in the digital twin environment using virtual reality (VR) devices.

6. The method according to claim 4, characterized in that, The skill parsing process for the multimodal teaching data includes: The collected multimodal teaching data is precisely analyzed using a time-space-event correlation model to obtain RGB-D images and three-dimensional force data. The assembly action sequence of the multimodal teaching data is identified and segmented using a time-space-contact feature segmentation system.

7. The method according to claim 4, characterized in that, The method of learning each primitive policy sequence using residual reinforcement learning and curriculum learning includes: At each time step t, for a given state-action pair, the objective function optimization follows the principle of maximizing the expected cumulative reward obtained when the skill policy is executed, where the action is an instruction output by the skill policy to control the robot's movement.

8. The method according to claim 4, characterized in that, The transfer of skill strategies from the digital twin environment to the real environment through the encoder-decoder model includes: By using the encoder as a feature extractor for image input, the skill strategy can be transferred from the digital twin environment to the assembly scene in the real environment.

9. The method according to claim 4, characterized in that, Also includes: The robot's basic operations are represented by a multi-level knowledge structure. The dynamic layer stores the action sequences derived from human action demonstrations, while the static layer stores relatively fixed basic knowledge in the task scenario.

Images