Active learning and augmentation method for complex skill in heterogeneous scenario and sim-to-real transfer method

Abstract

Provided in the present application are an active learning and augmentation method for a complex skill in a heterogeneous scenario and a sim-to-real transfer method. The active learning and augmentation method comprises: first, collecting state data by means of the interaction of multiple executors and an environment; storing same in a shared experience replay buffer; after sampling, alternately training the executors and discriminators; and finally selecting the optimal executor to be deployed, so as to implement active learning and augmentation of a complex skill. The sim-to-real transfer method comprises: in a 3C assembly digital twin environment, collecting multi-modal data to construct a skill knowledge base; generating a policy sequence; using technologies such as residual reinforcement learning to perform optimization; and, by means of a coding and decoding model, transferring a skill policy from simulation to reality. Without a teacher model or teaching data, the solution provided in the present application improves skill learning efficiency by means of combining reinforcement learning and knowledge distillation techniques, and implements the sim-to-real transfer of operating skills from a simulation environment to a real assembly environment.

Description

Methods for Active Learning, Enhancement, and Virtual-Real Transfer of Complex Skills in Heterogeneous Scenarios Technical Field

[0001] This application relates to the field of deep reinforcement learning, and in particular to a method for active learning, reinforcement, and virtual-real transfer of complex skills in heterogeneous scenarios. Background Technology

[0002] Current skill learning methods based on reinforcement learning, especially deep reinforcement learning, have achieved significant success in robotics-related applications. However, because reinforcement learning requires learning through trial and error in an interactive environment during agent training, it is difficult for the agent to efficiently train to obtain optimal or suboptimal strategies. This poses a significant challenge to the efficiency and stability of skill strategy training.

[0003] As reinforcement learning and imitation learning methods are gradually applied to the 3C assembly field, skill learning usually requires a large amount of training data. Data collection in real-world environments is costly, and direct policy training is usually not feasible.

[0004] Summary of the Invention

[0005] This disclosure presents a method for active learning, enhancement, and virtual-real transfer of complex skills in heterogeneous scenarios, as well as electronic devices, storage media, and computer program products.

[0006] According to one aspect of this disclosure, a method for active learning of complex skills in heterogeneous scenarios is proposed. In the absence of a teacher model or instructional data, this method leverages human-to-human learning mechanisms to enhance the performance and stability of skill learning. The method includes: initializing parameters for multiple executors and multiple discriminators, and collecting state transition pairs based on the interaction between each executor and the environment; storing the state transition pairs in a shared data experience replay pool, and sampling mini-batch training sample datasets from the data experience replay pool; using the training sample datasets, performing alternating training of the discriminators and executors during the skill policy learning process; and selecting the best-performing executor for execution in the environment after the skill policy training is complete.

[0007] In some embodiments, the collection of state transition pairs based on the interaction between each actuator and the environment {(s) t a t r t s t+1 )},include:

[0008] At each interaction time step t, each executor π θi Receive state s t And output the state corresponding to s tThe actions follow a normal distribution, where i represents the i-th policy function and θ represents the parameters of the policy function;

[0009] Sampling action a from the normal distribution t :a t ~π θi (·|s t ), and control the actuator π θi Execute a in the interactive environment t ;

[0010] After completing action a t Afterwards, actuator π θi The state transitions to the new state s t+1 And receive reward r t .

[0011] In some embodiments, it also includes:

[0012] All actuators share a data experience replay pool. The state transition pairs obtained from the interaction between different actuators and the environment are stored in the same data experience replay pool. middle;

[0013] When the data experience replay pool When the maximum capacity is reached, the first-in, first-out (FIFO) principle is used to replace the earliest collected interactive collection state transition pair with the latest collected interactive collection state transition pair.

[0014] In some embodiments, the alternating training of the discriminator and the actuator during the skill strategy learning process includes:

[0015] Each discriminator Training begins with calculating the target value y, and all discriminators are optimized based on minimizing the mean square error.

[0016] Each actuator π θi Training begins with calculating the distillation loss and then using gradient ascent to optimize the parameters of each actuator.

[0017] According to the second aspect of this disclosure, a virtual-to-real migration method is proposed, comprising: constructing a digital twin environment based on a real 3C assembly scenario, and collecting multimodal teaching data in the digital twin environment; establishing a skill knowledge base based on skill analysis of the multimodal teaching data, and generating a basic strategy sequence for operation tasks according to the skill instruction base; learning each primitive strategy sequence using residual reinforcement learning and curriculum learning, and realizing the migration of skill strategies from the digital twin environment to the real environment through an encoder-decoder model, wherein the skill strategies are obtained through the active learning and enhancement method for complex skills in heterogeneous scenarios described in any of the first aspects above.

[0018] In some embodiments, the construction of a digital twin environment based on a real 3C assembly scenario, and the collection of multimodal teaching data in the digital twin environment, includes:

[0019] The twin environment is constructed based on the Unity simulation engine and is equivalent to and consistent with the real 3C assembly scene.

[0020] The multimodal teaching data is collected in the digital twin environment using virtual reality (VR) devices.

[0021] In some embodiments, the skill parsing process of the multimodal teaching data includes:

[0022] The collected multimodal teaching data is precisely analyzed using a time-space-event correlation model to obtain RGB-D images and three-dimensional force data.

[0023] The assembly action sequence of the multimodal teaching data is identified and segmented using a time-space-contact feature segmentation system.

[0024] In some embodiments, the learning of each primitive policy sequence using residual reinforcement learning and curriculum learning includes:

[0025] At each time step t, for a given state-action pair, the objective function optimization follows the principle of maximizing the expected cumulative reward obtained when the skill policy is executed, where the action is an instruction output by the skill policy to control the robot's movement.

[0026] In some embodiments, the migration of skill strategies from a digital twin environment to a real environment via an encoder-decoder model includes:

[0027] By using the encoder as a feature extractor for image input, the skill strategy can be transferred from the digital twin environment to the assembly scene in the real environment.

[0028] In some embodiments, it also includes:

[0029] The robot's basic operations are represented using a multi-level knowledge structure. The dynamic layer stores the action sequences derived from human action demonstrations, while the static layer stores relatively fixed basic knowledge specific to the task scenario.

[0030] According to a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein the memory stores instructions executable by said at least one processor, said instructions being executed by said at least one processor to enable said at least one processor to perform the hand-raising recognition method proposed in the first aspect above.

[0031] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform the hand-raising recognition method proposed in the first aspect above.

[0032] According to a fifth aspect of this disclosure, a computer program product is proposed, comprising a computer program that, when executed by a processor, implements the hand-raising recognition method proposed in the first aspect above.

[0033] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0034] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0035] Figure 1 is a flowchart illustrating a method for active learning and enhancement of complex skills in heterogeneous scenarios according to an embodiment of this application;

[0036] Figure 2 is a training principle diagram of a method for active learning and enhancement of complex skills in heterogeneous scenarios according to an embodiment of this application;

[0037] Figure 3 is a flowchart illustrating a virtual-to-real migration method according to an embodiment of this application;

[0038] Figure 4 is a schematic diagram illustrating the construction of a multimodal digital twin environment according to an embodiment of this application;

[0039] Figure 5 is a diagram of a motion recognition framework based on spatiotemporal-touch-sound features according to an embodiment of this application;

[0040] Figure 6 is a schematic diagram of a primitive policy learning method according to an embodiment of this application;

[0041] Figure 7 is a schematic block diagram of an electronic device according to an embodiment of the present application. Detailed Implementation

[0042] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0043] The following description, with reference to the accompanying drawings, illustrates an embodiment of this application of a method for active learning, enhancement, and virtual-real transfer of complex skills in heterogeneous scenarios.

[0044] Figure 1 is a flowchart illustrating a method for active learning and enhancement of complex skills in heterogeneous scenarios according to an embodiment of this application. As shown in Figure 1, the method includes the following steps:

[0045] Step 101: Initialize the parameters of multiple executors and multiple discriminators, and collect state transition pair data based on the interaction between each executor and the environment.

[0046] In the field of robot manipulation skill learning, there is often a lack of powerful teacher models or a large amount of high-quality demonstration data. Therefore, methods such as imitation learning or policy distillation are typically limited in their application in this area. To address these shortcomings, the purpose of embodiments of this application is to enhance the performance and stability of skill learning by drawing on mechanisms of human peer learning in the absence of teacher models or teaching data.

[0047] In the embodiments of this application, a method for active learning and enhancement of complex skills in heterogeneous scenarios is essentially a skill enhancement learning algorithm based on knowledge distillation. The algorithm includes multiple executors and multiple discriminators, and establishes a mutual learning mechanism based on knowledge distillation among multiple executors. In order to introduce the diversity of policy training, the parameters of each executor and discriminator are randomly initialized to different values.

[0048] First, the embodiments of this application are based on each actuator π θi Interaction with the environment to collect state transition pairs of data {(s) t a t r t s t+1 )}, where i represents the i-th policy function and θ represents the parameters of the policy function.

[0049] Specifically, the process of collecting state transition data is as follows: at each interaction time step t, each executor π θi Receive state s t And output the state corresponding to st The actions follow a normal distribution; sample action a from the normal distribution. t :a t ~π θi (.|s t ), and control the actuator π θi Execute a in the interactive environment t After completing action a t Afterwards, actuator π θi The state transitions to the new state s t+1 And receive reward r t .

[0050] Step 102: Store the state transition pair data in a shared data experience replay pool, and sample mini-batch training sample datasets from the data experience replay pool.

[0051] In the embodiments of this application, after obtaining the state transition pair data {(s)} t a t r t s t+1 After that, it is stored in the shared data experience replay pool.

[0052] It should be noted that all actuators share the data experience replay pool. The state transition pairs obtained from the interaction between different actuators and the environment are stored in the same data experience replay pool. middle.

[0053] Additionally, in one embodiment of this application, a data experience replay pool is used. The size is set to 1.0 × 10⁵, when the data experience replay pool... When the maximum capacity is reached, the first-in, first-out (FIFO) principle is used to replace the earliest collected interactive collection state transition pair with the latest collected interactive collection state transition pair.

[0054] Then, a small batch of training sample datasets is sampled from the initial data experience replay pool and continuously updated. The data experience pool data recycling method described above ensures the diversity of training data and improves the sample utilization efficiency of the knowledge distillation-based skill reinforcement learning algorithm to a certain extent.

[0055] Step 103: Using the training sample dataset, perform alternating training of the discriminator and the executor during the skill policy learning process.

[0056] In the embodiments of this application, as shown in FIG2, the discriminator and the executor are trained alternately throughout the skill strategy learning process.

[0057] Specifically, each discriminator Training begins with computing the target value y, and all discriminators are optimized based on minimizing the mean square error; each executor π θi Training begins with calculating the distillation loss and then using gradient ascent to optimize the parameters of each actuator.

[0058] Step 104: After the skill strategy training is completed, select the best-performing executor to execute in the environment.

[0059] In the embodiments of this application, after the skill strategy training is completed, the algorithm converges and selects the best-performing executor to execute in the environment.

[0060] Through the above algorithms, in the absence of teacher models or teaching data, the performance and stability of skills learning can be enhanced by drawing on the mechanism of human mutual learning.

[0061] According to the first aspect of this application, a skill reinforcement learning algorithm based on knowledge distillation is used to improve the performance of skill learning and the stability of training. In order to facilitate skill transfer, this application also proposes a virtual-real transfer method.

[0062] Figure 3 is a flowchart illustrating a virtual-to-real migration method according to an embodiment of this application, including:

[0063] Step 201: Construct a digital twin environment based on a real 3C assembly scenario, and collect multimodal teaching data in the digital twin environment.

[0064] To facilitate the collection of training data and the verification of the transfer of training strategies from the simulation environment to the real 3C assembly scenario, as shown in Figure 4, this application implements a twin environment that is equivalent to and consistent with the real 3C assembly scenario based on the Unity simulation engine to simulate the real assembly scenario. In the constructed digital twin simulation environment, a perception model containing multimodal information such as vision, touch, and depth is established.

[0065] Then, multimodal teaching data is collected in a digital twin environment using virtual reality (VR) devices.

[0066] Step 202: Establish a skill knowledge base based on skill analysis of multimodal teaching data, and generate a basic strategy sequence for operation tasks based on the skill instruction base.

[0067] In one embodiment of this application, the process of skill analysis of multimodal teaching data is shown in Figure 5. The collected multimodal teaching data is accurately analyzed through a time-space-event correlation model to obtain RGB-D images and three-dimensional force data; and the assembly action sequence of the multimodal teaching data is identified and segmented using a time-space-contact feature segmentation system.

[0068] By performing feature extraction, contact force feature extraction, and temporal action segmentation on multimodal teaching data, accurate parsing of multimodal teaching data was achieved.

[0069] Step 203: Residual reinforcement learning and curriculum learning are used to learn each primitive policy sequence, and the skill policy is transferred from the digital twin environment to the real environment through the encoder-decoder model.

[0070] Based on a constructed skill knowledge base, the robot can infer primitive policy sequences for performing complex assembly tasks. However, in real-world 3C assembly scenarios, executing primitive policy sequences to manipulate 3C components involves contact dynamics and friction, and is also affected by unavoidable random perturbations. To address these issues, this application proposes a novel primitive policy learning method, as shown in Figure 6, which integrates domain randomization and residual reinforcement learning into the training of primitive policies for common 3C assembly scenarios.

[0071] Specifically, to improve the generalization performance of the strategy, residual reinforcement learning and curriculum learning are used to learn each primitive policy sequence. At each time step t, for a given state-action pair, the objective function optimization follows the principle of maximizing the expected cumulative reward obtained when the skill policy is executed. The action is the instruction output by the skill policy to control the robot's movement.

[0072] It should be noted that the skill strategy in the embodiments of this disclosure is obtained based on a method for active learning and enhancement of complex skills in a heterogeneous scenario, as shown in Figure 1.

[0073] Furthermore, a randomization method is employed to transfer learned skill strategies from the digital twin environment to the real environment. An encoder-decoder model is used, with the encoder serving as a feature extractor for image input, to facilitate the transfer of skill strategies from the digital twin environment to the assembly scene of the real environment.

[0074] Furthermore, in some embodiments of this application, a multi-level knowledge structure is used to represent the robot's basic operations, which can be divided into static and dynamic layers. These two knowledge representations not only ensure the stability of the skill knowledge base structure but also guarantee its flexibility.

[0075] As one possible implementation, the dynamic layer stores action sequences derived from demonstrations of human actions, while the static layer primarily stores relatively fixed basic knowledge within the task scenario. The knowledge provided by the static and dynamic layers serves as semantic guidance for subsequent skill learning.

[0076] By employing the aforementioned algorithm, the accuracy and robustness of robots in skill learning under dynamic scenarios with uncertainties are improved, enabling multi-task transfer of operational skills in complex assembly scenarios.

[0077] Figure 7 illustrates a schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0078] As shown in Figure 7, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1004. RAM 1004 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0079] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0080] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the hand-raising recognition method. For example, in some embodiments, the hand-raising recognition method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1004 and executed by the computing unit 1001, one or more steps of the hand-raising recognition method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the hand-raising recognition method by any other suitable means (e.g., by means of firmware).

[0081] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0082] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0083] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0084] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0085] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0086] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers in distributed systems or servers integrated with blockchain technology.

[0087] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0088] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for actively learning and enhancing complex skills in heterogeneous scenarios, comprising: Initialize the parameters of multiple executors and multiple discriminators, and collect state transition pair data based on the interaction between each executor and the environment; The state transition pairs are stored in a shared data experience replay pool, and mini-batch training sample datasets are sampled from the data experience replay pool. Using the training sample dataset, the discriminator and the executor are trained alternately during the skill policy learning process; Once the skill strategy training is complete, select the best-performing executor to execute it in the environment.

2. The method according to claim 1, wherein, The state transition data {(s) collected based on the interaction between each actuator and the environment t a t r t s t+1 )},include: At each interaction time step t, each executor π θi Receive state s t And output the state corresponding to s t The actions follow a normal distribution, where i represents the i-th policy function and θ represents the parameters of the policy function; Sampling action a from the normal distribution t :a t ~π θi (·|s t ), and control the actuator π θi Execute a in the interactive environment t ; After completing action a t Afterwards, actuator π θi The state transitions to the new state s t+1 And receive reward r t .

3. The method according to claim 1, wherein, Also includes: All actuators share a data experience replay pool. The state transition pairs obtained from the interaction between different actuators and the environment are stored in the same data experience replay pool. middle; When the data experience replay pool When the maximum capacity is reached, the first-in, first-out (FIFO) principle is used to replace the earliest collected interactive collection state transition pair with the latest collected interactive collection state transition pair.

4. The method according to claim 1, wherein, The alternating training of the discriminator and the actuator during the skill strategy learning process includes: Each discriminator Training begins with calculating the target value y, and all discriminators are optimized based on minimizing the mean square error. Each actuator π θi Training begins with calculating the distillation loss and then using gradient ascent to optimize the parameters of each actuator.

5. A virtual-to-real transfer method, comprising: A digital twin environment is constructed based on a real 3C assembly scenario, and multimodal teaching data is collected in the digital twin environment; A skill knowledge base is established based on the skill analysis of the multimodal teaching data, and a basic strategy sequence for operation tasks is generated based on the skill instruction base. Each primitive policy sequence is learned by using residual reinforcement learning and curriculum learning. The skill policy is transferred from the digital twin environment to the real environment through an encoder-decoder model. The skill policy is obtained by the active learning and enhancement method for complex skills in heterogeneous scenarios as described in any one of claims 1-6.

6. The method according to claim 5, wherein, The digital twin environment, constructed based on a real 3C assembly scenario, collects multimodal teaching data within this environment, including: The twin environment is constructed based on the Unity simulation engine and is equivalent to and consistent with the real 3C assembly scene. The multimodal teaching data is collected in the digital twin environment using virtual reality (VR) devices.

7. The method according to claim 5, wherein, The skill parsing process for the multimodal teaching data includes: The collected multimodal teaching data is precisely analyzed using a time-space-event correlation model to obtain RGB-D images and three-dimensional force data. The assembly action sequence of the multimodal teaching data is identified and segmented using a time-space-contact feature segmentation system.

8. The method according to claim 5, wherein, The method of learning each primitive policy sequence using residual reinforcement learning and curriculum learning includes: At each time step t, for a given state-action pair, the objective function optimization follows the principle of maximizing the expected cumulative reward obtained when the skill policy is executed, where the action is an instruction output by the skill policy to control the robot's movement.

9. The method according to claim 5, wherein, The transfer of skill strategies from the digital twin environment to the real environment through the encoder-decoder model includes: By using the encoder as a feature extractor for image input, the skill strategy can be transferred from the digital twin environment to the assembly scene in the real environment.

10. The method according to claim 5, wherein, Also includes: The robot's basic operations are represented by a multi-level knowledge structure. The dynamic layer stores the action sequences derived from human action demonstrations, while the static layer stores relatively fixed basic knowledge in the task scenario.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4 or 5-10.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-4 or 5-10.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-4 or 5-10.

Images