Robot skill continuous learning method and device based on imitation learning

Abstract

The invention relates to a robot skill continuous learning method and device based on imitation learning, and the method comprises the steps: sequentially aiming at each task in a plurality of tasks; before a preset termination condition is met, circularly executing the following operations: inputting the observation image, the state information, a pre-acquired text instruction of the current task and a skill code book of the current task into a simulation learning strategy network, and predicting to obtain action information of the robot at the current moment, the skill code book of the current task is a plurality of pre-initialized learnable sub-skill vectors, and the skill code book comprises skill information of the skill code book of the previous task under the condition that the previous task exists; calculating loss based on the predicted action information of the robot and the real action information of the robot in the expert demonstration data; and updating the imitation learning strategy network and the skill code book of the current task based on the loss. The problems that skill selection is not flexible, and it is difficult to abstract the sub-skill hierarchical structure through manual skill design are solved.

Description

Technical Field

[0001] The present disclosure relates to the fields of information science and robotics technology, and in particular to a method and device for continuous learning of robot skills based on imitation learning. Background Art

[0002] Robots are able to sense the open and dynamic environment in which they are located, complete tasks according to human instructions, interact with humans, and are safe, flexible to deploy, and easy to operate. In recent years, they have been widely used in industrial manufacturing, medical rehabilitation, home services and other fields. In the real world, many of the tasks we want robots to solve may be different from the tasks they have completed in the past, but the skills they require are similar. Humans accumulate experience through extensive skill learning, and then quickly adapt to the environment and solve new tasks. Therefore, the continuous learning of robot skills is of great significance to the future development of robotics technology and is an important basis for the widespread application of robots in various fields in the future.

[0003] The robot policy networks in related technologies have problems such as inflexible skill selection, inconsistent execution, and difficulty in finding abstract hierarchical structures containing meaningful sub-behaviors in human-designed skills. Summary of the invention

[0004] The method and device for continuous learning of robot skills based on imitation learning provided by the exemplary embodiments of the present disclosure can at least solve the above-mentioned technical problems and other technical problems not mentioned above.

[0005] According to one aspect of the present disclosure, a method for continuous learning of robot skills based on imitation learning is provided, the method comprising: for each of a plurality of tasks in turn, cyclically performing the following operations before a preset termination condition is met: obtaining an observed image and state information of the robot at the current moment; inputting the observed image, the state information, a pre-acquired text instruction of the current task, and a skill code book of the current task into an imitation learning strategy network, and predicting the action information of the robot at the current moment, wherein the skill code book of the current task is a plurality of pre-initialized learnable sub-skill vectors, and in the case of a previous task, contains the skill information of the skill code book of the previous task; calculating the loss based on the predicted action information of the robot and the actual action information of the robot in expert demonstration data; and updating the imitation learning strategy network and the skill code book of the current task based on the loss to execute the continuous skill learning process of the robot.

[0006] Optionally, the imitation learning strategy network includes a perception layer, a skill inference layer and an action execution layer; wherein, the observation image, the state information, the pre-acquired text instructions of the current task, and the skill code book are input into the imitation learning strategy network to predict the action information of the robot at the current moment, including: inputting the observation image, the state information and the text instruction into the perception layer to obtain the multimodal fusion information at the current moment; inputting the multimodal fusion information and the skill code book into the skill inference layer to predict the skill vector at the current moment; inputting the skill vector into the action execution layer to obtain the action information of the robot at the current moment.

[0007] Optionally, the perception layer includes a pre-trained text encoder, a learnable multi-layer perceptron, a pre-trained learnable visual encoder, a learnable state encoder, and a modal fusion module; wherein, the observed image, the state information and the text instruction are input into the perception layer to obtain multimodal fusion information at the current moment, including: inputting the text instruction into the text encoder to obtain a text vector; inputting the text vector into the multi-layer perceptron to obtain a text representation; inputting the observed image into the visual encoder to obtain an image representation; inputting the state information into the state encoder to obtain a state representation; after fusing the text vector and the image representation through the modal fusion module, splicing the fused features with the text representation and the state representation to obtain the multimodal fusion information.

[0008] Optionally, the skill inference layer includes an attention module and a high-level decoder, and the high-level decoder includes a multi-head self-attention layer, a multi-head cross-attention layer and a feedforward network layer; wherein, the multimodal fusion information and the skill code book are input into the skill inference layer to predict the skill vector at the current moment, including: inputting the multimodal fusion information into the attention module to obtain weights for multiple learnable sub-skill vectors in the skill code book; weighting the multiple learnable sub-skill vectors based on the weights to obtain multiple learnable weighted sub-skill vectors; inputting the multimodal fusion information into the multi-head self-attention layer to obtain observation information containing timing information; inputting the features obtained by splicing the weighted sub-skill vectors and the observation information, and the observation information into the multi-head cross-attention layer, and inputting the output of the multi-head cross-attention layer into the feedforward network layer to predict the skill vector.

[0009] Optionally, the features obtained by splicing the weighted sub-skill vector and the observation information, and the observation information are input into the multi-head cross-attention layer, including: using the features obtained by splicing the weighted sub-skill vector and the observation information as the key and value of the multi-head cross-attention layer, and using the observation information as the query of the multi-head cross-attention layer to perform multi-head cross-attention.

[0010] Optionally, the action execution layer includes an underlying decoder and a Gaussian mixture model distribution layer, and the underlying decoder includes a multi-head self-attention layer and a feedforward network layer; wherein, inputting the skill vector into the action execution layer to obtain the action information of the robot at the current moment includes: processing the skill vector through the multi-head self-attention layer, the feedforward network layer and the Gaussian mixture model distribution layer to obtain the action information of the robot at the current moment.

[0011] Optionally, for the calculation of each head in the multi-head self-attention layer and the multi-head cross-attention layer in the skill inference layer, and the multi-head self-attention layer in the action execution layer, pattern approximation of the current head is achieved through the following operations: mapping the input of the current head into query, key and value through three projection matrices; stacking the weight matrix of the current head to obtain a first tensor; using CP decomposition to decompose the second tensor into the sum of at least one rank-one component, wherein the second tensor is a learnable tensor for the current task pre-created using pattern approximation parameters, and the components are obtained based on multiple shared components between the multiple tasks, specific components of the current task, and a learnable coefficient vector pre-randomly initialized for the current task; based on the query, key and value, the first tensor, and the second tensor after CP decomposition, the calculation of the current head is performed.

[0012] Optionally, the preset termination condition is to test the success rate of the imitation learning strategy network on the current task at a preset number of training checkpoints until a preset number of termination training times is reached, or after the success rate is higher than a first preset ratio, the success rate obtained from subsequent consecutive preset number of checkpoint tests is lower than the first preset ratio; if the preset termination condition is met, the training of the current task is terminated, and the model parameters of the imitation learning strategy network with the highest success rate are saved.

[0013] Optionally, the method further includes: when the preset termination condition is met, freezing the most important parameters of the second preset ratio in the convolutional layer and the linear layer in the imitation learning strategy network as specific parameters of the current task.

[0014] According to another aspect of the present disclosure, a robot skill continuous learning device based on imitation learning is also provided, characterized in that the device includes: a task switching module, configured to: for each of a plurality of tasks, cyclically execute the operations in the information acquisition module, the action prediction module, the loss calculation module, and the parameter updating module before satisfying a preset termination condition; the information acquisition module, configured to: obtain the observation image and state information of the robot at the current moment; the action prediction module, configured to: input the observation image, the state information, the pre-acquired text instructions of the current task, and the skill code book of the current task into the imitation learning strategy network, and predict the action information of the robot at the current moment, wherein the skill code book of the current task is a plurality of pre-initialized learnable sub-skill vectors, and in the case of a previous task, contains the skill information of the skill code book of the previous task; the loss calculation module, configured to: calculate the loss based on the predicted action information of the robot and the real action information of the robot in the expert demonstration data; the parameter updating module, configured to: update the imitation learning strategy network and the skill code book of the current task based on the loss to execute the continuous skill learning process of the robot.

[0015] Optionally, the imitation learning strategy network includes a perception layer, a skill inference layer and an action execution layer; wherein the action prediction module is configured to: input the observed image, the state information and the text instruction into the perception layer to obtain the multimodal fusion information at the current moment; input the multimodal fusion information and the skill code book into the skill inference layer to predict the skill vector at the current moment; input the skill vector into the action execution layer to obtain the action information of the robot at the current moment.

[0016] Optionally, the perception layer includes a pre-trained text encoder, a learnable multi-layer perceptron, a pre-trained learnable visual encoder, a learnable state encoder, and a modal fusion module; wherein the action prediction module is configured to: input the text instruction into the text encoder to obtain a text vector; input the text vector into the multi-layer perceptron to obtain a text representation; input the observed image into the visual encoder to obtain an image representation; input the state information into the state encoder to obtain a state representation; after fusing the text vector and the image representation through the modal fusion module, the fused features are spliced ​​with the text representation and the state representation to obtain the multimodal fusion information.

[0017] Optionally, the skill inference layer includes an attention module and a high-level decoder, and the high-level decoder includes a multi-head self-attention layer, a multi-head cross-attention layer and a feedforward network layer; wherein the action prediction module is configured to: input the multimodal fusion information into the attention module to obtain weights for multiple learnable sub-skill vectors in the skill coding book; weight the multiple learnable sub-skill vectors based on the weights to obtain multiple learnable weighted sub-skill vectors; input the multimodal fusion information into the multi-head self-attention layer to obtain observation information containing timing information; input the features obtained by splicing the weighted sub-skill vectors and the observation information, and the observation information into the multi-head cross-attention layer, and input the output of the multi-head cross-attention layer into the feedforward network layer to predict the skill vector.

[0018] Optionally, the action prediction module is configured to: use the features obtained by concatenating the weighted sub-skill vector and the observation information as the key and value of the multi-head cross-attention layer, and use the observation information as the query of the multi-head cross-attention layer to perform multi-head cross-attention.

[0019] Optionally, the action execution layer includes an underlying decoder and a Gaussian mixture model distribution layer, and the underlying decoder includes a multi-head self-attention layer and a feedforward network layer; wherein the action prediction module is configured to: process the skill vector through the multi-head self-attention layer, the feedforward network layer and the Gaussian mixture model distribution layer to obtain the action information of the robot at the current moment.

[0020] Optionally, for the calculation of each head in the multi-head self-attention layer and the multi-head cross-attention layer in the skill inference layer, and the multi-head self-attention layer in the action execution layer, the pattern approximation of the current head is achieved through the operation of a pattern approximation module; wherein the pattern approximation module is configured to: map the input of the current head into query, key and value through three projection matrices; stack the weight matrix of the current head to obtain a first tensor; use CP decomposition to decompose the second tensor into the sum of at least one rank-one component, wherein the second tensor is a learnable tensor for the current task pre-created using pattern approximation parameters, and the components are obtained based on multiple shared components between the multiple tasks, specific components of the current task, and a learnable coefficient vector pre-randomly initialized for the current task; based on the query, key and value, the first tensor, and the second tensor after CP decomposition, perform the calculation of the current head.

[0021] Optionally, the preset termination condition is to test the success rate of the imitation learning strategy network on the current task at a preset number of training checkpoints until a preset number of termination training times is reached, or after the success rate is higher than a first preset ratio, the success rate obtained from subsequent consecutive preset number of checkpoint tests is lower than the first preset ratio; if the preset termination condition is met, the training of the current task is terminated, and the model parameters of the imitation learning strategy network with the highest success rate are saved.

[0022] Optionally, the device also includes: a parameter isolation module, configured to: when the preset termination condition is met, freeze the most important parameters of the second preset ratio in the convolutional layer and the linear layer in the imitation learning strategy network as specific parameters of the current task.

[0023] According to another aspect of an embodiment of the present disclosure, there is also provided an electronic device, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute any of the above-described methods for continuous learning of robot skills based on imitation learning.

[0024] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is also provided. When the instructions are executed by at least one processor, the at least one processor is prompted to execute any of the above-described methods for continuous learning of robot skills based on imitation learning.

[0025] According to another aspect of an embodiment of the present disclosure, a system is also provided, comprising at least one computing device and at least one storage device storing instructions, wherein when the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute any of the above-described methods for continuous learning of robot skills based on imitation learning.

[0026] According to another aspect of an embodiment of the present disclosure, a computer program product is also provided, including a computer program / instruction, which, when executed by a processor, implements the method for continuous learning of robot skills based on imitation learning as described in any one of the above.

[0027] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0028] According to a method and device for continuous learning of robot skills based on imitation learning disclosed in the present invention, a skill code book is introduced to create a sub-skill library for each task separately. Through network training, the sub-skills used in each task are captured. When executing the current task, the sub-skill vector can also be selected from the skill code book of the previous task, thereby realizing sub-skill reuse, so that the actions generated by the robot when executing a long sequence (more complex or involving multiple targets) of tasks are more logical, smoother and more coherent.

[0029] In addition, by introducing a hierarchical architecture, the robot can adjust its decisions online based on real-time perception information when performing long-sequence tasks (more complex or involving multiple goals), and learn a reasonable skill abstraction level. Compared with other strategy architectures, it has better forward transfer capabilities and a higher average task success rate.

[0030] In addition, after learning each task, the main network is pruned and fine-tuned to freeze the previous task parameters. Task-specific parameters (sub-skill library and task-specific decomposition vector) and task-shared parameters are created for each task separately, giving the robot the ability to discover the same atomic skills implicit in different tasks. Compared with existing continuous learning algorithms, it has better backward transfer capabilities and anti-forgetting properties.

[0031] In addition, on long-viewing-distance datasets, it is possible to effectively capture similarities between tasks (such as the skills used or information about the environment layout), transfer the knowledge gained from learning new tasks, and improve the success rate of completing previous tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0033] Figure 1 The environment setting of each task in the target data set and the initial state of the robot's mechanical arm are shown in the exemplary embodiment of the present disclosure;

[0034] Figure 2 A flowchart showing a method for continuous learning of robot skills based on imitation learning in an exemplary embodiment of the present disclosure is shown;

[0035] Figure 3 A schematic diagram showing a flow chart of data processing by imitating a learning strategy network in an exemplary embodiment of the present disclosure;

[0036] Figure 4 The success rate of each task of the three strategy network architectures on the target data set in the continuous learning process in the exemplary embodiment of the present disclosure is shown;

[0037] Figure 5The success rate of each task of three continuous learning algorithms on the target data set in the continuous learning process in an exemplary embodiment of the present disclosure is shown;

[0038] Figure 6 A block diagram showing a robot skill continuous learning device based on imitation learning in an exemplary embodiment of the present disclosure is shown;

[0039] Figure 7 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0042] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0043] First, the definitions of abbreviations, English and key terms involved in this disclosure are introduced:

[0044] Catastrophic forgetting: After learning new knowledge, the previously acquired knowledge is almost completely forgotten. It makes artificial intelligence lack the ability to continuously adapt to the environment and learn incrementally (continuously) like biological organisms.

[0045] Imitation Learning (IL): Training machines to replicate continuous human actions, thereby giving them the ability to imitate humans to complete task goals.

[0046] Multitask Learning: It is a kind of joint learning, where multiple tasks are learned in parallel and the results influence each other.

[0047] Continual Learning: Like humans, it has the ability to learn continuously, and uses the experience and knowledge learned from historical tasks to help learn new tasks that continue to emerge. These experiences and knowledge are continuously accumulated, and old knowledge will not be forgotten because of new tasks.

[0048] Experience Replay: A continuous learning method that maintains a replay pool that stores previous task training data. During the training of a task, a portion of the replay data is selected to be trained together with the current task data to achieve an anti-forgetting effect.

[0049] Transformer: is a deep learning model architecture for natural language processing (NLP) and other sequence-to-sequence tasks.

[0050] PackNet: A state-of-the-art approach to augment multiple tasks in a single deep neural network while avoiding catastrophic forgetting.

[0051] Forward Transfer (FWT) is used to evaluate the ability of an agent to use prior knowledge to promote learning of new tasks.

[0052] Backward Transfer (BT) is used to evaluate the agent’s ability to use newly acquired knowledge to improve the model’s performance on previous tasks.

[0053] Mask: It is used to select network parameters. Its value represents the task index to which the parameter belongs.

[0054] FiLM: An existing method for fusing multimodal information, which is used to fuse text information and image information in the present invention.

[0055] CODA-Prompt: An existing approach to prompt learning for large models.

[0056] Next, we introduce the related technologies and their existing problems:

[0057] Imitation learning has made great progress in efficiently teaching robots to perform common manipulation tasks, especially through supervised learning using human teleoperation demonstrations or expert demonstration trajectories. Despite this promise, imitation learning methods have been limited to single short sequence tasks, such as opening a door or picking up a specific object, and have not performed well on multiple long sequence tasks.

[0058] When faced with long sequence tasks, traditional policy architectures process perception planning and action execution separately, which not only increases the complexity and latency of the system, but also makes it difficult to adjust decisions online based on real-time perception information, and the connection between skills is not smooth enough. By introducing skills, the robot can operate in the upper skill space instead of the lower action space, thereby achieving accurate reasoning over a longer time frame. Skills can be learned from demonstrations, trial-and-error learning, or manual definition, and these works show that a single Transformer policy can learn multiple skills. However, when sorting skills, connection problems may occur because the skills cannot transition smoothly. Some works solve this problem by learning to match the start and end sets of skills or treating the connection itself as a new skill, but this makes the skill library composition more complex and difficult to maintain. Decision Diffuser learns a generative model to combine skills, but the skill combination order needs to be specified in advance, which is not flexible enough.

[0059] In addition, the robot's state and action space is complex, and the behavior is abstract. Therefore, some hierarchical architectures that use skill classification before executing the skill often have difficulty determining the appropriate hierarchical level. That is, the artificially defined skills may be too general, resulting in the same skill corresponding to many different robot arm trajectories, and over-reliance on the subsequent action network; or they may be too specific, resulting in too small gaps between skill categories, and over-reliance on the accuracy of the previous skill network. Both of these situations will affect the robot's ability to perform tasks. Artificially defined sub-behaviors, such as option theory used in related technologies, hinder the exploration capabilities required for autonomous operation. In addition, referring to the skills corresponding to the tasks in the dataset, artificially explicitly defining and constraining the skill space will greatly reduce the flexibility of skill selection and the robot's ability to continuously learn new skills to complete new tasks, because the number of skills defined in this way is fixed and the types of skills cannot be continuously expanded as the robot continues to learn.

[0060] As an intelligent agent that interacts with the world, robots should be able to learn and perform different tasks in a variety of application scenarios. Traditional practices often adopt a multi-task learning paradigm to learn all the data at once to train the model. In the face of incremental learning goals, continuous retraining is required to avoid catastrophic forgetting, which will consume a lot of time and resources. In addition, robots and other intelligent agents cannot know all specific tasks when deployed in real environments. The storage and computing resources they can carry are extremely limited and should be carefully planned to avoid overflow. Therefore, multi-task learning cannot meet the needs of robot skill learning. Continuous learning allows robots to quickly adapt and learn in different environments and tasks without retraining or redesigning. Only incremental updates to existing models are required, which greatly improves the efficiency of task execution and gives robots the ability to continuously adapt to open dynamic environments and different task goals.

[0061] Traditional robot policy network continuous learning often uses the experience replay method, but in actual deployment, the robot agent may not have enough storage space to save previous task data. Some other works collect a large number of data sets of robots completing different tasks, store the different skills (actions) taken by the robot when completing the task as strategies, and use large language models or large models pre-trained on large-scale robot task data sets to decompose the tasks into these skills and obtain the final control output. When there are new tasks to learn, the skill library is expanded through fine-tuning. Although the powerful performance of the large language model gives the robot a strong learning ability, the large language model also has catastrophic forgetting and can produce "hallucinations", that is, it does not fully consider the robot's actual environment, target requirements, and the execution ability of the robot itself, and outputs unrealistic task execution plans. In addition, collecting a large number of robot skill data sets and pre-training not only consumes a lot of manpower and material resources, but also incurs unimaginable time costs.

[0062] In general, existing robot policy networks are unable to adjust decisions online based on real-time perception information, find it difficult to determine a reasonable level of skill abstraction, and the connection between skills is not smooth; most of them adopt simple continuous learning methods, resulting in low anti-forgetting capabilities and deployment feasibility.

[0063] In order to solve the above problems, the present disclosure provides a method and device for continuous learning of robot skills based on imitation learning. By introducing a skill code book, a sub-skill library is created for each task separately. Through network training, the sub-skills used in each task are captured. When executing the current task, the sub-skill vector can also be selected from the skill code book of the previous task, thereby realizing sub-skill reuse, so that the actions generated by the robot when executing a long sequence (more complex or involving multiple targets) of tasks are more logical, smoother and more coherent.

[0064] First, a training data set used in an exemplary embodiment of the present disclosure is introduced.

[0065] According to an exemplary embodiment of the present disclosure, a data set containing a variety of tasks and used for continuous learning in decision-making in the field of robot operation may be used for training.

[0066] Specifically, the dataset may include a variety of tasks that reflect daily human activities, such as turning on the stove, moving books, opening drawers, etc. Each task may have a corresponding language instruction, such as "open the top drawer of the cabinet and put the bowl in it".

[0067] The data sets can be distinguished to obtain spatial data sets, object data sets, target data sets and 100 data sets. The first three task sets can each contain ten tasks, which are used to study the transferability of model spatial information, object information and task target information knowledge.

[0068] Specifically, all tasks in the spatial dataset may require the robot to place a bowl on a plate in the same set of objects. Although the shapes, graphics, etc. of the two bowls are the same, they are different in position or spatial relationship with other objects. Therefore, in order to successfully complete the tasks in the spatial dataset, the robot needs to continuously learn and memorize new spatial relationships.

[0069] All tasks in the object dataset require the robot to pick up and place a unique object. Therefore, in order to complete the tasks in the object dataset, the robot needs to continuously learn and memorize new object types.

[0070] All tasks in the target dataset may have the same operational objects with fixed spatial relationships and may differ only in the task objectives. Therefore, in order to complete the tasks in the target dataset, the robot needs to continuously learn new knowledge about motion and behavior.

[0071] The 100 dataset can contain a total of 100 tasks that require interaction with different objects and utilize multifunctional motor skills. Among them, these 100 tasks can be divided into 90 short-viewing tasks (only one task target or only one object is regarded as an operation object) and 10 long-viewing tasks (including two task targets or needing to operate two objects). The default task order can be used during the experiment.

[0072] Figure 1 The environment setting of each task in the target data set and the initial state of the robot's manipulator arm are shown in an exemplary embodiment of the present disclosure.

[0073] Reference Figure 1The environment is set to operate in a kitchen environment. The tasks that the robot needs to complete are, for example, the first one on the upper left is to open the middle drawer, the second one is to open the top drawer and put the bowl in it... The 10 pictures correspond to 10 task objectives.

[0074] Next, we will refer to Figures 2 to 7 The present invention specifically describes the method and device for continuous learning of robot skills based on imitation learning.

[0075] Figure 2 A flowchart of a method for continuous learning of robot skills based on imitation learning in an exemplary embodiment of the present disclosure is shown.

[0076] Reference Figure 2 , for each of the multiple tasks in turn, the following operations 201 to 204 are cyclically performed until a preset termination condition is met.

[0077] According to an exemplary embodiment of the present disclosure, the robot skill learning problem can be regarded as a finite-time Markov decision process.

[0078] According to an exemplary embodiment of the present disclosure, to end the current task T i For learning, an early stopping method can be used: that is, the preset termination condition can be testing the success rate of the imitation learning strategy network on the current task at a preset number of training checkpoints until the preset termination training times are reached, or after the success rate is higher than the first preset ratio, the success rate obtained by subsequent consecutive preset number of checkpoint tests is lower than the first preset ratio; when the preset termination condition is met, the training of the current task is terminated, and the model parameters of the imitation learning strategy network with the highest success rate are saved.

[0079] Specifically, the first preset ratio can be 90%, and the consecutive preset number of times can be two or three consecutive times, that is, the success rate of the model on the current task is tested at the preset training number checkpoints until the preset termination training number is reached or the success rate is higher than 90% and there are two or three subsequent test success rates that are all lower than 90%, then the training is stopped and the model parameters of the imitation learning strategy network at the highest success rate are saved.

[0080] For checkpoint settings, one round of training can use all the data of the current task, and the test is performed every 5 rounds, for a total of 50 rounds. Setting intermediate test checkpoints can facilitate viewing the success rate of the model in completing tasks during training, so as to perform early stopping.

[0081] In operation 201, the observation image and state information of the robot at the current moment are obtained.

[0082] Specifically, at the beginning of training for each task, you can first initialize the continuous learning task data set, including a text description of each task goal (i.e., the text instruction of the current task), the observation image of the robot's wrist at different times when it completes the corresponding task, the third-person observation image (the robot's observation image), and the state information of the robot body at each time (joint posture information, torque, gripper state, etc.). You can also choose the type of task set, the number of tasks, and the order of tasks.

[0083] A task in the data set corresponds to a text instruction description. A task has multiple demonstration data and can be divided into multiple time series. Each time series contains two observation images at each moment and the state information of the robot body.

[0084] In operation 202, the observed image, state information, pre-acquired text instructions of the current task, and the skill code book of the current task are input into the imitation learning strategy network to predict the action information of the robot at the current moment, wherein the skill code book of the current task is a plurality of pre-initialized learnable sub-skill vectors, and in the case of a previous task, contains the skill information of the skill code book of the previous task.

[0085] Specifically, the skill codebook of the current task is a part of the continuous learning algorithm disclosed in the present invention and can be obtained through initialization.

[0086] According to an exemplary embodiment of the present disclosure, the imitation learning strategy network may include but is not limited to a perception layer, a skill inference layer, and an action execution layer. The observed image, state information, and text instructions may be input into the perception layer to obtain the multimodal fusion information at the current moment; the multimodal fusion information and the skill codebook may be input into the skill inference layer to predict the skill vector at the current moment; the skill vector may be input into the action execution layer to obtain the action information of the robot at the current moment.

[0087] Specifically, the perception layer can help the robot perceive the ever-changing real environment and extract key environmental information based on the task objectives; the skill inference layer can infer the skills to be performed based on the environmental information, task objectives and the robot's state to obtain the skill vector; the action execution layer can output the continuous value of the end effector action based on the skill vector corresponding to each decision time step.

[0088] The modal fusion of the perception layer can use FiLM to fuse text information and visual information to obtain the encoding vector of multimodal fusion information or t ; The skill inference layer is a high-level decoder, which can be a Transformer decoder, and the input is the encoding vector of multimodal fusion information or t , the output is the skill vector z predicted at the current moment tThe action execution layer, i.e. the bottom decoder, is a Transformer decoder with the same structure as the skill inference layer, but they have their own parameters, and their input is the skill vector z predicted at the current moment t , the output is action information, that is, the action vector a t .

[0089] For the perception layer, its main function is to help the robot recognize the ever-changing real environment, extract key environmental information according to the task objectives, and help the subsequent strategy network generate a logical sequence of actions.

[0090] According to an exemplary embodiment of the present disclosure, the perception layer may include but is not limited to a pre-trained text encoder, a learnable multi-layer perceptron, a pre-trained learnable visual encoder, a learnable state encoder, and a modal fusion module. The text instruction may be input into the text encoder to obtain a text vector; the text vector may be input into the multi-layer perceptron to obtain a text representation; the observed image may be input into the visual encoder to obtain an image representation; the state information may be input into the state encoder to obtain a state representation; after the text vector and the image representation are fused through the modal fusion module, the fused features may be spliced ​​with the text representation and the state representation to obtain multi-modal fusion information.

[0091] Specifically, the perception layer can include a pre-trained text encoder - CLIP and a learnable multi-layer perceptron (MLP), a pre-trained learnable visual encoder - ResNet-18, a learnable state encoder - a multi-layer perceptron (MLP), and a multimodal fusion module - FiLM. Its input can be: the task objectives described in English (such as Open the top drawer of the cabinet and put the bowl in it, etc.), that is, text instructions; the camera image and the third-person perspective image of the robot's wrist at the current moment, that is, the observation image; the robot's joint posture information, torque, gripper state, etc., that is, the state information; its output is the multimodal fusion information at the current moment or t .

[0092] In order to fuse the textual task instructions with the robot's visual information, the multimodal deep learning technology FiLM can be used to extract key environmental information according to the task objectives, so as to adjust the visual information features of the neural network based on the task instruction information, thereby realizing modal fusion. The modal fusion module implemented by FiLM can use the task objectives described by the textual instructions to correct the robot's visual information by extracting key environmental information; for key environmental information, for example, for the textual instruction "Open the top drawer of the cabinet and put the bowl in it", the top drawer and bowl in the visual information are key information because they are the objects that need to be directly operated.

[0093] Specifically, for a task with a length of seq l en time series, the text encoder can map the text instructions of the task into a text vector ir with a dimension of (task i ndex,clip o uput d im); the visual encoder can map the image sequence information (i.e., observed images / visual information) of the robot's two perspectives (third-person perspective and robot perspective) into image representations The dimension is (batch s ize,seq l en,embed s ize); the state encoder can map the robot's joint posture information, torque, and gripper state sequence information into a state representation with the same dimension

[0094] The ontological characteristics of the robot are a set of vectors, which refer to the joint posture information and torque of the robot arm and the state of the gripper. After encoding, they correspond to two representations and two modes.

[0095] Finally, we can use FiLM technology to fuse the two-modal representation of text information and image information. Then, it is combined with the text representation and state representation obtained by the multi-layer perceptron to obtain the multimodal fusion information output by the perception layer, with a dimension of (batch s ize,seq l en,num m odalities,embed s ize), where seq l en=10,num m odalities=5.

[0096] For the modal fusion module FiLM, we can consider a neural network f with an intermediate feature x and an external network g that can output modulation parameters γ and β. The adjusted feature x′ is:

[0097] γ,β=g(z)

[0098] x′=γ⊙x+β

[0099] Where z is the input of the external network g, representing the text vector that has only been encoded by CLIP (without further encoding by the multilayer perceptron at this time), ⊙ represents element-wise multiplication, γ and β are vectors of the same size as x, and x represents the image representation, each of which is used to modulate the corresponding feature in x, that is, the text information is encoded by g, and the image representation is modified using the obtained γ and β to achieve the fusion effect of text modality and visual modality, so that the image representation contains text information. This will eventually allow FiLM to allow conditional computation of features without explicitly changing the architecture, that is, the visual representation (equivalent to the encoded representative information about the image) is modulated by the text information vector (equivalent to the encoded representative information about the text). Therefore, the task text instruction embedding (i.e., the text vector) is used as the input of the fully connected feedforward network g in the modality fusion module, which outputs the scaling and transformation parameters of the image embedding, i.e., γ and β, which also belong to the modality fusion module. These parameters modulate the image embedding before passing it to the high-level decoder.

[0100] Afterwards, the skills to be performed can be inferred based on the environmental information, task objectives and the state of the robot to obtain the skill vector.

[0101] According to an exemplary embodiment of the present disclosure, the skill inference layer may include but is not limited to an attention module and a high-level decoder, and the high-level decoder may include but is not limited to a multi-head self-attention layer, a multi-head cross-attention layer, and a feedforward network layer. The multimodal fusion information may be input into the attention module to obtain weights for multiple learnable sub-skill vectors in the skill codebook; the multiple learnable sub-skill vectors are weighted based on the weights to obtain multiple learnable weighted sub-skill vectors; the multimodal fusion information is input into the multi-head self-attention layer to obtain observation information containing timing information; the features obtained by concatenating the weighted sub-skill vectors and the observation information, as well as the observation information, are input into the multi-head cross-attention layer, and the output of the multi-head cross-attention layer is input into the feedforward network layer to predict the skill vector.

[0102] Specifically, the input of the attention module is multimodal fusion information, and the output is the weight of the skill vector in the skill codebook (skill library). According to the weight, all skill vectors in the skill library can be weighted and summed to obtain the selected skill, which is then combined with the h of the Transformer in the skill inference layer. K and hV Combined, Transformer finally outputs the predicted skill at the current moment.

[0103] The specific role of the attention module is to calculate the cosine similarity γ between the result of the Hadamard product of the multimodal fusion information and A and K to obtain a weight vector. After that, the similarity can be used as a weight to select a skill vector from the skill library to be spliced ​​into the h of the Transformer in the skill inference layer. K and h V Above. The attention module is part of the skill inference layer.

[0104] In the high-level decoder of the skill inference layer, the input of the multi-head self-attention layer is multimodal fusion information, and the output is multimodal fusion information fused with temporal information, that is, observation information; the input of the multi-head cross-attention layer is multimodal fusion information fused with temporal information and the weighted sub-skill vector, and its output passes through the feedforward network to obtain the skill vector at the current moment.

[0105] According to an exemplary embodiment of the present disclosure, the features obtained by concatenating the weighted sub-skill vectors and the observation information can be used as the keys and values ​​of the multi-head cross-attention layer, and the observation information can be used as the query of the multi-head cross-attention layer to perform multi-head cross-attention.

[0106] Specifically, we can first obtain observation information that takes into account temporal information (i.e., multimodal fusion information that integrates temporal information) through the multi-head self-attention layer, and then concatenate the weighted sub-skill vectors at each moment, and then perform "multi-head cross attention". The query of "multi-head cross attention" is the output of the multi-head self-attention, and the key and value are the output of the multi-head self-attention concatenated with the sub-skill vectors.

[0107] The high-level decoders in the skill inference layer can use the Transformer decoder to infer the skills that should be performed based on environmental information, task objectives, and the state of the robot. In traditional decoders, the keys and values ​​of the cross-attention mechanism are the output of the Transformer encoder, which requires explicit encoding of all the skills that the robot may need to complete different tasks. The labor cost of this process is huge; specifically, in traditional decoders, the keys and values ​​of the cross-attention mechanism are the output of the Transformer encoder, which means that the Transformer encoder needs to be used to encode the skills that the robot may take (an additional training process is required and it is necessary to manually define which skills the robot has taken when completing different tasks, and which data represents which skills, that is, explicitly encoding the skills), which requires a lot of manpower and material resources.

[0108] The present disclosure can use skill codebooks (a set of learnable feature vectors) as keys and values ​​to implicitly model skills. This is because the motion trajectory of human experts when completing tasks, although seemingly complex, can usually be decomposed into more specific and repetitive sub-skills; creating task-specific skill codebooks can capture the sub-skills used in each task through network training. Specifically, a learnable skill codebook can be created separately for the i-th subtask in different continuous learning task sets, and each task skill codebook is guaranteed to be orthogonal to each other. When using the loss function to train the network, the skill codebook can be updated accordingly to model the sub-skills used by the robot's motion trajectory.

[0109] In this way, the model can learn the underlying structure of expert motion more effectively. In addition, when executing the current task, sub-skill vectors can be selected from the skill codebook of the previous task, which enables sub-skill reuse, helps generate diverse and realistic motion trajectories, and completes new tasks by recombining learned sub-skills according to task text instructions. Specifically, the robot multimodal fusion information encoded by the perception layer or t The weight of the sub-skill vector is obtained through the attention module, and then the weighted sub-skill vector is calculated. t First, it passes through the multi-head self-attention layer, and its output is then concatenated with the weighted sub-skill vector to obtain the key and value of the multi-head cross-attention. Its output itself is used as a query input to the multi-head cross-attention layer and the feedforward neural network to output the predicted skill z t , the dimension is (batch s ize,seq l en,D).

[0110] The present disclosure can create a learnable skill codebook for the i-th subtask (i=1, 2, 3, ..., 10) in different continuous learning task sets. Where N represents the number of skill primitives (sub-skills), L represents the total number of heads of keys and values ​​in the multi-head self-attention layer, and D represents the dimension of the hidden layer (in this invention, N = 10, L = 12, D = embed s ize). Whenever a new task needs to be learned later, the skill codebooks belonging to the old tasks will also participate in the weighted summation in order to utilize the learned knowledge and improve the learning speed. For one of the self-attention heads, the skill codebook matrix For example, each row corresponds to an atomic skill, describing its unique characteristics in the latent space. In addition, there is a matrix corresponding to the task skill encoding matrix It is used to calculate the similarity between the input and different skill vectors. In addition, the prefix fine-tuning method in the prompt learning can be used in the multi-head self-attention layer of the Transformer decoder (i.e., the high-level decoder) to calculate the weight of each sub-skill based on the observation information at the current time t to obtain the weighted sub-skill. The weighted sub-skill vector s t Concatenated with the keys and values ​​in the multi-head cross attention layer respectively.

[0111] Since the robot arm needs to perform a variety of similar skill actions when completing different tasks, and each task also has its own specific skill actions, for example, the two tasks of opening a drawer and rotating a button both include skill actions that allow the robot arm to move forward, left, or right, so that the gripper reaches the target position, and also include skill actions that control the gripper to close and grab the target object and then perform the corresponding operation; however, the task of opening a drawer requires the robot arm to move backward after grabbing the drawer handle to complete the task goal, while the task of rotating a button requires the robot arm to rotate the gripper after grabbing the button to complete the task goal. In order to achieve skill sharing while maintaining the independence of different task skills, thereby improving the learning rate and success rate of new tasks, the present disclosure can obtain a skill vector by weighted summing the skill codebook:

[0112]

[0113] Among them, α t,n is the weight vector of the nth sub-skill (n=1,2,…,N) matrix at time t,

[0114] The nth sub-skill matrix in the skill codebook (sub-skill library) learned from the first i-1 tasks and initialized for the i-th task is included. The proportion of each sub-skill vector in the skill vector to be executed is the multimodal fusion information output by the perception layer or t It is determined by the similarity with each sub-skill vector in the sub-skill library. In addition, each sub-skill matrix S n A corresponding attention matrix A will be initialized for it n , which helps to filter out features related to the current task skill selection from high-dimensional multimodal mixed information. The present disclosure can use a simple feature selection attention method to select the multimodal fusion information output by the perception layer or t With the attention matrix A n Do element-wise multiplication to create a matrix with attention, and finally perform similarity matching with the K matrix to obtain the final skill codebook weighted weight.

[0115] α t =γ(or t ⊙A,K)

[0116] =γ(or t ⊙A1,K1),…,γ(or t ⊙A N ,K N )

[0117] In order to reduce the interference between existing knowledge and newly learned knowledge, the skill codebook of the previous task can be frozen and an orthogonal loss can be introduced for P, K, and A:

[0118]

[0119] Where B represents an arbitrary matrix.

[0120] Take one of the multi-head self-attention layers as an example, its input is or t , query, key and value can be used respectively Q 、h K and h V In the exemplary embodiment of the present disclosure, Q =h K =h V =or′ t , o′ t is or t The value after dimension transformation, the dimension is (batch size ,seq len ×num m odalities,embed s size).

[0121] Then the output of this layer can be expressed as:

[0122]

[0123] Among them, W O , is the mapping matrix.

[0124] The present invention can convert the weighted sub-skill vector s t Divide into two equal parts:

[0125]

[0126] Next, you can use prefix fine-tuning to combine them with h K and h V Stitching it together:

[0127] f P-T (s,h)=MSA(h Q ,[s K ;h K ],[sV ;h V ])

[0128] For the action execution layer, the continuous value of the end effector action can be output according to the skill vector corresponding to each decision time step.

[0129] According to an exemplary embodiment of the present disclosure, the action execution layer may include but is not limited to an underlying decoder and a Gaussian mixture model distribution layer, and the underlying decoder may include but is not limited to a multi-head self-attention layer and a feedforward network layer. The skill vector may be processed by the multi-head self-attention layer, the feedforward network layer, and the Gaussian mixture model distribution layer to obtain the action information of the robot at the current moment.

[0130] Specifically, the bottom decoder in the action execution layer is a Transformer decoder with the same structure as the skill inference, but they have their own parameters. In addition, the action execution layer can also include a Gaussian mixture model distribution module. The input of the action execution layer is the skill z predicted at the current moment t , the output is the action vector a t The input of the bottom decoder is the skill vector at the current moment. The vector obtained after the Transformer decoder is then input into the Gaussian mixture model distribution to obtain the reference signal vector of the end effector, that is, the action vector.

[0131] Among them, a skill vector of one time step can correspond to one actuator action. For the dimension (batch_size, seq_len, D), it means that batch_size groups of training data are used during training, each group of training data includes skill vectors of seq_len time length, and each skill vector contains D values.

[0132] According to the skill vector corresponding to the decision time step, a reference signal, i.e., action (including three-dimensional coordinates of position, attitude angle information, and gripper switch information) can be generated. The end effector is a structure of the physical robot arm, which is responsible for calculating the joint torque information required to reach the position and attitude in the reference signal based on the current position and attitude of the robot arm.

[0133] The low-level decoder in the action execution layer can use a Transformer decoder, whose input is the skill vector selected by the skill inference layer at each decision time step, and the output is the corresponding action distribution. However, simple regression cannot fully cover the rich, multi-modal distribution of expert motion trajectories. Even if the same expert faces the same task, the strategies (action trajectories) adopted may not be exactly the same. To solve this problem, the present disclosure can use a Gaussian mixture model (GMM) based on a multi-layer perceptron to model the potential multi-modal trajectory distribution of the same skill z. Finally, the robot samples the continuous value a of the end effector action from the output distribution t (batch s ize,seq l en,ac d im) to execute the strategy, where ac d im=7. When evaluating, the average value of the Gaussian model with the highest probability can be used to parameterize the next action. The Gaussian mixture model expression is:

[0134]

[0135] Among them, τ is the demonstration trajectory when the expert completes the task, is the parameter of GMM, p(τ|θ,c m ) is the mth sub-Gaussian distribution model c contains M sub-distributions. Weight η m , represents the probability proportion of the mth sub-model, 0≤η k ≤1, GMMs are more expressive than simple multi-layer perceptrons because they are better able to capture the multimodal trajectories in the expert action data. The final learning goal of the GMM model is to minimize the negative log-likelihood of the expert demonstration action trajectory τ:

[0136]

[0137] In addition, model optimization can be performed through pattern approximation.

[0138] According to an exemplary embodiment of the present disclosure, for the calculation of each head in the multi-head self-attention layer and the multi-head cross-attention layer in the skill inference layer, and the multi-head self-attention layer in the action execution layer, pattern approximation of the current head can be achieved through the following operations: mapping the input of the current head to query, key and value through three projection matrices; stacking the weight matrix of the current head to obtain a first tensor; using CP decomposition to decompose the second tensor into the sum of at least one rank-one component, wherein the second tensor is a learnable tensor for the current task pre-created using pattern approximation parameters, and the components are obtained based on multiple shared components between multiple tasks, specific components of the current task, and a learnable coefficient vector pre-randomly initialized for the current task; based on the query, key and value, the first tensor, and the second tensor after CP decomposition, the calculation of the current head is performed.

[0139] Specifically, the CP decomposition vector is a part of the continuous learning algorithm of the present disclosure. The CP decomposition vector can be initialized when initializing the continuous learning algorithm.

[0140] The present disclosure proposes a robot skill continuous learning algorithm based on pattern approximation, including but not limited to the following steps:

[0141] Initialize the skill codebook S, the A and K matrices used to calculate the weights, and the CP decomposition vectors (task-shared vectors U, V and task-specific vector P) used to achieve pattern approximation;

[0142] Learning Task T i Initialize the current task's S i , A i , K i , the joint task shares the decomposition vector U i 、V i and the task-specific decomposition vector P i Participate in training.

[0143] More specifically, the task skill codebook S and A, K matrices can be initialized using uniform distribution and then Schmidt orthogonalized; U and P can be randomly initialized using Gaussian distribution, and V can be initialized to 0.

[0144] Learning Task T i Initialize the current task's S i , A i , K i , the joint task shares the decomposition vector U i 、V i and the task-specific decomposition vector P i Participate in training.

[0145] Perform Schmidt orthogonalization on S, A, and K, and freeze the skill codebook S corresponding to the first i-1 tasks (1,i-1), only train the skill encoding book S corresponding to the current task i , task sharing decomposition vector U i 、V i and the task-specific decomposition vector P i .

[0146] The core operation of Transformer is multi-head attention. For the calculation of one head of a multi-head self-attention, that is, h i , the input can be passed through three projection matrices Mapped into queries, keys, and values, the attention weight matrix of one head of the multi-head attention can be stacked to obtain a tensor Where D is embed s ize, that is, the dimension of embedding. According to the parameter efficient fine-tuning learning technology, the present disclosure can use the same set of mode approximation (MA) parameters to create a new learnable tensor Δw for learning new tasks and completing task migration. Specifically, the idea of ​​CANDECOMP / PARAFAC (CP) decomposition can be borrowed to decompose Δw into the sum of R rank-one components, each of which can be formalized as the outer product of three decomposition vectors, for example:

[0147]

[0148] in, is the decomposed vector of the rth component, each vector belongs to the corresponding mode matrix, such as u r It is U=[u1,…,u r ,…,u R ], represents the outer product, λ r is the coefficient scalar of each component, R is the rank of the CP decomposition, and for better understanding, each component is a tensor corresponding to the sum of scalar products:

[0149]

[0150] Among them, i, j, and k represent the subscripts of the three modes. U and V are global components shared by tasks, which can realize the transfer of knowledge between different tasks. P is a task-specific component, and each task has its own P. In addition, in order to further capture the characteristics of each task, the present disclosure can randomly initialize the learnable coefficient vector of all attention weight matrices in the i-th task Using these three mode components, when the input tensor is or t , the learnable tensor ΔW after CP decomposition can be incorporated into the calculation of multi-head attention during the forward propagation process:

[0151]

[0152] The present disclosure can apply the approximation of the attention-based weight matrix in the multi-head self-attention and multi-head cross-attention modules of the skill inference layer and the multi-head self-attention module of the action execution layer.

[0153] Reference Figure 2 , in operation 203, the loss is calculated based on the predicted action information of the robot and the actual action information of the robot in the expert demonstration data.

[0154] Specifically, the loss function of the entire robot skill learning deep network can be:

[0155]

[0156] Wherein, ξ is a weighted hyperparameter of the loss function, which can be set to 0.1 in this disclosure.

[0157] In operation 204, the imitation learning policy network and the skill codebook of the current task are updated based on the loss to perform a continuous skill learning process of the robot.

[0158] According to an exemplary embodiment of the present disclosure, when a preset termination condition is met, the most important parameters of the second preset ratio in the convolutional layer and the linear layer in the imitation learning strategy network can be frozen as specific parameters of the current task.

[0159] Specifically, the present disclosure proposes a robot skill continuous learning algorithm based on parameter isolation and pattern approximation, which can set the PackNet cropping object and percentage when initializing the skill encoding book S; for example, the PackNet cropping object can be set to the convolution layer and the linear layer, and the cropping percentage can be 75%, that is, 25% of the parameters of the corresponding part of the current entire neural network can be retained and frozen as parameters specific to the current task.

[0160] Next, we will introduce how to process the network structure according to the PackNet idea, that is, the specific process of trimming, fine-tuning and freezing some network parameters.

[0161] The PackNet network parameter mask can be initialized when the continuous learning algorithm is initialized. PackNet is a continuous learning algorithm based on a dynamic structure, which aims to prevent changes to parameters that are important for previous tasks in the continuous learning. To achieve this, PackNet can iteratively train, prune, fine-tune, and freeze parts of the network.

[0162] The pruning process in PackNet includes two stages. First, the parameters that can be used to learn the current task are determined based on the neural network parameter mask of task i, and then the network is trained on the task. At the end of the training, the top 25% of the most important parameters are selected from each convolutional layer and fully connected layer, and the remaining parameters are pruned for learning subsequent tasks. Then, in order to regain better performance on the task, only the retained parameters are used to perform learning training for the current task, that is, fine-tuning, and then freezing. In a specific implementation, the present disclosure may not train all bias and normalization layers, and the fine-tuning process can execute the same number of cycles as training (50 cycles).

[0163] Next, the entire interactive process of the robot skill learning problem disclosed in the present invention is introduced in detail.

[0164] Since the robot skill learning problem can be regarded as a finite-time Markov decision process This Markov decision process can be modeled as where S and A are the state-action space of the robot. μ0 is the initial state distribution, is the reward function, is the transfer function.

[0165] If you want to learn a robot strategy that can solve the corresponding task from visual and task instruction information, you can consider a sequential decision environment. At time step t=0, the text instruction i, the initial image observation x0, and the robot body information p0 are provided to the strategy π. The strategy π generates an action distribution π(·|i,x0,p0) based on this, and then samples the action a0 from the distribution and applies it to the robot. Finally, the transfer function allows the robot and the environment to enter the next state according to the current state and the action taken. This process is repeated, and the strategy π continuously changes from the learned distribution Sampling gets action a t The robot stops the loop iteration when the termination condition is reached (task completion or robot collides with the environment, etc.). The entire interaction process from the starting step t = 0 to the ending step t = T is i, It is called an episode or an expert demonstration data.

[0166] The present disclosure is based on the assumption of sparse reward setting, and can use the judgment value of whether the target completes a given task Instead of the reward function, the robot's goal can be to maximize the expectation of the reward function on the distribution of the state transition function with the starting state μ0:

[0167]

[0168] Then, based on the robot skill learning deep network at time t-1 and the robot image observation in the time period 0-t, the body posture information, the corresponding task text instructions and the real action at time t in the expert demonstration data, the network predicted robot action instructions at time t can be obtained, and the loss between the predicted action instructions and the real action instructions can be calculated. For example, the negative log-likelihood loss function can be used for calculation to update the robot skill learning deep network at time t.

[0169] definition For task T k N examples of l k ≤H, where o0 refers to the robot observation, including text instruction i, initial image observation x0 and robot body information p0 (joint and gripper information). t It does not have the Markov property, so based on the work of partially observable Markov processes, historical observations can be integrated to represent the current state s t ,Right now During the training process, the following objective function can be used for imitation learning:

[0170]

[0171] in, is a supervised learning loss function, which can be used as mentioned above π is a Gaussian mixture model, assuming that in the learning task T K The data of the first K-1 tasks cannot be obtained.

[0172] Let t=t+1, and repeat the above network update steps until the current training round is completed.

[0173] In addition, in the robot continuous learning problem, the robot uses a strategy π to successively perform K tasks T 1 ,…,T K Assume that π is task-conditioned, that is, π(2|s;T). For each task It is defined as the initial state distribution Target decision value g k and text instructions i k Assume that each task has the same H. It is the turn of task T k When , the robot's goal is to optimize the following formula:

[0174]

[0175] When learning the kth task, for the current task and the k-1 tasks that have been learned, the action suitable for the pth task at time t can be obtained based on the policy network π(·|s;T), which is the "deep network for robot skill learning based on hierarchical thinking and implicit skill codebook". The state obtained after interacting with the environment The reward function g for the pth task p The expected sum of all the values ​​at all moments is the largest.

[0176] Simply put, it is to search for a policy network that can maximize the reward function value of each task.

[0177] According to an exemplary embodiment of the present disclosure, a method for continuous learning of robot skills based on imitation learning is disclosed, which aims to give the robot arm a higher task completion capability and continuous learning capability for new tasks. The method may include the following steps:

[0178] Initialize the continuous learning task dataset, including a text instruction description of each task goal, observation images of the robot's wrist at different times when it completes the corresponding task, third-person observation images, and the robot's body state information at each time (joint posture information, torque, gripper state, etc.); select the task set type, task number, and task order;

[0179] Construct a hierarchical imitation learning strategy network, which can include a perception layer, a skill inference layer, and an action execution layer;

[0180] Initialize the continuous learning algorithm: initialize the current task skill codebook, PackNet network parameter mask, and CP decomposition vector;

[0181] Learning training task T i : Use imitation learning to train the network on the current task dataset;

[0182] End Task T i Learning: Use the early stopping method to end the learning of the current task, and then process and fine-tune the network structure according to the PackNet idea;

[0183] Let i=i+1 and continue learning the next task.

[0184] Among them, a hierarchical imitation learning strategy network is constructed. Specifically, it can be a robot skill learning deep network based on hierarchical thinking and implicit skill codebook, denoted as Hierarchy-TC

[0185] (Hierarchy-Transformer-CodeBook). This network can avoid the limitations of manually defined skills, making robot skill selection more flexible and the connection smoother.

[0186] Initialize the continuous learning algorithm, specifically, initialize the robot skill continuous learning algorithm based on parameter isolation and mode approximation, denoted as PackNet-MA (PackNet-Mode Approximation). This algorithm can achieve the effect of almost no forgetting, and can also use the similarity between tasks to improve the execution ability of previous tasks, and has certain deployment feasibility. It includes initializing the skill codebook S, the A and K matrices used to pay attention to the multimodal fusion information to obtain the weight vector, and the CP decomposition vector (task shared vectors U, V and task specific vector P) used to achieve mode approximation, and setting the PackNet clipping object and percentage.

[0187] Learning Task T i When S of the current task can be initialized i , A i , K i , the joint task shares the decomposition vector U i 、V i and the task-specific decomposition vector P i Participate in training.

[0188] Figure 3 A schematic diagram showing a flow chart of data processing by an imitation learning strategy network in an exemplary embodiment of the present disclosure.

[0189] Reference Figure 3 , showing the process of the imitation learning strategy network processing data at two moments. The input of each encoder is the original image observation (visual information) at the current moment, the robot body posture (state) information, and the task target text instruction corresponding to the task. The rectangle above each encoder represents the output of the encoder. Modal fusion is used to fuse text information and image information, that is, text vector and image representation. The fusion result is spliced ​​with the state representation and the text representation further encoded by the multi-layer perceptron to obtain multimodal fusion information, that is, the output of the perception layer.

[0190] The output of the perception layer is divided into two paths: one path is used to obtain the weight of the sub-skill vector of the sub-skill codebook through the attention module, and the weighted sub-skill vector is used for the multi-head cross attention of the skill inference layer; the other path is directly input to the high-level decoder, and after multi-head self-attention, its output is divided into three paths: one path is used as the query of multi-head cross attention, and the remaining two paths are spliced ​​with the weighted sub-skill vector to obtain the key and value. The query, key and value are used together for multi-head cross attention, and the output enters the feedforward network. After N layers of such operations, the skill vector at the current time t is output.

[0191] The specific role of attention is to perform Hadamard product on the multimodal fusion information and A, and then calculate the cosine similarity between the result and K. After that, the similarity can be used as a weight to select the skill vector from the sub-skill codebook to be spliced ​​into the h of the Transformer decoder in the skill inference layer. K and h V Above. The attention module is part of the skill inference layer.

[0192] The skill vector is input into the bottom decoder, and after passing through N layers of the same (multi-head self-attention + feedforward network) structure, its output enters the Gaussian mixture model distribution, and the action at the current moment can be obtained.

[0193] Among them, the mode approximation can be achieved through CP decomposition for model optimization. The above Δw can be decomposed into the sum of R rank-one components, each of which can be formalized as the outer product of three decomposition vectors P, V, and U, as follows Figure 3 As shown in the lower right corner, λ is the coefficient scalar for each component.

[0194] Finally, the loss can be calculated based on the predicted robot motion information and the real robot motion information in the expert demonstration data.

[0195] According to the exemplary embodiments of the present disclosure, the present disclosure selects skills from a skill code book through multimodal perception information, and then outputs action control based on the skills, thereby achieving the goal of online adjustment of decisions based on real-time perception information and smooth and coherent connection between skills, overcoming the problem that it is difficult to find an abstract hierarchical structure containing rich and meaningful sub-behaviors for artificially designed skills; making full use of the PackNet idea and the pattern approximation idea to create shared and task-specific parameters and skill libraries, effectively overcoming the forgetting of learned tasks, maintaining a good learning ability and having a high deployment feasibility.

[0196] According to an exemplary embodiment of the present disclosure, in order to explore the knowledge transfer ability and anti-forgetting performance of the agent in decision-making tasks, the present disclosure may adopt the following three evaluation indicators: forward transfer (FWT) is used to evaluate the ability of the agent to use prior knowledge to promote the learning of new tasks; negative backward transfer (NBT) is used to evaluate the agent's use of newly acquired knowledge to improve the performance of the model on previous tasks; the area under the success rate curve (AUC) is used to comprehensively evaluate the overall continuous learning ability of the agent. All indicators are calculated based on the success rate, because previous literature has shown that for operational strategies, the success rate is a more reliable indicator than the training loss. A lower NBT means that the strategy has better performance in previously seen tasks and has better anti-forgetting; a higher FWT means that the strategy learns faster on new tasks; and a higher AUC means that the overall performance of NBT and FWT is better. Specifically, define c i,j,e It represents the success rate of an agent that has learned the first i-1 tasks on the jth task after training for e rounds (e∈0,5,…,50) on the i-th task. Define c i,i is the highest success rate of the agent in all test rounds e of the current task i, Definition * is the earliest training round when the agent achieves the best performance on task i, And assume that for all training rounds For a different task j≠i, define To sum up, the specific calculation methods of the three evaluation indicators are as follows:

[0197]

[0198] The following are the existing methods used in the experiment:

[0199] In terms of continuous learning, this paper uses three representative algorithms as comparison objects, namely the representative method based on data replay - Experience Replay (ER), the representative method based on regularization - Elastic Weight Consolidation (EWC) and the representative method based on structure - PackNet. Experience Replay (ER) stores up to 1000 trajectories to avoid catastrophic forgetting. For each training iteration, this paper uniformly extracts 32 replay data trajectories from the memory and merges them with each batch of training data from the new task to participate in the training. This paper uses the online update version of EWC, and the loss function is This version uses an exponential moving average along the continuous learning process to update the Fisher information matrix. in k is the number of tasks. γ is set to 0.9 and λ is set to 5·10 4In the PackNet benchmark, this experiment sets the cropping ratio to 75% and uses the same fine-tuning period as the training period.

[0200] In terms of the robot skill learning strategy network, this paper uses the RESNET-RNN architecture and RESNET-T architecture provided by LIBERO as comparison objects. The former uses pre-trained ResNet as the visual backbone network to encode the visual observations of each time step, and uses LSTM as the temporal backbone network to process a series of encoded visual information. The language instructions are merged into the ResNet features through the FiLM technology to realize the fusion of text modality and visual modality, and then added to the LSTM input respectively; the latter uses a similar ResNet-based visual backbone network, a pre-trained BERT encoder and a Transformer decoder as the temporal backbone network to process visual word units and language word unit sequences.

[0201] In the actual experimental process, the selection of relevant parameters is as follows:

[0202] Hierarchy-TC embed s ize=64, the skill codebook A has N=100, storing 100 skill vectors, both Transformer decoders have 4 layers, each layer has 6 heads, and dropout=0.1. When used with the PackNet-MA continuous learning algorithm, embed s ize is 384. For each task, the agent is trained for a total of 50 rounds on 50 expert demonstration trajectories, evaluated every 5 rounds, and the agent is tested 20 times on the task, with a maximum number of steps of 600 for each test (if the agent cannot complete the task within 600 steps, the test is deemed to have failed), and the average success rate is recorded. The present disclosure can use the Adam optimizer with a batch size of 32, and use the cosine scheduler to adjust the learning rate during the training of each task, from 0.0001 at the beginning to 0.00001 in the end. After completing the training of a task, the model checkpoint with the highest success rate is selected as the final strategy for the current task, and then the agent is evaluated on all learned tasks, each task is tested 20 times, and the average success rate is recorded.

[0203] Table 1 records the continuous learning performance of three agents using different neural network architectures on four task sets. Experience replay (ER) is a classic, simple and efficient continuous learning algorithm, which is widely used as a benchmark algorithm. Therefore, this part of the simulation experiments uniformly selects experience replay as the continuous learning algorithm to compare the performance of the policy neural network.

[0204] Table 1

[0205]

[0206] As shown in Table 1, the AUC of RESNET-T and Hierarchy-TC are higher than that of RESNET-RNN on almost all datasets, which indicates that using Transformer may be better than using RNN model when processing "time" sequence. Hierarchy-TC has significantly higher AUC than RESNET-T on the three datasets except the long-view dataset, and has similar AUC on the long-view dataset, which indicates that the hierarchical architecture is more advantageous than the traditional flat strategy architecture in comprehensive ability. In terms of NBT indicators, Hierarchy-TC improves by about 17%, 16%, 34% and 17% on four datasets (from top to bottom in the table) compared with RESNET-T, and by about 1%, 18%, 18% and 18% compared with RESNET-RNN, especially on the object dataset (e.g., LIBERO-OBJECT) that only involves the single task goal of "pick up and put down", NBT is nearly doubled. This shows that the skill codebook can effectively store the atomic skills that the robot may use in completing different tasks. As the training process progresses, it is constantly revised, which helps the implicit modeling of robot skills and reuses them on different tasks to improve the anti-forgetting ability. In addition, the NBT of RESNET-RNN is significantly higher than that of Hierarchy-TC except for long-viewing distance data, indicating that the flat policy architecture is not suitable for the scenario of continuous learning of robots. However, in terms of FWT indicators, Hierarchy-TC and RESNET-T have their own advantages and are better than RESNET-RNN. Hierarchy-TC is better than RESNET-T on the target dataset, and they perform similarly on the spatial dataset, but RESNET-T is worse than Hierarchy-TC on the object dataset and the long-viewing distance dataset, indicating that Hierarchy-TC's forward transfer ability and the speed of learning new tasks are not as good as RESNET-T. This may be because the skill codebook does not reasonably isolate and reuse the skills used in previous tasks when learning new tasks, resulting in interference between tasks and affecting the plasticity of Hierarchy-TC.

[0207] Figure 4 The success rate of each task on the target dataset of the three strategy network architectures in the continuous learning process in an exemplary embodiment of the present disclosure is shown.

[0208] Reference Figure 4, the data points corresponding to the black plumb line represent the success rate on the kth task after training on the expert demonstration data of the kth task, and the continuous learning data points on the right represent the success rate of the agent on the kth task after learning the k+1, k+2...10th tasks. As can be seen from the figure, the success rate of RESNET-RNN in each continuous learning stage in all tasks is significantly lower than that of the other two strategy architectures, which once again proves that the learning ability and anti-forgetting performance of RNN are not satisfactory, and its ability to handle time series tasks is not as good as Transformer. It is worth noting that Hierarchy-TC has the highest success rate in each continuous learning stage in all tasks, especially Task 3, which has an improvement of about 50% compared with the other two strategy architectures. In addition, by observing the success rate curve of Task 1 ("open the drawer in the middle layer of the cabinet"), it was found that after learning the fourth task ("open the top drawer and put the bowl in"), the success rates of Hierarchy-TC and RESNET-T on Task 1 increased by about 30% and 18% respectively. Because both tasks contain the goal of "opening the drawer", this shows that the Transformer network can utilize the similarity between tasks and perform knowledge transfer to help improve the success rate of previous tasks, but RNN basically does not have this ability.

[0209] Table 2 shows the continuous learning performance of the four continuous learning algorithms. Since Hierarchy-TC performs best among all strategy architectures, all simulation experiments in this section use this neural network architecture.

[0210] Table 2

[0211]

[0212] As can be seen from Table 2, the PackNet-MA continuous learning algorithm proposed in this disclosure is superior to the existing classical algorithms in all evaluation indicators. In particular, compared with PackNet, this method has better forward transfer ability, and FWT is improved by about 16% and 9% respectively, which shows that the additional task-specific decomposition parameters effectively improve the learning ability of the policy network, enabling it to master new skills more quickly. In addition, the NBT index of this algorithm is improved by about 4.4% and 0.7% respectively compared with PackNet, and both are close to zero, indicating that the additional task-specific decomposition parameters can store the information of each task separately, and prevent subsequent tasks from affecting it by fine-tuning and freezing, which greatly alleviates catastrophic forgetting. In the long-viewing distance dataset, NBT even has negative numbers, which means that as the robot continues to learn new tasks, its success rate in completing previous tasks is also increasing, indicating that task-sharing decomposition parameters can effectively capture the similarity between tasks, and the introduction of skill coding books can also store atomic skills covered in a series of long-viewing distance tasks, reuse similar parts, and add and save different parts, which not only achieves the purpose of anti-forgetting, but also achieves the effect of using newly learned knowledge to help complete previous tasks. In addition, in terms of overall performance AUC, this method also has an improvement of approximately 14% and 11% compared to PackNet.

[0213] Figure 5 The success rate of each task of the three continuous learning algorithms on the target data set in the continuous learning process in an exemplary embodiment of the present disclosure is shown.

[0214] Reference Figure 5 The data points corresponding to the black plumb line represent the success rate of the agent on the kth task after training on the expert demonstration data of the kth task. The data points of the continuous learning process on the right represent the success rate of the agent on the kth task after learning the k+1th, k+2th, ... 10th tasks. Since the success rate of the EWC continuous learning algorithm is too low, it is not shown.

[0215] observe Figure 5It can be seen that the success rates of PackNet-MA and ER on the corresponding tasks after training on Task 2, Task 4, Task 6, Task 9 and Task 10 are significantly higher than that of PackNet, indicating that the plasticity of these two continuous learning algorithms is relatively strong, which once again confirms that they have good forward transfer capabilities throughout the continuous learning stage. This is because in the later stage of learning, after multiple prunings, there are very few parameters left for learning new tasks for PackNet. Even with prior knowledge of multiple tasks, it does not have the ability to complete new tasks. Secondly, on almost all task sets, the success rates of PackNet-MA and PackNet in the continuous learning process can be maintained near the success rate of the test just after learning the task, with only a slight fluctuation relative to ER, confirming that the dynamic architecture method has excellent anti-forgetting ability. It is worth noting that in the middle and late stages of continuous learning (such as Task 6 and Task 7), the success rate of PackNet-MA can be maintained at a high level, while the success rate of PackNet on these two tasks fluctuates greatly and decreases significantly, indicating that task-specific decomposition parameters help improve anti-forgetting. In addition, on Task 1, Task 5, Task 6, and Task 7, the success rate of PackNet-MA can continue to rise and remain at a high level as the continuous learning process continues to advance, which shows that the decomposition vectors shared between tasks can capture similar skills between different tasks, perform backward transfer of knowledge, and improve the success rate on previous tasks. But at the same time, it can be seen that in Task 4, after learning the 9th task, the success rate on Task 4 dropped significantly, while the success rates of the other two continuous learning algorithms were not affected, indicating that this task sharing may also cause interference between similar tasks.

[0216] Figure 6 A block diagram of a robot skill continuous learning device based on imitation learning in an exemplary embodiment of the present disclosure is shown.

[0217] Reference Figure 6 An exemplary embodiment of the present disclosure also provides a robot skill continuous learning device 600 based on imitation learning, which may include but is not limited to a task switching module 601, an information acquisition module 602, an action prediction module 603, a loss calculation module 604, and a parameter updating module 605.

[0218] The task switching module 601 can, for each of the multiple tasks in turn, loop through the operations in the information acquisition module 602, the action prediction module 603, the loss calculation module 604, and the parameter updating module 605 before a preset termination condition is met.

[0219] The information acquisition module 602 can acquire the observed image and state information of the robot at the current moment.

[0220] The action prediction module 603 can input the observed image, state information, pre-acquired text instructions of the current task, and the skill code book of the current task into the imitation learning strategy network to predict the action information of the robot at the current moment, wherein the skill code book of the current task is a plurality of pre-initialized learnable sub-skill vectors, and in the case of a previous task, contains the skill information of the skill code book of the previous task.

[0221] The loss calculation module 604 may calculate the loss based on the predicted motion information of the robot and the actual motion information of the robot in the expert demonstration data.

[0222] The parameter updating module 605 can update the imitation learning strategy network and the skill codebook of the current task based on the loss to execute the robot's continuous skill learning process.

[0223] According to an exemplary embodiment of the present disclosure, the imitation learning strategy network includes a perception layer, a skill inference layer and an action execution layer; wherein the action prediction module 603 can input the observed image, state information and text instructions into the perception layer to obtain the multimodal fusion information at the current moment; input the multimodal fusion information and the skill code book into the skill inference layer to predict the skill vector at the current moment; input the skill vector into the action execution layer to obtain the action information of the robot at the current moment.

[0224] According to an exemplary embodiment of the present disclosure, the perception layer includes a pre-trained text encoder, a learnable multi-layer perceptron, a pre-trained learnable visual encoder, a learnable state encoder, and a modal fusion module; wherein the action prediction module 603 can input text instructions into the text encoder to obtain a text vector; input the text vector into the multi-layer perceptron to obtain a text representation; input the observed image into the visual encoder to obtain an image representation; input the state information into the state encoder to obtain a state representation; after fusing the text vector and the image representation through the modal fusion module, the fused features are spliced ​​with the text representation and the state representation to obtain multimodal fusion information.

[0225] According to an exemplary embodiment of the present disclosure, the skill inference layer includes an attention module and a high-level decoder, and the high-level decoder includes a multi-head self-attention layer, a multi-head cross-attention layer and a feedforward network layer; wherein, the action prediction module 603 can input multimodal fusion information into the attention module to obtain weights for multiple learnable sub-skill vectors in the skill coding book; weight the multiple learnable sub-skill vectors based on the weights to obtain multiple learnable weighted sub-skill vectors; input the multimodal fusion information into the multi-head self-attention layer to obtain observation information containing timing information; input the features obtained by concatenating the weighted sub-skill vectors and the observation information, and the observation information into the multi-head cross-attention layer, and input the output of the multi-head cross-attention layer into the feedforward network layer to predict the skill vector.

[0226] According to an exemplary embodiment of the present disclosure, the action prediction module 603 can use the features obtained by splicing the weighted sub-skill vectors and the observation information as the keys and values ​​of the multi-head cross-attention layer, and use the observation information as the query of the multi-head cross-attention layer to perform multi-head cross-attention.

[0227] According to an exemplary embodiment of the present disclosure, the action execution layer includes an underlying decoder and a Gaussian mixture model distribution layer, and the underlying decoder includes a multi-head self-attention layer and a feedforward network layer; wherein the action prediction module 603 can process the skill vector through the multi-head self-attention layer, the feedforward network layer and the Gaussian mixture model distribution layer to obtain the action information of the robot at the current moment.

[0228] According to an exemplary embodiment of the present disclosure, for the calculation of each head in the multi-head self-attention layer and the multi-head cross-attention layer in the skill inference layer, and the multi-head self-attention layer in the action execution layer, the pattern approximation of the current head is achieved through the operation of a pattern approximation module (not shown in the figure); wherein the pattern approximation module can map the input of the current head into query, key and value through three projection matrices; the weight matrix of the current head is stacked to obtain a first tensor; the second tensor is decomposed into the sum of at least one rank-one component using CP decomposition, wherein the second tensor is a learnable tensor for the current task pre-created using pattern approximation parameters, and the components are obtained based on multiple shared components between multiple tasks, specific components of the current task, and a learnable coefficient vector pre-randomly initialized for the current task; based on the query, key and value, the first tensor, and the second tensor after CP decomposition, the calculation of the current head is performed.

[0229] According to an exemplary embodiment of the present disclosure, the preset termination condition is to test the success rate of the imitation learning strategy network on the current task at a preset number of training checkpoints until a preset number of termination training times is reached, or after the success rate is higher than a first preset ratio, the success rate obtained by subsequent consecutive preset number of checkpoint tests is lower than the first preset ratio; when the preset termination condition is met, the training of the current task is terminated, and the model parameters of the imitation learning strategy network with the highest success rate are saved.

[0230] According to an exemplary embodiment of the present disclosure, the robot skill continuous learning device 600 based on imitation learning may also include but is not limited to: a parameter isolation module (not shown in the figure), which can freeze the most important parameters of the second preset ratio in the convolutional layer and the linear layer in the imitation learning strategy network as specific parameters of the current task when the preset termination conditions are met.

[0231] It is understandable that in the exemplary embodiment of the above-mentioned robot skill continuous learning device 600 based on imitation learning, the specific implementation process is roughly the same as the exemplary embodiment of the above-mentioned robot skill continuous learning method based on imitation learning, and will not be repeated here. The robot skill continuous learning device 600 based on imitation learning can be configured as software, hardware, firmware or any combination of the above items to perform specific functions. For example, these devices may correspond to dedicated integrated circuits, pure software codes, or modules that combine software and hardware. In addition, one or more functions implemented by these devices may also be uniformly executed by components in physical entity devices (for example, processors, clients or servers, etc.).

[0232] Figure 7 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure.

[0233] Reference Figure 7 The electronic device 700 includes at least one memory 701 and at least one processor 702, wherein the at least one memory 701 stores a set of computer executable instructions. When the set of computer executable instructions is executed by the at least one processor 702, the method for continuous learning of robot skills based on imitation learning according to the exemplary embodiment of the present disclosure is executed.

[0234] As an example, the electronic device 700 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instruction set. Here, the electronic device 700 is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above instruction (or instruction set) individually or in combination. The electronic device 700 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0235] In the electronic device 700, the processor 702 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0236] The processor 702 may execute instructions or codes stored in the memory 701, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.

[0237] The memory 701 may be integrated with the processor 702, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 701 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 701 and the processor 702 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 702 can read files stored in the memory.

[0238] In addition, the electronic device 700 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 700 may be connected to each other via a bus and / or a network.

[0239] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein, when the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the above-mentioned method for continuous learning of robot skills based on imitation learning.

[0240] Examples of computer-readable storage media include read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, nonvolatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers. It should be noted that the instructions can also be used to execute additional steps in addition to the above steps or to perform more specific processing when executing the above steps. These additional steps and further processing contents have been mentioned in the description of the relevant methods, so they will not be repeated here to avoid repetition.

[0241] Another embodiment of the present disclosure relates to a system comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, prompt the at least one computing device to execute the above-mentioned method for continuous learning of robot skills based on imitation learning.

[0242] It should be noted that the system according to the exemplary embodiments of the present disclosure can completely rely on the execution of computer programs or instructions to realize corresponding functions, that is, each unit corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (e.g., lib library) to realize the corresponding functions.

[0243] On the other hand, when the above-mentioned system is implemented in software, firmware, middleware or microcode, the program code or code segment for performing the corresponding operation can be stored in a computer-readable medium such as a storage medium, so that at least one processor or at least one computing device can perform the corresponding operation by reading and running the corresponding program code or code segment.

[0244] According to an exemplary embodiment of the present disclosure, the storage device may be integrated with the computing device, for example, RAM or flash memory is arranged within an integrated circuit microprocessor, etc. In addition, the storage device may include an independent device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The storage device and the computing device may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the computing device can read instructions stored in the storage device.

[0245] Another embodiment of the present disclosure relates to a computer program product, including a computer program / instruction, which, when executed by a processor, implements the method for continuous learning of robot skills based on imitation learning as described in any one of the above.

[0246] According to the robot skill continuous learning method and device based on imitation learning provided by the present invention, by introducing a skill code book, a sub-skill library is created for each task separately, and through network training, the sub-skills used in each task are captured. When executing the current task, the sub-skill vector can also be selected from the skill code book of the previous task, thereby realizing sub-skill reuse, so that the robot's actions when executing long sequence (more complex or involving multiple targets) tasks are more logical, smoother, and more coherent.

[0247] In addition, by introducing a hierarchical architecture, the robot can adjust its decisions online based on real-time perception information when performing long-sequence tasks (more complex or involving multiple goals), and learn a reasonable skill abstraction level. Compared with other strategy architectures, it has better forward transfer capabilities and a higher average task success rate.

[0248] In addition, after learning each task, the main network is pruned and fine-tuned to freeze the previous task parameters. Task-specific parameters (sub-skill library and task-specific decomposition vector) and task-shared parameters are created for each task separately, giving the robot the ability to discover the same atomic skills implicit in different tasks. Compared with existing continuous learning algorithms, it has better backward transfer capabilities and anti-forgetting properties.

[0249] In addition, on long-viewing-distance datasets, it is possible to effectively capture similarities between tasks (such as the skills used or information about the environment layout), transfer the knowledge gained from learning new tasks, and improve the success rate of completing previous tasks.

[0250] The above describes various exemplary embodiments of the present disclosure, and it should be understood that the above description is only exemplary and not exhaustive, and the present disclosure is not limited to the disclosed exemplary embodiments. Without departing from the scope and spirit of the present disclosure, many modifications and changes are obvious to those of ordinary skill in the art. Therefore, the scope of protection of the present disclosure should be based on the scope of the claims.

Claims

1. A robot skills continuous learning method based on imitation learning, characterized in that: The method comprises: For each of the multiple tasks in turn, loop and perform the following operations until a preset termination condition is met: Get the robot's observation image and status information at the current moment; Inputting the observed image, the state information, the pre-acquired text instructions of the current task, and the skill codebook of the current task into the imitation learning strategy network, predicting the action information of the robot at the current moment, wherein the skill codebook of the current task is a plurality of pre-initialized learnable sub-skill vectors, and in the case of a previous task, contains the skill information of the skill codebook of the previous task; Calculating the loss based on the predicted motion information of the robot and the real motion information of the robot in the expert demonstration data; The imitation learning strategy network and the skill codebook of the current task are updated based on the loss to execute the continuous skill learning process of the robot.

2. The method for continuous learning of robot skills based on imitation learning as claimed in claim 1, characterized in that: The imitation learning strategy network includes a perception layer, a skill inference layer and an action execution layer; The step of inputting the observed image, the state information, the pre-acquired text instructions of the current task, and the skill codebook into the imitation learning strategy network to predict the action information of the robot at the current moment includes: Inputting the observed image, the state information and the text instruction into the perception layer to obtain multimodal fusion information at the current moment; Inputting the multimodal fusion information and the skill codebook into the skill inference layer to predict the skill vector at the current moment; The skill vector is input into the action execution layer to obtain the action information of the robot at the current moment.

3. The method for continuous learning of robot skills based on imitation learning as claimed in claim 2, characterized in that: The perception layer includes a pre-trained text encoder, a learnable multi-layer perceptron, a pre-trained learnable visual encoder, a learnable state encoder, and a modality fusion module; The step of inputting the observed image, the state information and the text instruction into the perception layer to obtain the multimodal fusion information at the current moment includes: Inputting the text instruction into the text encoder to obtain a text vector; Inputting the text vector into the multi-layer perceptron to obtain text representation; Inputting the observed image into the visual encoder to obtain an image representation; Inputting the state information into the state encoder to obtain a state representation; After the text vector and the image representation are fused by the modal fusion module, the fused features are spliced ​​with the text representation and the state representation to obtain the multimodal fusion information.

4. The method for continuous learning of robot skills based on imitation learning as claimed in claim 2, characterized in that: The skill inference layer includes an attention module and a high-level decoder, and the high-level decoder includes a multi-head self-attention layer, a multi-head cross-attention layer, and a feedforward network layer; The step of inputting the multimodal fusion information and the skill codebook into the skill inference layer to predict the skill vector at the current moment includes: Inputting the multimodal fusion information into the attention module to obtain weights for a plurality of learnable sub-skill vectors in the skill codebook; Weighting the multiple learnable sub-skill vectors based on the weights to obtain multiple learnable weighted sub-skill vectors; Inputting the multimodal fusion information into the multi-head self-attention layer to obtain observation information containing time series information; The features obtained by concatenating the weighted sub-skill vector and the observation information, and the observation information are input into the multi-head cross-attention layer, and the output of the multi-head cross-attention layer is input into the feedforward network layer to predict the skill vector.

5. The method for continuous learning of robot skills based on imitation learning as claimed in claim 4, characterized in that: The step of concatenating the weighted sub-skill vector and the features obtained by the observation information, and inputting the observation information into the multi-head cross attention layer includes: The features obtained by concatenating the weighted sub-skill vector and the observation information are used as the key and value of the multi-head cross-attention layer, and the observation information is used as the query of the multi-head cross-attention layer to perform multi-head cross-attention.

6. The method for continuous learning of robot skills based on imitation learning as claimed in claim 4, characterized in that: The action execution layer includes a bottom layer decoder and a Gaussian mixture model distribution layer, and the bottom layer decoder includes a multi-head self-attention layer and a feedforward network layer; The step of inputting the skill vector into the action execution layer to obtain the action information of the robot at the current moment includes: The skill vector is processed by the multi-head self-attention layer, the feedforward network layer and the Gaussian mixture model distribution layer to obtain the action information of the robot at the current moment.

7. The method for continuous learning of robot skills based on imitation learning as claimed in claim 6, characterized in that: For the calculation of each head in the multi-head self-attention layer and the multi-head cross-attention layer in the skill inference layer, and the multi-head self-attention layer in the action execution layer, the mode approximation of the current head is achieved by the following operations: Mapping the input of the current head into query, key and value through three projection matrices; Stacking the weight matrices of the current head to obtain a first tensor; Decomposing the second tensor into a sum of at least one rank-one component using CP decomposition, wherein the second tensor is a learnable tensor for the current task that is pre-created using a pattern approximation parameter, and the component is obtained based on a plurality of shared components between the plurality of tasks, a specific component of the current task, and a learnable coefficient vector that is pre-randomly initialized for the current task; Based on the query, the key and value, the first tensor, and the CP-decomposed second tensor, a computation of the current head is performed.

8. The method for continuous learning of robot skills based on imitation learning as claimed in claim 1, characterized in that: The preset termination condition is to test the success rate of the imitation learning strategy network on the current task at a preset number of training checkpoints until a preset number of termination training times is reached, or after the success rate is higher than a first preset ratio, the success rate obtained by subsequent consecutive preset number of checkpoint tests is lower than the first preset ratio; if the preset termination condition is met, the training of the current task is terminated, and the model parameters of the imitation learning strategy network with the highest success rate are saved.

9. The method for continuous learning of robot skills based on imitation learning as claimed in claim 1, characterized in that: The method further comprises: When the preset termination condition is met, the most important parameters of the second preset ratio in the convolutional layer and the linear layer in the imitation learning strategy network are frozen as specific parameters of the current task.

10. A robot skill continuous learning device based on imitation learning, characterized in that: The device comprises: The task switching module is configured to: for each of the multiple tasks, cyclically execute the operations in the information acquisition module, the action prediction module, the loss calculation module, and the parameter updating module before a preset termination condition is met; The information acquisition module is configured to: acquire the observed image and state information of the robot at the current moment; The action prediction module is configured to: input the observed image, the state information, the pre-acquired text instructions of the current task, and the skill codebook of the current task into the imitation learning strategy network, and predict the action information of the robot at the current moment, wherein the skill codebook of the current task is a plurality of pre-initialized learnable sub-skill vectors, and contains the skill information of the skill codebook of the previous task in the case of a previous task; The loss calculation module is configured to: calculate the loss based on the predicted action information of the robot and the real action information of the robot in the expert demonstration data; The parameter updating module is configured to update the imitation learning strategy network and the skill codebook of the current task based on the loss to execute the continuous skill learning process of the robot.

Images