Double-arm operation imitation learning method and device and storage medium

Abstract

The invention provides a two-arm operation imitation learning method and device, a storage medium, electronic equipment and a computer program product. The method comprises the following steps: obtaining human two-arm operation demonstration data and robot two-arm operation demonstration data; performing space-time alignment on the human double-arm operation demonstration data and the robot double-arm operation demonstration data to obtain human double-arm operation demonstration data after space-time alignment and robot double-arm operation demonstration data after space-time alignment; and on the basis of the human double-arm operation demonstration data after space-time alignment and the robot double-arm operation demonstration data after space-time alignment, track data and action control data of double-arm operation are acquired through a pre-trained imitation learning network. According to the technical scheme, the defects of poor data diversity and weak generalization ability are overcome while the data acquisition cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an imitation learning method, device and storage medium for dual-arm operation, as well as an electronic device and a computer program product. Background Art

[0002] As one of the core difficulties in robot operation tasks, dual-arm operation requires the two arms to be highly coordinated in time and space to complete complex tasks such as pouring water, bagging, and assembling. In recent years, imitation learning has become the mainstream solution. By collecting demonstration data of the robot performing dual-arm operation and using the above demonstration data to train a vision-driven policy model, a high-precision dual-arm operation learning task can be completed. However, for the above demonstration data of the robot performing dual-arm operation, it is necessary to control the master-slave manipulator actions by professional operators to collect the demonstration data during the dual-arm operation process.

[0003] The acquisition cost of the above demonstration data of the robot performing dual-arm operation is relatively high. And due to the limitation of the acquisition cost, it is difficult to cover different scenarios, objects, and action types, and there are also defects in poor data diversity, resulting in weak generalization ability. Summary of the Invention

[0004] The present invention provides an imitation learning method, device and storage medium for dual-arm operation, as well as an electronic device and a computer program, to solve the problems of high acquisition cost of the demonstration data of the robot performing dual-arm operation, poor data diversity, and weak generalization ability, and to achieve reducing the data acquisition cost while also solving the problems of poor data diversity and weak generalization ability.

[0005] The present invention provides an imitation learning method for dual-arm operation, including: Obtaining human dual-arm operation demonstration data and robot dual-arm operation demonstration data; Performing spatio-temporal alignment on the human dual-arm operation demonstration data and the robot dual-arm operation demonstration data to obtain the spatio-temporally aligned human dual-arm operation demonstration data and the spatio-temporally aligned robot dual-arm operation demonstration data; Based on the spatio-temporally aligned human dual-arm operation demonstration data and the spatio-temporally aligned robot dual-arm operation demonstration data, obtaining the trajectory data and action control data of the dual-arm operation through a pre-trained imitation learning network; Wherein, the imitation learning network is pre-trained according to human dual-arm operation demonstration data samples, robot dual-arm operation demonstration data samples, and corresponding trajectory data labels and action control data labels of the dual-arm operation.

[0006] In some embodiments, the above-mentioned human double-arm operation demonstration data includes first demonstration video data and first demonstration trajectory data, and the robot double-arm operation demonstration data includes first demonstration video data and second demonstration trajectory data; The spatio-temporal alignment of the human double-arm operation demonstration data and the robot double-arm operation demonstration data includes: Performing time alignment using the key frames of the first demonstration video data and the key frames of the second demonstration video data; and, Performing spatial coordinate normalization on the first demonstration trajectory data and the second demonstration trajectory data.

[0007] In some embodiments, before performing time alignment using the key frames of the first demonstration video data and the key frames of the second demonstration video data, it further includes: Using the entity masking technique to mask the background area in the first demonstration video data and retain the image of the two hands area; and using the entity masking technique to mask the background area in the second demonstration video data and retain the image of the end effector area.

[0008] In some embodiments, the first demonstration video data includes self-view video data, and the second demonstration video data includes double-wrist view video data; the first demonstration trajectory data includes hand trajectory data, and the second demonstration trajectory data includes end effector trajectory data.

[0009] In some embodiments, the above-mentioned imitation learning network includes a vision-driven policy network and a prediction head; Based on the spatio-temporally aligned human double-arm operation demonstration data and the spatio-temporally aligned robot double-arm operation demonstration data, obtaining the trajectory data and action control data of the double-arm operation through a pre-trained imitation learning network, includes: Through the vision-driven policy network, performing feature extraction based on the spatio-temporally aligned human double-arm operation demonstration data to obtain human double-arm operation feature data; and performing feature extraction based on the spatio-temporally aligned robot double-arm operation demonstration data to obtain robot double-arm operation feature data; Through the prediction head, performing prediction based on the human double-arm operation feature data and the robot double-arm operation feature data to predict the trajectory data of the double-arm operation and the action control data of the double-arm operation.

[0010] In some embodiments, the above-mentioned prediction head includes a trajectory data prediction head and a control data prediction head; Based on the human double - arm operation feature data and the robot double - arm operation feature data, prediction is performed through the prediction head to predict the trajectory data and the motion control data of the double - arm operation, including: Through the trajectory data prediction head, trajectory prediction is performed based on the human double - arm operation feature data and the robot double - arm operation feature data to predict the trajectory data of the double - arm operation; Through the control data prediction head, control prediction is performed based on the robot double - arm operation feature data to predict the motion control data of the double - arm operation.

[0011] In some embodiments, the above - mentioned imitation learning network is also provided with a loss function; the loss function includes: A trajectory loss sub - function for calculating the trajectory prediction error based on the predicted trajectory data; An action control loss sub - function for calculating the action control prediction error based on the predicted motion control data.

[0012] In some embodiments, it further includes: In the pre - training stage, the loss value of the loss function is calculated according to the trajectory prediction error, the action control prediction error, and a preset adjustment value; The parameters of the visual driving policy network and the prediction head are adjusted according to the loss value of the loss function.

[0013] In some embodiments, the above - mentioned visual policy network includes an action chunk transformer (ACT) network or a diffusion policy (DP) network.

[0014] The embodiment of the present application also provides an imitation learning device for double - arm operation, including: A data acquisition module for acquiring human double - arm operation demonstration data and robot double - arm operation demonstration data; A spatio - temporal alignment module for spatio - temporally aligning the human double - arm operation demonstration data and the robot double - arm operation demonstration data to obtain the spatio - temporally aligned human double - arm operation demonstration data and the spatio - temporally aligned robot double - arm operation demonstration data; An imitation learning module for obtaining the trajectory data and the motion control data of the double - arm operation through a pre - trained imitation learning network based on the spatio - temporally aligned human double - arm operation demonstration data and the spatio - temporally aligned robot double - arm operation demonstration data; Wherein, the imitation learning network is pre - trained according to human double - arm operation demonstration data samples, robot double - arm operation demonstration data samples, and the corresponding trajectory data labels and motion control data labels of the double - arm operation.

[0015] An embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and running on the processor. When the processor executes the computer program, the imitation learning method for dual-arm operation as described above is implemented.

[0016] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the imitation learning method for dual-arm operation as described above is implemented.

[0017] An embodiment of the present application also provides a computer program product. When the computer program is executed by a processor, the imitation learning method for dual-arm operation as described above is implemented.

[0018] The imitation learning method for dual-arm operation provided by the present invention can perform imitation learning by simultaneously using human dual-arm operation demonstration data and robot dual-arm operation demonstration data, enabling the simultaneous utilization of the respective advantages of the above two types and avoiding the disadvantages when using only one type of data. It can learn general trajectory expressions from human demonstration data and embodied motion control from robot demonstration data, maintain a robust execution ability in complex scenarios, and implement an efficient and scalable hybrid learning mechanism for dual-arm operation strategies. This hybrid learning mechanism can solve the problems of high cost in collecting demonstration data for robot dual-arm operation, poor data diversity, and weak generalization ability, achieving the reduction of data collection costs while also solving the problems of poor data diversity and weak generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0020] Figure 1 is a flowchart of the imitation learning method for dual-arm operation provided by an embodiment of the present invention; Figure 2 is Figure 1 a specific execution flowchart of step 102 shown; Figure 3 is Figure 1 a specific execution flowchart of step 103 shown; Figure 4 is a structural diagram of a dual-arm operation imitation learning framework provided by an embodiment of the present application; Figure 5 is a schematic diagram for comparing the technical effects of the embodiments of the present applicationFigure 1 ; Figure 6 Schematic diagram for comparing the technical effects of the embodiments of the present application Figure 2 ; Figure 7 Schematic diagram for comparing the technical effects of the embodiments of the present application Figure 3 ; Figure 8 Schematic structural diagram of the imitation learning device with dual-arm operation provided by the embodiments of the present invention; Figure 9 Schematic structural diagram of an electronic device provided by the embodiments of the present invention. Detailed implementation manners

[0021] In order to make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

[0022] As described in the background art section, in the related art, in order to implement the dual-arm operation of a robot, professional personnel will operate and demonstrate the dual-arm operation of the robot to obtain demonstration data of the robot's dual-arm operation, and then further train a visual drive strategy model according to the demonstration data of the robot's operation to complete a high-precision dual-arm operation learning task. However, on the one hand, there are defects in the relatively high acquisition cost. The teleoperation equipment is expensive and the setup is complex, and the data acquisition efficiency is low. For example, the average acquisition time for each piece of demonstration video data exceeds 2 minutes. On the other hand, due to the limitation of the acquisition cost, it is difficult to cover different scenarios, objects and action types, so there are also defects in poor data diversity, which leads to the fact that the model training relies heavily on demonstrations under specific scenarios / objects and is difficult to be generalized to new tasks or changing environments, resulting in weak generalization ability of the model.

[0023] In addition, in some related technologies, it is also possible to use human two-arm operation demonstration data to replace the above-mentioned robot's two-arm operation demonstration data for training the vision-driven policy model. The collection efficiency of human two-arm operation demonstration data is high, thereby reducing its collection cost. For example, for each segment of human two-arm operation demonstration video data on average, its collection time can be only 1 / 6 of that of the robot collecting demonstration video data. Moreover, human two-arm operation demonstration data also has the advantage of clear intention expression. Humans naturally have a goal-oriented gaze and action pattern, which makes human two-arm operation demonstration videos focus on the interactive content of actions. Additionally, human two-arm operation demonstration data also has the advantage of wide coverage of scenarios. The wearable devices used have the characteristic of being lightweight, facilitating the collection of diverse and natural human operation behaviors. Methods in related technologies, such as Vid2Robot, RAD, R+X, YOTO, etc., propose to extract visual intentions, operation strategies, language plans, etc. from human demonstration video data and use them in the imitation learning of robot two-arm operations.

[0024] However, human two-arm operation demonstration data still has two limitations. One is the neglect of fine-grained low-level control transfer. For example, most methods mainly focus on extracting high-level abstract strategies, such as object recognition or intention understanding, but fail to effectively utilize the fine manipulation actions in human demonstration videos, such as trajectory patterns, pose dynamics, etc. Therefore, it is difficult to improve the refined control performance. The other is the failure to solve the embodiment differences between humans and robots. There are significant differences in the geometric structure and perception mechanism between human hands and robot grippers. Directly transferring operation actions will result in execution results that violate expectations. In addition, there are also large differences between human demonstration video data and robot execution conditions in terms of shooting angle, action granularity, and movement speed, resulting in the difficulty of directly using the features extracted from human demonstration video data in the robot's two-arm operation tasks. Thus, although related technologies have proposed using human two-arm operation demonstration data for the imitation learning of robot two-arm operations, there are still some inevitable technical problems. The embodiments of the present invention provide an imitation learning method for robot two-arm operations, in which robot two-arm operation demonstration data and human two-arm operation demonstration data are combined for hybrid imitation learning, so as to overcome the defects when using either one for imitation learning. The technical solutions in the embodiments of the present invention will be described below with reference to the accompanying drawings in the embodiments of the present invention.

[0025] Figure 1 It is a schematic flowchart of the imitation learning method for two-arm operations provided by the embodiments of the present invention. This imitation learning method for two-arm operations can be applied to various electronic devices to run, so as to realize the imitation learning of robot two-arm operations. Specifically, the method includes the following steps: Step 101: Obtain the human double-arm operation demonstration data and the robot double-arm operation demonstration data; As described above, the embodiment of the present application provides a hybrid imitation learning method, which requires simultaneous use of the human double-arm operation demonstration data and the robot double-arm operation demonstration data for imitation learning. Therefore, both types of data are obtained simultaneously in this step.

[0026] The human double-arm operation demonstration data can be collected by wearable devices and can include two types: demonstration video data and demonstration trajectory data. The demonstration video data can be self-visual video data, and the demonstration trajectory data can be hand trajectory data, which can be collected by the same wearable device or by two wearable devices respectively. For the robot double-arm operation demonstration data, it can be the double-arm teaching data collected by a professional operator controlling the master-slave manipulator. Representative methods can include ALOHA, Gello, etc. to generate high-quality demonstration data, which can also include two types: demonstration video data and demonstration trajectory data. The demonstration video data can include the video data collected by the double-wrist camera and the self-view video data. At the same time, the demonstration trajectory data can be the end effector trajectory data.

[0027] Both of the above two types of demonstration data can be collected in advance by the corresponding devices and stored in the database, and then retrieved from the above database when imitation learning is required.

[0028] Step 102: Align the human double-arm operation demonstration data and the robot double-arm operation demonstration data in space and time to obtain the human double-arm operation demonstration data after space-time alignment and the robot double-arm operation demonstration data after space-time alignment; Based on the above two types of demonstration data, the space-time alignment is mainly because the double-arm operation is collected by different types of devices, and there are certain differences in the execution speed, data collection position, angle, etc. of the double-arm operation. Therefore, before inputting the above two types of demonstration data into the imitation learning network for imitation learning, the two types of data can be aligned in space and time in advance to make them consistent in time and space, so as to avoid adverse effects on the subsequent imitation learning accuracy.

[0029] The space-time alignment in this step can include two aspects of processing: time alignment and spatial coordinate normalization.

[0030] Step 103: Based on the human double-arm operation demonstration data after space-time alignment and the robot double-arm operation demonstration data after space-time alignment, obtain the trajectory data and action control data of the double-arm operation through a pre-trained imitation learning network; Among them, the above-mentioned imitation learning network is pre-trained according to human double-arm operation demonstration data samples, robot double-arm operation demonstration data samples, and corresponding trajectory data labels and action control data labels of double-arm operations.

[0031] In the above steps, after obtaining human double-arm operation demonstration data and robot double-arm operation demonstration data and performing spatio-temporal alignment, the pre-trained imitation learning network can be used for imitation learning. Specifically, the trajectory data and action control data of double-arm operations are obtained. The trajectory data can include the specific operation trajectories of the end effectors of the robot's double arms. Through this trajectory data, a general operation strategy can be learned; while the action control data is the control signal for the robot's double arms. Through this action control data, an accurate control signal for the robot's double arms can be learned.

[0032] In the embodiments of the present invention, sample data such as human double-arm operation demonstration data samples, robot double-arm operation demonstration data samples, etc. can be used, and then the internal parameters of the imitation learning network are adjusted through pre-training according to the above sample data and the corresponding trajectory data labels and action control data labels of double-arm operations until the output of the imitation learning network can meet certain accuracy requirements.

[0033] The technical solution for the robot to perform double-arm operation imitation learning provided by the embodiments of the present application uses both human double-arm operation demonstration data and robot double-arm operation demonstration data for imitation learning, so that the respective advantages of the above two types can be utilized simultaneously, and the disadvantages when using only one type of data can be avoided. It can learn general trajectory expressions from human demonstration data and embodied action control from robot demonstration data, can maintain a robust execution ability in complex scenarios, and realize an efficient and scalable hybrid learning mechanism for double-arm operation strategies.

[0034] This hybrid learning mechanism enables human demonstration data to contribute high-quality general operation modes, while the machine's demonstration videos provide necessary embodied control signals. The hybrid learning mechanism significantly improves the generalization performance while reducing data costs. Specifically, it can be a hybrid learning mechanism mainly based on human demonstration data and supplemented by robot demonstration data, that is, the advantages of low data cost and high acquisition efficiency of human demonstration data can be fully utilized to collect diverse and natural human operation behaviors; various scenarios, objects, and action types can be collected, thereby improving the diversity of demonstration data, and the problem of weak generalization ability of the imitation learning network can also be solved according to the above diverse demonstration data. In addition, by using robot double-arm operation demonstration data, embodied action data can be provided, thus solving the problem of embodied differences between humans and machines that exists when only using human double-arm operation demonstration data.

[0035] In some embodiments, the human double-arm operation demonstration data may include first demonstration video data and first demonstration trajectory data, and the robot double-arm operation demonstration data may include first demonstration video data and second demonstration trajectory data. Moreover, for the above-mentioned first demonstration video data, it may include self-view video data, while for the second demonstration video data, it may include double-wrist view video data and self-view video data; for the first demonstration trajectory data, it may include hand trajectory data, and for the second demonstration trajectory data, it may include end-effector trajectory data, and the end-effector may include a gripper.

[0036] At this time, for step 102 in the above Figure 1 illustrated embodiment, it may include two steps of performing time alignment on the video data and performing spatial coordinate normalization on the trajectory data. Specifically, reference may be made to Figure 2 as shown, which may include: Step 1021: Perform time alignment using the key frames of the first demonstration video data and the key frames of the second demonstration video data; In this step, based on the spatial trajectory change, redundant frames may be filtered out from the above-mentioned first demonstration video data of humans first, so as to extract key frames, and redundant frames may also be filtered out from the second demonstration video data of the robot, so as to also extract key frames, so that the two have the same task-phase anchor points, and then time alignment is performed, so as to make the two types of videos consistent in the time dimension and effectively align the double-arm operation task process.

[0037] Step 1022: Perform spatial coordinate normalization on the first demonstration trajectory data and the second demonstration trajectory data.

[0038] In this step, mainly the trajectory representations of the human demonstration trajectory data and the robot demonstration trajectory data are unified to construct a unified spatial representation space. Specifically, the spatial trajectories of the human hand trajectory data and the end-effector trajectory data in the world coordinate system may be uniformly mapped to a unified first-view coordinate system and normalized, so as to reduce the difference in the representation distribution.

[0039] In the embodiments of the present invention, the human demonstration video data and the robot demonstration video data may include the visual regions of the hand and the end-effector during recording, and also include other visual regions. However, in the embodiments of the present application, mainly the task-related visual regions are concerned. Therefore, this part of the visual region may be extracted from the video data. Specifically, an image segmentation model may be used to mask the background regions irrelevant to the double-arm operation and only retain the visual regions of the human hand or the end-effector to highlight the visual regions related to the double-arm operation task.

[0040] Specifically, still referring to the above Figure 2 , before step 201, it may further include: Step 1020: Use the entity masking technology to mask the background area in the first demonstration video data and retain the image of the two-handed area; and use the entity masking technology to mask the background area in the second demonstration video data and retain the image of the end effector area.

[0041] In the embodiment of the present invention, after obtaining the spatio-temporally aligned human two-arm operation demonstration data and the robot two-arm operation demonstration data, based on the above data, and through a pre-trained imitation learning network, the trajectory data and action control data of the two-arm operation can be obtained. Specifically, the above imitation learning network may include a vision-driven policy network and a prediction head. The vision-driven policy network mainly performs feature extraction and can extract the human two-arm operation feature data and the robot two-arm operation feature data, while the prediction head can perform prediction based on the above feature data to predict the trajectory data and action control data.

[0042] Specifically, as Figure 3 shown, for Figure 1 step 103 in the embodiment shown, it may specifically include: Step 1031: Through the vision-driven policy network, perform feature extraction based on the spatio-temporally aligned human two-arm operation demonstration data to obtain human two-arm operation feature data; and perform feature extraction based on the spatio-temporally aligned robot two-arm operation demonstration data to obtain robot two-arm operation feature data; In this step, a unified vision-driven policy network can be used to simultaneously learn the two-arm operation skills and control signals from the above human two-arm operation demonstration data and robot two-arm operation demonstration data, so as to extract a general action feature representation and output the above action feature representation. The above feature data may exist in the form of feature vectors.

[0043] The vision-driven policy network in the embodiment of the present application may adopt the network architecture in the prior art. The difference is that in the network architecture in the prior art, only one of the human two-arm operation demonstration data and the robot two-arm operation demonstration data is input at the input end, while in the embodiment of the present invention, the above two kinds of demonstration data are used simultaneously, and spatio-temporal alignment and other processing are also performed. The above vision-driven policy network may include an action chunk transformer (ACT) network or a diffusion policy (DP) network. In the embodiment of the present invention, it is equivalent to expanding and processing the used demonstration data on the basis of these two networks, so as to improve the performance of the overall imitation learning scheme. The specific situation of performance improvement will be introduced in the subsequent embodiments.

[0044] Step 1032: Based on the human double-arm operation feature data and the robot double-arm operation feature data, perform prediction through the prediction head to predict the trajectory data of the double-arm operation and the motion control data of the double-arm operation.

[0045] In the above embodiments of the present invention, the prediction head includes a trajectory data prediction head and a control data prediction head, and the trajectory data prediction or the motion control data prediction is respectively performed by using the above two types of prediction heads. Specifically, step 1032 may specifically include: Through the trajectory data prediction head, perform trajectory prediction based on the human double-arm operation feature data and the robot double-arm operation feature data to predict the trajectory data of the double-arm operation; and, Through the control data prediction head, perform control prediction based on the robot double-arm operation feature data to predict the motion control data of the double-arm operation.

[0046] In the embodiments of the present invention, the trajectory data prediction head may be a 12-degree-of-freedom trajectory data prediction head, which can perform prediction based on the human double-arm operation feature data and the robot double-arm operation feature data to predict the trajectory data of the double-arm operation, that is, the actual spatial path of the end effector of the robot when performing the double-arm operation. In the training stage, the above actual control path can be used as the training target of the prediction head; the control data prediction head may be a 14-degree-of-freedom control data prediction head, which mainly performs prediction based on the robot double-arm operation feature data to predict the motion control data of the double-arm operation, that is, the specific motion control signal of the end effector of the robot. The above two prediction heads can jointly supervise the double-arm operation task, so that the imitation learning can obtain the control data in two aspects of the trajectory and the control signal to guide the robot to complete the double-arm operation task.

[0047] In some embodiments of the present application, the above imitation learning network may also be provided with a loss function, which mainly calculates the difference between the predicted value of the prediction head and the actual trajectory data label and the motion control data label in the training stage, so as to adjust the parameters in the imitation learning network, that is, the parameters in the above visual driving policy network and the prediction head.

[0048] In some embodiments, when the above prediction head includes a trajectory data prediction head and a control data prediction head, the above loss function may include a trajectory loss sub-function and a motion control loss sub-function, and the method further includes: In the training stage, calculate the trajectory prediction error based on the predicted trajectory data; and, Calculate the motion control prediction error based on the predicted motion control data.

[0049] In some embodiments, based on the above-obtained trajectory prediction error and action control prediction error, the training loss of the entire imitation learning network can be calculated according to the above errors, and the parameters can be adjusted. Specifically, the loss value of the loss function can be calculated according to the trajectory prediction error, the action control prediction error, and a preset adjustment value; finally, the parameters of the vision-driven policy network and the prediction head are adjusted according to the loss value of the loss function.

[0050] The training loss therein can be calculated through the following formula: wherein, represents the trajectory prediction error, represents the action control prediction error, is a regulation coefficient, which can regulate the weight ratio of the above two errors when calculating the final training loss, and its specific value can be set according to the actual situation.

[0051] In some embodiments, considering that the trajectory data prediction head performs trajectory prediction based on human double-arm operation feature data and robot double-arm operation feature data, therefore, the trajectory prediction error therein may come from the above two types of data. That is, the trajectory prediction error may be the error brought by the human double-arm operation feature data , and the error brought by the robot double-arm operation feature data ; and for the action control prediction error , it is only related to the error brought by the human double-arm operation feature data.

[0052] Figure 4 FIG. Figure 4As shown, the learning framework can include five parts: a demonstration data acquisition module, spatio-temporal alignment, a vision-driven policy network, features, training, and an interface. The demonstration data acquisition module mainly acquires human demonstration data and robot demonstration data. The human demonstration data can include human demonstration video data (Human Video) and demonstration trajectory data (Human Proprio). For the demonstration video data (Human Video), it can only include ego-view video data (Egoview only); while the robot demonstration data can also include robot demonstration video data (Robot Video) and demonstration trajectory data (Robot Proprio). For the demonstration video data (Robot Video), it can include both ego-view video data (Ego view) and dual-wrist view video data (Wrist Views), which can be left-wrist view video data and right-wrist view video data. After collecting the above-mentioned demonstration data, it can be input into the spatio-temporal alignment module (Temporal-Spatial Alignment, abbreviated as: TSA) for spatio-temporal alignment processing.

[0053] The process of spatio-temporal alignment processing can specifically be to construct video data streams, that is, including the data streams of the above-mentioned human ego-view video data ( ), the data streams of the robot ego-view video data ( ), the data streams of the robot left-wrist view video data ( ), and the data streams of the robot right-wrist view video data ( ). And in the spatio-temporal alignment module, the visual regions related to the dual-arm operation task are extracted through an embodiment mask respectively, and the key frame sequences are aligned through temporal synchronization, so that the video data of humans and robots are consistent in the time dimension. Finally, the human demonstration video data and the robot demonstration video data after spatio-temporal alignment are formed. From the perspective of ego-view and dual-wrist view, the above-mentioned demonstration video data can specifically include shared ego-view video (shared Ego-View vision tower) and robot dual-wrist view video (Robot Wrist-Cam Vision Tower). In addition, for the demonstration trajectory data, it can include constructing the data stream of the human demonstration trajectory data , and constructing the data stream of the robot demonstration trajectory data , The above data stream is input into the Spatial Proprio. Normalization to unify the human-robot trajectory representation, constructing a unified spatial representation space. Specifically, the Proj1. of the human demonstration trajectory data after spatial normalization and the Proj2. of the robot demonstration trajectory data after normalization.

[0054] After undergoing the above spatio-temporal pair processing and obtaining the spatio-temporally aligned demonstration video data and demonstration trajectory data, they can be input into the shared Visuomotor Policy for feature extraction. The above network can also be called the Action Encoder&Decoder. The above visual drive policy network can extract features from the aligned demonstration video data and demonstration trajectory data to extract a general action representation, while learning operation skills and embodied control signals. After this feature extraction process, corresponding feature data (Feature) can be obtained, which can specifically include human two-arm operation feature data ( ), and robot two-arm operation feature data ( ). The above feature data can be a feature vector.

[0055] After obtaining the above feature data, it can be sent to the prediction head for prediction. And in the training stage, the loss value can be calculated through the loss function, thereby adjusting the parameters of the visual drive policy network and the prediction head. Specifically, the above prediction head can include a 12-degree-of-freedom trajectory data prediction head (12DoF Trajectory Head Ф) and a 14-degree-of-freedom control data prediction head (14DoF Action Head ѱ). And, the above human two-arm operation feature data ( ), and robot two-arm operation feature data ( ) are both input into the 12-degree-of-freedom trajectory data prediction head to predict the trajectory data; while the robot two-arm operation feature data ( ) will be input into the 14-degree-of-freedom control data prediction head to predict the action control data. The above predicted trajectory data and action control data can be output through the Interface to guide the robot to perform two-arm operations.

[0056] In addition, as described above, in the training stage (Training only), the loss value can be calculated through the loss function. Specifically, the above loss value can include the trajectory prediction error and the action control prediction error These two parts, and the final loss value is calculated based on these two parts. And the trajectory prediction error among them Errors possibly caused by the characteristic data of human double-arm operations , and those caused by the characteristic data of robot double-arm operations ; for the action control prediction error , it is only related to the errors caused by the characteristic data of human double-arm operations . The loss function provided by the embodiments of this application can take into account both the generalization performance and the control accuracy.

[0057] The imitation learning framework provided by the embodiments of this application can also be referred to as the Video2BM framework, which can be applied to various double-arm operation learning tasks. For example, it can be applied to tasks such as Groceries, PickPour, and PourDrink. Specifically, the Groceries task refers to opening a cloth bag with the left robotic arm and classifying and placing the objects on the table into the bag with the right robotic arm; the PickPour task refers to the right arm grasping a bowl and keeping it stable, and the left arm continuously completing multiple picking and pouring actions; the PourDrink task refers to the two arms cooperating to control the position and angle of the soda can and the cup to achieve precise pouring operation.

[0058] In the embodiments of this application, the technical effects of the above embodiments of this application can be comprehensively evaluated through the above three tasks. The demonstration video data of the robot can be collected through a remote control operating system, and the demonstration video data can be collected through two wrist cameras and a self-view camera; the demonstration video data and demonstration trajectory data of humans can be captured through dedicated wearable devices, such as Apple Vision Pro, to capture the hand trajectory data and self-view video data. For the above video data, 400 frames of images can be uniformly extracted from each video data, and the chunk of each action is set to 50 frames, corresponding to about 2 seconds. When training the imitation learning model, it can be completed on 8 A100 GPUs, with a total of 3000 training rounds, taking about 10 hours.

[0059] When conducting a technical effect review, we can introduce several evaluation metrics to quantify the performance of the imitation learning model in the dual-arm operation task. These evaluation metrics can include the Success Rate (SR), Retry Success Rate (RSR), and Points (Pts), etc. Among them, the Success Rate (SR) refers to the proportion of successfully completing all task stages in one attempt; the Retry Success Rate (RSR) refers to the proportion of successfully completing the task within a maximum of 5 attempts, measuring the model's recovery ability from mid-course failures; the Points (Pts) metric is a fine-grained performance measure used to evaluate the completion of long-range tasks. This metric divides complex tasks into several phased sub-goals, scores the completion of each phase, and thus reflects the overall progress of task execution. If the task is fully completed, the full score is obtained; if there are errors or interruptions during the process, the score is calculated based on the number of completed phases. Taking the "tidy up the desktop" task as an example, assume that this task is divided into 4 phases. If the robot successfully completes the first two steps and makes an error in the third step, the final score is 2 points (the full score is 4 points). This metric can more precisely measure the stability, robustness, and fault tolerance of the system in long-range multi-step tasks.

[0060] Moreover, we introduce two benchmark imitation learning methods provided in the related art, such as the Action Chunk Transformer model (ACT), which is an imitation learning framework based on action segment prediction; and the Diffusion Policy model (DP), which is a model for generating control actions based on the diffusion process. In the above benchmark imitation learning methods, only the demonstration data of the robot is input into the above models for imitation learning. In the embodiments of this application, both the demonstration data of humans and the demonstration data of the robot are introduced, and the above two benchmark imitation learning methods can be introduced into the overall learning framework, respectively serving as the vision-driven policy network in the overall learning framework to form an improved enhanced version (Video2BM). By comparing the performances of the enhanced version before and after the improvement in different aspects, the advantages of the technical solution provided in this application can be obtained.

[0061] Specifically, reference can be made to Figure 5As shown in (a), (b), and (c), in the Groceries task, PickPour task, and PourDrink task, the two indicators of the one-time completion rate SR and points Pts of the improved enhanced version have both improved. Among them, "w / o" represents the indicators before improvement, and "w / o" represents the indicators after improvement. For example, in the PourDrink task, the one-time success rate of the ACT baseline method increased from 25% to 40% before and after improvement, and the one-time success rate of the DP baseline method increased from 15% to 35% before and after improvement. The points of the ACT baseline method increased from 35% to 47% before and after improvement, and the points of the DP baseline method increased from 18% to 36% before and after improvement. This shows that introducing human demonstration data can help make up for the control blind spots in the robot's demonstration data, especially in long-term sequential tasks that require fine operations, and the improvement effect is particularly significant.

[0062] In terms of generalization ability, the improved imitation learning framework was further tested for its performance in the following three aspects of generalization ability: (1) Object generalization: The types of objects used in the test are completely different from the training data; (2) Viewpoint generalization: The camera angle changes; (3) Position generalization: The initial placement position of the object is different from that in the training stage.

[0063] Refer to as Figure 6 Shown in the comparison chart of the generalization ability effect, where "w / o" represents the indicators before improvement, and "w / o" represents the indicators after improvement. The results show that in both the ACT baseline method and the DP baseline method, the improved enhanced version has improved in the above three aspects of generalization ability. Whether in terms of the one-time success rate SR or the points Pts, it is better than the basic model that only uses the robot's demonstration data before improvement, indicating that human demonstration data helps the imitation learning model learn more abstract and adaptable strategies.

[0064] In addition, to evaluate the impact of different amounts of human demonstration data on the model performance, we designed experiments with different ratios of human demonstration data to robot demonstration data. As Figure 7 Shown in the results of (a), (b), and (c), as the number of human demonstration data samples increases, the task success rate of the model steadily increases and finally exceeds the performance ceiling that can be achieved by only using the robot's demonstration data for training. This result verifies that human demonstration data has significant scalability and high cost-effectiveness.

[0065] Refer to Figure 7As shown, it presents the analysis of the expansion effect of the improved imitation learning framework under different data scales and acquisition efficiencies, and shows the performance expansion trend of the improved imitation learning framework under the training of human-robot hybrid demonstration data. Among them, Figure (a) shows the change of the retry success rate (RSR) based on the ACT benchmark method under different numbers of demonstration data (Query Number). In the case of a large number of demonstration data, the improved imitation learning framework, that is, using the demonstration data of humans (Human) and robots (Robot), has a higher retry success rate (RSR); Figures (b) and (c) show the performance comparison of the improved imitation learning framework between the ACT-based benchmark method and the DP-based benchmark method under the condition that the video acquisition durations of humans (Human) and robots (Robot) are the same. Among them, "w / o" represents the index before improvement, and "w / o" represents the index after improvement. The above comparison results show that the improved imitation learning framework has higher learning efficiency within the unit acquisition time (Collection Time), and the retry success rate (RSR) has been significantly improved, highlighting the superiority of human demonstration data in data expansion.

[0066] The following describes the imitation learning device for dual-arm operation provided by the present invention. The imitation learning device for dual-arm operation described below can be correspondingly referred to the imitation learning method for dual-arm operation described above.

[0067] Figure 8 is a schematic structural diagram of the imitation learning device for dual-arm operation provided by the present invention, as Figure 8 shown. The device 800 includes: A data acquisition module 801, configured to acquire human dual-arm operation demonstration data and robot dual-arm operation demonstration data; A spatio-temporal alignment module 802, configured to perform spatio-temporal alignment on the human dual-arm operation demonstration data and the robot dual-arm operation demonstration data to obtain the spatio-temporally aligned human dual-arm operation demonstration data and the spatio-temporally aligned robot dual-arm operation demonstration data; An imitation learning module 803, configured to obtain the trajectory data and action control data of the dual-arm operation through a pre-trained imitation learning network based on the spatio-temporally aligned human dual-arm operation demonstration data and the spatio-temporally aligned robot dual-arm operation demonstration data; Wherein, the imitation learning network is pre-trained according to human dual-arm operation demonstration data samples, robot dual-arm operation demonstration data samples, and corresponding trajectory data labels and action control data labels of the dual-arm operation.

[0068] Based on any of the above embodiments, the human double-arm operation demonstration data includes first demonstration video data and first demonstration trajectory data, and the robot double-arm operation demonstration data includes first demonstration video data and second demonstration trajectory data; The above-mentioned spatio-temporal alignment module is specifically used for: Performing time alignment by using the key frames of the first demonstration video data and the key frames of the second demonstration video data; and Performing spatial coordinate normalization on the first demonstration trajectory data and the second demonstration trajectory data.

[0069] Based on any of the above embodiments, it further includes: A mask module, which is used to use entity masking technology to mask the background area in the first demonstration video data and retain the image of the two-handed area; and use the entity masking technology to mask the background area in the second demonstration video data and retain the image of the end effector area.

[0070] Based on any of the above embodiments, the first demonstration video data includes self-view video data, and the second demonstration video data includes double-wrist view video data; the first demonstration trajectory data includes hand trajectory data, and the second demonstration trajectory data includes end effector trajectory data.

[0071] Based on any of the above embodiments, the imitation learning network includes a vision-driven policy network and a prediction head; The imitation learning module is specifically used for: Through the vision-driven policy network, performing feature extraction based on the spatio-temporally aligned human double-arm operation demonstration data to obtain human double-arm operation feature data; and performing feature extraction based on the spatio-temporally aligned robot double-arm operation demonstration data to obtain robot double-arm operation feature data; Through the prediction head, performing prediction based on the human double-arm operation feature data and the robot double-arm operation feature data to predict the trajectory data of the double-arm operation and the action control data of the double-arm operation.

[0072] Based on any of the above embodiments, the prediction head includes a trajectory data prediction head and a control data prediction head; The step of performing prediction through the prediction head based on the human double-arm operation feature data and the robot double-arm operation feature data to predict the trajectory data of the double-arm operation and the action control data of the double-arm operation includes: The step of performing prediction through the prediction head based on the human double-arm operation feature data and the robot double-arm operation feature data to predict the trajectory data of the double-arm operation and the action control data of the double-arm operation includes: Through the trajectory data prediction head, trajectory prediction is performed based on the human double-arm operation feature data and the robot double-arm operation feature data to predict the trajectory data of the double-arm operation; Through the control data prediction head, control prediction is performed based on the robot double-arm operation feature data to predict the motion control data of the double-arm operation.

[0073] Based on any of the above embodiments, the imitation learning network is further provided with a loss function; the loss function includes a trajectory loss sub-function and an action control loss sub-function, and further includes: An error calculation module, configured to calculate a trajectory prediction error based on the predicted trajectory data during the pre-training phase; and calculate an action control prediction error based on the predicted action control data.

[0074] Based on any of the above embodiments, it further includes: A parameter adjustment module, configured to calculate the loss value of the loss function according to the trajectory prediction error, the action control prediction error, and a preset adjustment value; and adjust the parameters of the visual drive policy network and the prediction head according to the loss value of the loss function.

[0075] Based on any of the above embodiments, the visual policy network includes an action chunk converter ACT network or a diffusion policy DP network.

[0076] Figure 9 An example of a schematic physical structure diagram of an electronic device is shown as Figure 9 shown. The electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440. Among them, the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call the logic instructions in the memory 430 to execute the imitation learning method for double-arm operation, and the method includes: obtaining human double-arm operation demonstration data and robot double-arm operation demonstration data; performing spatio-temporal alignment on the human double-arm operation demonstration data and the robot double-arm operation demonstration data to obtain the spatio-temporally aligned human double-arm operation demonstration data and the spatio-temporally aligned robot double-arm operation demonstration data; based on the spatio-temporally aligned human double-arm operation demonstration data and the spatio-temporally aligned robot double-arm operation demonstration data, obtaining the trajectory data and action control data of the double-arm operation through a pre-trained imitation learning network; where the imitation learning network is pre-trained according to human double-arm operation demonstration data samples, robot double-arm operation demonstration data samples, and corresponding trajectory data labels and action control data labels of the double-arm operation.

[0077] In addition, when the logical instructions in the above-mentioned memory 430 can be implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0078] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the imitation learning method of double-arm operation provided by the above-mentioned various methods. The method includes: obtaining human double-arm operation demonstration data and robot double-arm operation demonstration data; performing spatio-temporal alignment on the human double-arm operation demonstration data and the robot double-arm operation demonstration data to obtain the spatio-temporally aligned human double-arm operation demonstration data and the spatio-temporally aligned robot double-arm operation demonstration data; based on the spatio-temporally aligned human double-arm operation demonstration data and the spatio-temporally aligned robot double-arm operation demonstration data, through a pre-trained imitation learning network, obtaining the trajectory data and action control data of the double-arm operation; wherein, the imitation learning network is pre-trained according to human double-arm operation demonstration data samples, robot double-arm operation demonstration data samples, and corresponding trajectory data labels and action control data labels of the double-arm operation.

[0079] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements an imitation learning method for dual-arm operation provided by the above-mentioned various methods. The method includes: obtaining human dual-arm operation demonstration data and robot dual-arm operation demonstration data; performing spatio-temporal alignment on the human dual-arm operation demonstration data and the robot dual-arm operation demonstration data to obtain the spatio-temporally aligned human dual-arm operation demonstration data and the spatio-temporally aligned robot dual-arm operation demonstration data; based on the spatio-temporally aligned human dual-arm operation demonstration data and the spatio-temporally aligned robot dual-arm operation demonstration data, through a pre-trained imitation learning network, obtaining the trajectory data and action control data of the dual-arm operation; wherein, the imitation learning network is pre-trained according to human dual-arm operation demonstration data samples, robot dual-arm operation demonstration data samples, and corresponding trajectory data labels and action control data labels of the dual-arm operation.

[0080] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0081] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0082] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An imitation learning method for dual-arm operation, characterized in that Including: Obtaining human double - arm operation demonstration data and robot double - arm operation demonstration data; Performing spatio - temporal alignment on the human double - arm operation demonstration data and the robot double - arm operation demonstration data to obtain the spatio - temporally aligned human double - arm operation demonstration data and the spatio - temporally aligned robot double - arm operation demonstration data; Based on the spatio - temporally aligned human double - arm operation demonstration data and the spatio - temporally aligned robot double - arm operation demonstration data, through a pre - trained imitation learning network, obtaining the trajectory data and action control data of the double - arm operation; Among them, the imitation learning network is pre - trained according to human double - arm operation demonstration data samples, robot double - arm operation demonstration data samples, and corresponding trajectory data labels and action control data labels of the double - arm operation.

2. The imitation learning method according to claim 1, wherein The human double - arm operation demonstration data includes first demonstration video data and first demonstration trajectory data, and the robot double - arm operation demonstration data includes first demonstration video data and second demonstration trajectory data; The performing spatio - temporal alignment on the human double - arm operation demonstration data and the robot double - arm operation demonstration data includes: Performing time alignment using the key frames of the first demonstration video data and the key frames of the second demonstration video data; And, Performing spatial coordinate normalization on the first demonstration trajectory data and the second demonstration trajectory data.

3. The imitation learning method according to claim 2, wherein Before performing time alignment using the key frames of the first demonstration video data and the key frames of the second demonstration video data, it further includes: Using the entity masking technique to mask the background area in the first demonstration video data and retain the image of the two - hand area; and using the entity masking technique to mask the background area in the second demonstration video data and retain the image of the end - effector area.

4. The imitation learning method according to claim 2, wherein The first demonstration video data includes self - perspective video data, and the second demonstration video data includes double - wrist perspective video data; the first demonstration trajectory data includes hand trajectory data, and the second demonstration trajectory data includes end - effector trajectory data.

5. The imitation learning method according to any one of claims 1-4, characterized in that, The imitation learning network includes a vision - driven policy network and a prediction head; The obtaining the trajectory data and action control data of the double - arm operation based on the spatio - temporally aligned human double - arm operation demonstration data and the spatio - temporally aligned robot double - arm operation demonstration data through a pre - trained imitation learning network includes: Through the vision - driven policy network, performing feature extraction based on the spatio - temporally aligned human double - arm operation demonstration data to obtain human double - arm operation feature data; and performing feature extraction based on the spatio - temporally aligned robot double - arm operation demonstration data to obtain robot double - arm operation feature data; Through the prediction head, performing prediction based on the human double - arm operation feature data and the robot double - arm operation feature data to predict the trajectory data and the action control data of the double - arm operation.

6. The imitation learning method according to claim 5, wherein The prediction head includes a trajectory data prediction head and a control data prediction head; Predicting, via the prediction head, based on the human double-arm operation feature data and the robot double-arm operation feature data to obtain the trajectory data of the double-arm operation and the motion control data of the double-arm operation, including: Predicting, via the prediction head, based on the human double-arm operation feature data and the robot double-arm operation feature data to obtain the trajectory data of the double-arm operation and the motion control data of the double-arm operation, including: Predicting the trajectory of the double-arm operation via the trajectory data prediction head based on the human double-arm operation feature data and the robot double-arm operation feature data to obtain the trajectory data of the double-arm operation; Predicting the motion control of the double-arm operation via the control data prediction head based on the robot double-arm operation feature data to obtain the motion control data of the double-arm operation.

7. The imitation learning method according to claim 5, characterized in that The imitation learning network is further provided with a loss function; the loss function includes a trajectory loss sub-function and a motion control loss sub-function, and the method further includes: In the pre-training stage, calculating the trajectory prediction error based on the predicted trajectory data; and calculating the motion control prediction error based on the predicted motion control data.

8. The imitation learning method according to claim 7, characterized in that, Further included is: Calculating the loss value of the loss function according to the trajectory prediction error, the motion control prediction error and a preset adjustment value; Adjusting the parameters of the vision-driven policy network and the prediction head according to the loss value of the loss function.

9. The imitation learning method according to claim 5, characterized in that The vision policy network includes an action chunk transformer (ACT) network or a diffusion policy (DP) network.

10. An imitation learning device for dual-arm operation, characterized in that, Including: A data acquisition module, configured to acquire human double-arm operation demonstration data and robot double-arm operation demonstration data; A spatio-temporal alignment module, configured to perform spatio-temporal alignment on the human double-arm operation demonstration data and the robot double-arm operation demonstration data to obtain the spatio-temporally aligned human double-arm operation demonstration data and the spatio-temporally aligned robot double-arm operation demonstration data; An imitation learning module, configured to obtain the trajectory data and the motion control data of the double-arm operation via a pre-trained imitation learning network based on the spatio-temporally aligned human double-arm operation demonstration data and the spatio-temporally aligned robot double-arm operation demonstration data; Wherein, the imitation learning network is pre-trained according to human double-arm operation demonstration data samples, robot double-arm operation demonstration data samples, and corresponding trajectory data labels and motion control data labels of the double-arm operation.

11. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the imitation learning method for double-arm operation according to any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the imitation learning method for double-arm operation according to any one of claims 1 to 9.

Images