Robot heterogeneous operation skill transfer learning method and device and storage medium

Abstract

The method, device and storage medium for robot heterogeneous operation skill migration learning provided by the embodiments of the present disclosure comprise: setting a pre-training task of robot heterogeneous operation skill migration learning, and realizing skill policy learning of the pre-training task by a robot in a source domain and a robot in a target domain based on a reinforcement learning method; respectively constructing a self-encoder of the source domain robot and the target domain robot, training the two types of self-encoders by using a contrast learning method and fusing a domain randomization method; constructing a migration reward function by using the two types of self-encoders, and migrating the skill policy of the source domain robot on a target 3C assembly task to the target domain robot in a manner of maximizing a sum of a target 3C assembly task reward function and the migration reward function based on the reinforcement learning method. The present disclosure can meet the requirement of rapid update learning of 3C assembly robot heterogeneous operation skills, can improve the production efficiency and intelligent level of 3C assembly, and can reduce the production manufacturing cost of the 3C assembly industry.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of robots, in particular to a robot heterogeneous operation skill transfer learning method, device and storage medium for 3C assembly. BACKGROUND

[0002] 3C (3C is the collective term of computer, communication and consumer electronics) assembly industry is large in scale and has become an important pillar industry at the present stage. Due to the increasingly scarce labor force, the production labor cost of enterprises is high, and currently robots are gradually widely used in 3C assembly industry. However, the 3C assembly industry has the characteristics of a large variety of products and a fast updating speed of new products, and robots need to perform different assembly tasks frequently. However, due to the different types of assembly objects or the different configurations of robots used, the dimensions of the input state vectors or output action vectors required by the robots performing assembly operations are different, resulting in heterogeneous differences in skill forms, so that the operation skills trained by the robots based on a certain assembly task cannot be directly applied to other assembly tasks. Transfer learning, as a rapid learning technology in the field of artificial intelligence, can quickly learn and generalize the operation skills of existing tasks to new operation tasks. Therefore, by combining robot technology and transfer learning technology in the field of artificial intelligence, rapid learning of heterogeneous assembly operation skills of robots is realized, and the intelligent level of 3C assembly is improved, which is extremely necessary to help the development of 3C industry. SUMMARY

[0003] The present disclosure aims to at least solve one of the technical problems in the related art to some extent.

[0004] To this end, the robot heterogeneous operation skill transfer learning method provided by the first aspect of the present disclosure comprises:

[0005] A pre-training task for robot heterogeneous operation skill transfer learning for 3C assembly is set, and skill strategy learning of robots in a source domain and robots in a target domain for the same pre-training task is realized based on a reinforcement learning method, to obtain a source domain robot pre-training task skill strategy and a target domain robot pre-training task skill strategy, respectively;

[0006] In a pre-training task operation environment, the source domain robot pre-training task skill strategy and the target domain robot pre-training task skill strategy are executed respectively, and the source domain robot state and the target domain robot state are recorded to obtain interaction training data;

[0007] The source domain robot auto-encoder and the target domain robot auto-encoder are constructed, each robot auto-encoder comprises an encoder and a decoder, the source domain robot auto-encoder and the target domain robot auto-encoder are trained based on the interaction training data by using a contrast learning method and fusing a domain randomization method, and the trained source domain encoder and target domain encoder are obtained to map the source domain robot state and the target domain robot state to a common feature domain space.

[0008] The source domain robot skill policy neural network and the target domain robot skill policy neural network are constructed, the source domain robot skill policy neural network is trained based on the demonstration data of performing the target 3C assembly task and the behavior cloning imitation learning method, the skill policy of the source domain robot on the target 3C assembly task is obtained, the trained source domain encoder and target domain encoder are used to construct a transfer reward function, and the target domain robot skill policy neural network is trained by maximizing the sum of the target 3C assembly task reward function and the transfer reward function based on the reinforcement learning method, and the skill policy of the source domain robot on the target 3C assembly task is transferred to the target domain robot.

[0009] The robot heterogeneous operation skill transfer learning method provided by the first aspect of the present disclosure has the following characteristics and beneficial effects:

[0010] The robot heterogeneous operation skill transfer learning method provided by the first aspect of the present disclosure realizes efficient transfer learning of 3C assembly operation skills of robots, can meet the requirements of rapid update and learning of heterogeneous operation skills of 3C assembly industry due to various product types and rapid update speed of new products, and can significantly improve the production efficiency and intelligent level of 3C assembly to a certain extent, and reduce the production cost of 3C assembly industry.

[0011] In some embodiments, the heterogeneous operation skills differ in skill form due to different configurations of the source domain robot and the target domain robot or different types of objects assembled by the source domain robot and the target domain robot, resulting in different dimensions of input state vectors or different dimensions of output action vectors of skill policies required for the source domain robot and the target domain robot to perform respective tasks.

[0012] In some embodiments, the pre-training task is an operation task for 3C parts.

[0013] In some embodiments, the source domain robot pre-training task skill policy and the target domain robot pre-training task skill policy are obtained according to the following steps:

[0014] constructing a source domain robot pre-training task skill policy neural network and a target domain robot pre-training task skill policy neural network; the input of the source domain robot pre-training task skill policy neural network is a 2M+18-dimensional state vector The output is a 7-dimensional action vector a S The state vector contains the angles and angular velocities corresponding to the M joints of the source domain robot, the 6-dimensional pose and the corresponding velocity of the end effector of the source domain robot, and the spatial 6-dimensional position and attitude of the 3C components to be assembled, the first 6 dimensions of the action vector a S are used to control the 6-dimensional position and attitude of the end effector of the source domain robot, and the seventh dimension is used to control the opening and closing of the end effector of the source domain robot; the input of the target domain robot pre-training task skill policy neural network is a 2L+18-dimensional state vector The output is a 7-dimensional action vector a T The state vector contains the angles and angular velocities corresponding to the L joints of the robot in the target domain, the 6-dimensional pose and the corresponding velocity of the end effector of the robot in the target domain, and the spatial 6-dimensional position and attitude of the 3C components to be assembled, the first 6 dimensions of the action vector a T are used to control the 6-dimensional position and attitude of the end effector of the robot in the target domain, and the seventh dimension is used to control the opening and closing of the end effector of the robot in the target domain.

[0015] The source domain robot pre-training task skill policy neural network and the target domain robot pre-training task skill policy neural network are learned in a common pre-training task by using a reinforcement learning method, and the trained source domain robot pre-training task skill policy and the target domain robot pre-training task skill policy are obtained.

[0016] In some embodiments, the interaction training data includes causing the source domain robot and the target domain robot to execute the generated source domain robot pre-training task skill policy and the target domain robot pre-training task skill policy several times respectively, recording the 2M+12-dimensional source domain robot state vector corresponding to each time step t in the execution process of the source domain robot pre-training task skill policy and the images of the working scene where the source domain robot is located and the 2L+12-dimensional target domain robot state vector corresponding to each time step t in the execution process of the target domain robot pre-training task skill policy and the images of the working scene where the target domain robot is located The source domain robot state vector comprises angles and angular velocities of M joint pairs of the source domain robot, and 6D pose and its corresponding velocity of the end-effector of the source domain robot, and the target domain robot state vector comprises angles and angular velocities of L joint pairs of the robot in the target domain, and 6D pose and its corresponding velocity of the end-effector of the robot in the target domain.

[0017] In some embodiments, the source domain robot autoencoder is AE S , the source domain encoder f and the source domain decoder D S are constituted, the input of the source domain encoder f is the source domain state s composed of the source domain robot state vector and the image S , the output of the source domain encoder f is the source domain latent space state feature z S , the input of the source domain decoder D T is the source domain latent space state feature z T , and the output of the source domain decoder D T is the source domain predicted state

[0018] The target domain robot autoencoder is AE T , the target domain encoder g and the target domain decoder D T are constituted, the input of the target domain encoder g is the target domain state s composed of the target domain robot state vector and the image T , the output of the target domain encoder g is the target domain latent space state feature z T , the input of the target domain decoder D S is the target domain latent space state feature z T , and the output of the target domain decoder D S is the target domain predicted state

[0019] The source domain robot autoencoder AE S and the target domain robot autoencoder AE T are trained using the following loss function:

[0020]

[0021] wherein α, β are weight coefficients, N is a Gaussian distribution, represents the expected value calculated by respectively collecting the source domain state s S and the target domain state s T from the data experience pool B and sampling noise data σ S and σ T from the Gaussian distribution N, and the data experience pool B stores the interaction training data.

[0022] In some embodiments, the demonstration data comprises a demonstration state vector and a demonstration action vector The demonstration state vector is recorded by the source domain robot performing a number of target 3C assembly task processes in a teaching manner, the source domain robot state vector at each time step t and the image of the working scene where the source domain robot is located are recorded, the dimension of the source domain robot state vector is 2M+12, which contains the angles and angular velocities corresponding to M joints of the source domain robot, and the 6-dimensional pose and the corresponding speed of the end effector, the teaching action vector The dimension of the teaching action vector is 7, the first 6 dimensions are used to control the 6-dimensional position and attitude of the end effector of the source domain robot, and the seventh dimension is used to control the opening and closing of the end effector of the source domain robot.

[0023] The input of the source domain robot skill policy neural network is the teaching state vector The output is a 7-dimensional predicted action vector The first 6 dimensions of the predicted action vector are used to control the 6-dimensional position and attitude of the end effector of the source domain robot, and the seventh dimension is used to control the opening and closing of the end effector of the source domain robot.

[0024] When the source domain robot skill policy neural network is learned by using the behavior cloning-based imitation learning method, the following loss function is used:

[0025]

[0026] In the formula, represents the expected value calculated from the teaching state vector and the corresponding teaching action vector sampled from the teaching pool D, and the teaching data is stored in the teaching pool D.

[0027] In some embodiments, the input of the target domain robot skill policy neural network is a migration state vector, and the output is a 7-dimensional migration action vector, the migration state vector contains a target domain robot state vector and an image of a working scene where the target domain robot is located, the dimension of the target domain robot state vector is 2L+12, including the angles and angular velocities corresponding to L joints, and the 6-dimensional pose and the corresponding speed of the end effector; The first 6 dimensions of the migration action vector are used to control the 6-dimensional position and attitude of the end effector of the target domain robot, and the seventh dimension is used to control the opening and closing of the end effector of the target domain robot.

[0028] In the process of training the target domain robot skill policy neural network, the source domain robot and the target domain robot perform the target 3C assembly task according to the source domain robot skill policy and the target domain robot skill policy respectively, record the trajectory sequences and and are the state vectors of the source domain robot and the target domain robot respectively at time step τ when completing a target 3C assembly task process once, and the following loss function is adopted:

[0029]

[0030]

[0031]

[0032] In the formula, γ and are the discount factor and the weight coefficient respectively, γ τ represents the τ power of the discount factor γ, is the transfer reward function at time step τ, r τ is a sparse reward function for the target domain robot to perform the target 3C assembly task at time step τ, is the entropy of the skill policy of the target domain robot, represents sampling the transfer action vector a from the target domain robot skill policy trans calculated expected value.

[0033] The robot heterogeneous operation skill transfer learning method provided by the second aspect embodiment of the present disclosure comprises:

[0034] A first module is configured to set a pre-training task for robot heterogeneous operation skill transfer learning for 3C assembly, and implement skill policy learning of a robot in a source domain and a robot in a target domain for the same pre-training task based on a reinforcement learning method, to obtain a source domain robot pre-training task skill policy and a target domain robot pre-training task skill policy respectively.

[0035] A second module is configured to control the source domain robot and the target domain robot to execute the source domain robot pre-training task skill policy and the target domain robot pre-training task skill policy respectively in a pre-training task operation environment, and record source domain robot states and target domain robot states to obtain interaction training data.

[0036] A third module is configured to construct a source domain robot autoencoder and a target domain robot autoencoder, each of which comprises an encoder and a decoder, and train the source domain robot autoencoder and the target domain robot autoencoder simultaneously based on a contrast learning method and a domain randomization method based on the interaction training data, to obtain a trained source domain encoder and a trained target domain encoder, so as to map the source domain robot states and the target domain robot states to a common feature domain space respectively; and

[0037] A fourth module is configured to construct a source domain robot skill policy neural network and a target domain robot skill policy neural network, train the source domain robot skill policy neural network based on a demonstration data and an action cloning imitation learning method for performing a target 3C assembly task, obtain a skill policy of a source domain robot on the target 3C assembly task, construct a transfer reward function by using the trained source domain encoder and the target domain encoder, and train the target domain robot skill policy neural network based on a reinforcement learning method by maximizing a sum of a target 3C assembly task reward function and the transfer reward function, and transfer the skill policy of the source domain robot on the target 3C assembly task to the target domain robot.

[0038] The computer readable storage medium provided in the third aspect of the present disclosure stores computer instructions for causing the computer to perform the robot heterogeneous operation skill transfer learning method described in any embodiment of the first aspect of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The flowchart of the robot heterogeneous operation skill transfer learning method provided in the first aspect of the present disclosure is shown.

[0040] Figure 2 The structure diagram of the corresponding source domain robot autoencoder and target domain robot autoencoder in the transfer learning method provided in the first aspect of the present disclosure is shown.

[0041] Figure 3 The feature mapping diagram of the corresponding source domain robot encoder and target domain robot encoder in the transfer learning method provided in the first aspect of the present disclosure is shown.

[0042] Figure 4 The structure diagram of the source domain robot skill policy neural network and the target domain robot skill policy neural network based on the target 3C assembly task in the transfer learning method provided in the first aspect of the present disclosure is shown.

[0043] Figure 5 The structure diagram of the electronic device provided in the third aspect of the present disclosure is shown. DETAILED DESCRIPTION

[0044] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0045] On the contrary, the present application covers any alternatives, modifications, equivalent methods and solutions defined by the claims within the spirit and scope of the present application. Further, in order to make the public have a better understanding of the present application, some specific details are described in the following detailed description of the present application. The present application can also be fully understood without the description of these details by those skilled in the art.

[0046] Referring to Figure 1 The robot heterogeneous operation skill transfer learning method provided by the first aspect of the embodiment of the present disclosure is used to realize the robot heterogeneous operation skill transfer between 3C assembly tasks. The source domain robot is a 7-DOF Fetch manipulator, the target source robot is a 6-DOF UR5 manipulator, the end effectors of the source domain robot and the target source robot are both suction cups or two-finger grippers, and the target 3C assembly task is a mobile phone soft wire buckle, a mobile phone front camera assembly, a mobile phone SIM card holder assembly, a mobile phone back cover assembly, etc. The strategy mentioned in the embodiment of the present disclosure is used for the robot to complete the corresponding skill operation, and the strategy is represented by a neural network. The input and output of the strategy neural network are state and action respectively. The state is the relevant information generated by the interaction between the robot and the environment and used to determine the output of the strategy, and the action is the instruction for controlling the movement of the robot. The steps include:

[0047] S1, set the pre-training task of the robot heterogeneous operation skill transfer facing 3C assembly, and realize the skill strategy learning of the robot in the source domain and the robot in the target domain for the same pre-training task based on the reinforcement learning method, to obtain the trained source domain robot pre-training task skill strategy and the target domain robot pre-training task skill strategy respectively; the specific steps are as follows:

[0048] S11, set the common pre-training task of the robot in the source domain and the robot in the target domain as the operation task of sucking 3C components such as front camera and SIM card holder.

[0049] S12, construct the source domain robot pre-training task skill strategy neural network and the target domain robot pre-training task skill strategy neural network. Set the source domain robot pre-training task skill strategy The input of the neural network is a 2M+18(M=7) dimensional state vector The output is a 7-dimensional action vector a S , wherein the state vector contains the angles and angular velocities corresponding to the M joints of the source domain robot (totally 2M dimensions), the 6-dimensional pose of the end effector and the corresponding velocity (totally 12 dimensions), and the 6-dimensional spatial position and attitude of the 3C component to be grasped (totally 6 dimensions), and the action vector a SThe first 6 dimensions of the state vector are used to control the 6-dimensional position and pose of the source domain robot end effector, and the 7th dimension is used to control the opening and closing of the source domain robot end effector. Set the target domain robot pre-training task skill policy The input of the neural network is a 2L+18 (L=6) dimensional state vector The output is a 7-dimensional action vector a T , wherein the state vector includes the angles and angular velocities (a total of 2L dimensions) of the L joints of the robot in the target domain, the 6-dimensional pose of the end effector and its corresponding velocity (a total of 12 dimensions), and the 6-dimensional spatial position and pose of the 3C component to be grasped (a total of 6 dimensions), and the action vector a T The first 6 dimensions of the state vector are used to control the 6-dimensional position and pose of the target domain robot end effector, and the 7th dimension is used to control the opening and closing of the target domain robot end effector. The source domain robot pre-training task skill policy neural network is composed of four fully connected layers, wherein the first layer is an input layer composed of 2M+18 neurons, the middle three layers are fully connected layers with neuron numbers of 64, 128, and 64 respectively, and the fourth layer is an output layer composed of 7 neurons, and ReLU function is used as the activation function for each fully connected layer; the target domain robot pre-training task skill policy neural network is composed of four fully connected neural networks, wherein the first layer is an input layer composed of 2L+18 neurons, the middle three layers are fully connected layers with neuron numbers of 64, 128, and 64 respectively, and the fourth layer is an output layer composed of 7 neurons, and ReLU function is used as the activation function for each fully connected layer.

[0050] S13, using a reinforcement learning method, the randomly initialized source domain robot pre-training task skill policy neural network and the target domain robot pre-training task skill policy neural network are respectively learned in a common pre-training task, to obtain the trained source domain robot pre-training task skill policy and the target domain robot pre-training task skill policy

[0051] S2, in the pre-training task operation environment, the trained source domain robot pre-training task skill policy and the target domain robot pre-training task skill policy are respectively executed, and the source domain robot state vector and the target domain robot state vector are recorded, to obtain interaction training data; the specific steps are as follows:

[0052] In the 3C assembly pre-training task, the trained source domain robot pre-training task skill policy and the target domain robot pre-training task skill policy are respectively executed K (K=100) times on the source domain robot and the target domain robot. 2M+12(M=7) dimensional source domain robot state vector corresponding to each time step t (t∈[1, K]) in the execution process and the scene RGB image with a pixel size of 640x480 collected by the camera installed in the working scene of the source domain robot and the target domain robot pre-training task skill policy 2L+12(L=6) dimensional target domain robot state vector corresponding to each time step t in the execution process and the scene RGB image with a pixel size of 640x480 collected by the camera installed in the working scene of the target domain robot to construct the interaction training data and store the interaction training data of each time step to the data experience pool B. Wherein the source domain robot state vector contains the angles and angular velocities (2M dimensions in total) corresponding to the M joints of the source domain robot, the 6-dimensional pose of the end effector and the corresponding velocity (12 dimensions in total), the target domain robot state vector contains the angles and angular velocities (2L dimensions in total) corresponding to the L joints of the target domain robot, the 6-dimensional pose of the end effector and the corresponding velocity (12 dimensions in total).

[0053] S3, construct a source domain robot autoencoder and a target domain robot autoencoder, the source domain robot autoencoder and the target domain robot autoencoder each contain an encoder and a decoder, and the source domain robot autoencoder and the target domain robot autoencoder are trained based on the interaction training data using a contrastive learning method and a domain randomization method, to obtain trained source domain encoder and target domain encoder, to respectively map the source domain robot state and the target domain robot state to a common feature domain space; The specific steps are as follows:

[0054] S31, to map the source domain robot state and the target domain robot state to a common feature domain space, construct the source domain robot autoencoder AE Figure 2 and the target domain robot autoencoder AE S as shown in (a), (b) of T . Wherein;

[0055] The source domain robot autoencoder AE S is composed of a source domain encoder f and a source domain decoder D SThe source domain encoder f comprises a fully connected neural network module 1, a convolutional neural network module 1, and a fully connected neural network module 2. The fully connected neural network module 1 consists of three fully connected layers with 2M+12 (M=7), 64, and 64 neurons respectively. The convolutional neural network module 1 consists of four convolutional layers with 16, 32, 64, and 32 channels respectively, and one fully connected layer with 64 neurons. All four convolutional layers have a stride of 2 and no pooling layers; the kernel size is 2×2. The fully connected neural network module 2 consists of three fully connected layers with 128, 64, and 64 neurons respectively. The source domain decoder D... S It consists of three fully connected layers with 128, 64, and 2M+12 neurons respectively (M=7). Source Domain Robot Autoencoder (AE) S The input is the source domain state s S This includes the source domain robot state vector sampled from the data experience pool B. Images of the working scene of the source domain robot The corresponding inputs are then fed into fully connected neural network module 1 and convolutional neural network module 1, respectively. The resulting feature vectors are concatenated and then fed into fully connected neural network module 2. After mapping, the source domain latent space state features f(s) are obtained. S ), and the latent space state features f(s) of the source domain S Input to source domain decoder D S After reverse mapping D S (f(s S The predicted state of the source domain is then obtained. The source domain prediction state As a source domain robot autoencoder AE S The output.

[0056] Target domain robot autoencoder AE T The target domain encoder g and the target domain decoder D T The target domain encoder g comprises a fully connected neural network module 3, a convolutional neural network module 2, and a fully connected neural network module 4. The fully connected neural network module 3 consists of three fully connected layers with 2L+12 (L=6), 64, and 64 neurons respectively. The convolutional neural network module 2 consists of four convolutional layers with 16, 32, 64, and 32 channels respectively, and one fully connected layer with 64 neurons. All four convolutional layers have a stride of 2 and no pooling layers; the kernel size is 2×2. The fully connected neural network module 4 consists of three fully connected layers with 128, 64, and 64 neurons respectively. The target domain decoder D... T It consists of three fully connected layers with 128, 64, and 2L+12 neurons respectively (L=6). Target domain autoencoder AET The input of the target domain state s T includes the target domain robot state vector sampled from the data experience pool B and the image of the working scene where the target domain robot is located and are respectively input into the full connection neural network module 3 and the convolutional neural network module 2, the obtained feature vectors are spliced and then input into the full connection neural network module 4, and the target domain hidden space state feature g(s T ) is obtained after mapping. T The target domain hidden space state feature g(s T ) is input into the target domain decoder D T , and the target domain predicted state s T is obtained after reverse mapping D T (g(s T )) of the target domain hidden space state feature g(s T ). The target domain predicted state s is output as the output of the target domain robot autoencoder AE T .

[0057] S32, learning of the source domain robot autoencoder AE S and the target domain robot autoencoder AE T is realized based on the contrast learning method. The source domain state s S is mapped by the source domain encoder f of the source domain robot autoencoder AE S to obtain the source domain hidden space state feature f(s S ). The target domain state s T is mapped by the target domain encoder g of the target domain robot autoencoder AE T to obtain the target domain hidden space state feature g(s T ). The learning mode of minimizing ||f(s S )-g(s T )||2 can make f(s S ) and g(s T ) mapped to a common feature domain space, see Figure 3 . In order to expand the distribution space of data and improve the robust performance of the source domain encoder f and the target domain encoder g, the domain randomization method is adopted, and noise data σ S , σ T are added to the input source domain state s S corresponding to the source domain encoder f and the input target domain state s T corresponding to the target domain encoder g respectively to expand the distribution of the input state. In order to ensure that the hidden space state features f(s S +σ S ) and g(s T +σ T) effectiveness, a method of self-supervised learning, source domain robot autoencoder AE S of the source domain decoder D S f(s S +σ S ) as input to be reverse mapped to get the predicted state of the source domain set the target domain robot autoencoder AE T of the target domain decoder D T g(s T +σ T ) as input to be reverse mapped to get the predicted state of the target domain ultimately get the learning loss function of the source domain robot autoencoder AE S and the target domain robot autoencoder AE T is:

[0058]

[0059] Where min means minimum, alpha, beta (alpha = 0.5, beta = 0.5) are weight coefficients, N is a Gaussian distribution, represent the expected values calculated from the data experience pool B S and the target domain state s T and the noise data sigma S and sigma T sampled from the Gaussian distribution N.

[0060] After training based on the above formula, the source domain encoder f and the target domain encoder g that meet the subsequent migration requirements can be obtained.

[0061] S4, obtain the teaching data of the target 3C assembly task, construct the source domain robot skill policy neural network and the target domain robot skill policy neural network, train the source domain robot skill policy neural network based on the teaching data and the behavior cloning imitation learning method, obtain the skill policy of the source domain robot in the target 3C assembly task, use the source domain encoder f and the target domain encoder g to construct a transfer reward function, based on the reinforcement learning method, train the target domain robot skill policy neural network by maximizing the sum of the target 3C assembly task reward function and the transfer reward function, and migrate the skill policy of the source domain robot in the target 3C assembly task to the target domain robot, the specific implementation process is as follows:

[0062] S41, invite skilled workers to teach the operation of the source domain robot to perform the target 3C assembly task, and record the teaching state vector and the teaching action vector at each time t in the assembly process is stored into the demonstration pool D, thus obtaining the demonstration data, where the demonstration state vector is a 2M+12 dimensional source domain robot state vector recorded at time step t when the source domain robot performs the target 3C assembly task and the scene RGB image with pixel size of 640x480 captured by the camera installed in the source domain robot working scene is a 2M+12 dimensional source domain robot state vector recorded at time step t when the source domain robot performs the target 3C assembly task is a 2M+12 dimensional source domain robot state vector recorded at time step t when the source domain robot performs the target 3C assembly task has 7 dimensions, the first 6 dimensions are used to control the 6 dimensional position and pose of the end effector when the source domain robot performs the demonstration operation of the flexible flat cable buckling assembly task, and the 7th dimension is used to control the opening and closing of the end effector when the source domain robot performs the demonstration operation of the target 3C assembly task.

[0063] S42, construct the source domain robot skill policy neural network as shown in (a) of Figure 4 The input of the source domain robot skill policy neural network is the demonstration state vector The output is a 7 dimensional predicted action vector The first 6 dimensions of the predicted action vector are used to control the 6 dimensional position and pose of the end effector of the source domain robot, and the 7th dimension is used to control the opening and closing of the end effector of the source domain robot. The source domain robot skill policy neural network includes a fully connected neural network module 5, a convolutional neural network module 3, and a fully connected neural network module 6. The fully connected neural module 5 is composed of 3 layers of fully connected layers with neuron numbers of 2M+12 (M=7), 64, and 64, respectively. The convolutional neural network module 3 is composed of 4 layers of convolutional layers with channel numbers of 16, 32, 64, and 32, respectively, and 1 layer of fully connected layer with neuron number of 64, wherein the sliding step of the 4 layers of convolutional layers is 2 and there is no pooling layer, and the convolution kernel size is 2x2. The fully connected neural network 6 is composed of 3 layers of fully connected layers with neuron numbers of 128, 64, and 7, respectively. The neural network parameters of the fully connected neural network module 5 and the convolutional neural network module 3 are initialized based on the parameter values of the aforementioned trained fully connected neural network module 1 and convolutional neural network module 1, respectively, and the neural network parameters of the fully connected neural network 6 are randomly initialized.

[0064] The source domain robot skill policy neural network for performing the target 3C assembly task operation is learned by using a behavior cloning based imitation learning method, wherein the following loss function is used:

[0065]

[0066] In the formula, min represents finding the minimum value. This indicates sampling the teaching state vector from the teaching pool D. and corresponding teaching action vectors Calculate the expected value.

[0067] Utilizing the trained source domain robot skill strategy It can successfully execute the target 3C assembly task (mobile phone soft snap-fit ​​task).

[0068] S43, construct as follows Figure 4 The target domain robot skill policy neural network is shown in (b). The target domain robot skill policy is defined. The input to the neural network at time step t is the transition state vector. The output is a 7-dimensional transfer action vector. Where the migration state vector Includes target domain robot state vector The scene RGB image with a pixel size of 640×480 captured by the camera in the target domain robot's scene. Among them, the target domain robot state vector The dimensions are 2L+12, including the angles and angular velocities corresponding to L joints (2L dimensions in total), and the 6-dimensional pose of the end effector and its corresponding velocity (12 dimensions in total). Transfer motion vectors The first six dimensions are used to control the six-dimensional position and orientation of the end effector of the target domain robot, and the seventh dimension is used to control the opening and closing of the end effector. The target domain robot skill strategy neural network consists of a fully connected neural network module 7, a convolutional neural network module 4, and a fully connected neural network module 8. The fully connected neural network module 7 consists of three fully connected layers with 2L+12 (L=6), 64, and 64 neurons respectively. The convolutional neural network module 4 consists of four convolutional layers with 16, 32, 64, and 32 channels respectively, and one fully connected layer with 64 neurons. The stride of the four convolutional layers is 2, and none of them have pooling layers. The kernel size of the convolutional layers is 2×2. The fully connected neural network 8 consists of three fully connected layers with 128, 64, and 7 neurons respectively. The neural network parameters of the fully connected neural network module 7 and the convolutional neural network module 4 are initialized based on the parameter values ​​of the fully connected neural network module 3 and the convolutional neural network module 2 that have been trained. The neural network parameters of the fully connected neural network 8 are randomly initialized.

[0069] Skill policy transfer learning is performed on the target domain robot skill policy neural network used to perform the target 3C assembly task. During this process, the source domain robot executes the trained source domain robot skill policy. Operations of target 3C assembly tasks are performed, and a trajectory sequence in a process of completing an operation task is recorded denotes a state vector of the source domain robot at time step τ in a process of completing a target 3C assembly task, and has the same dimension as the demonstration state vector , τ ∈ [0, H], and the target domain robot executes a target domain robot skill policy Operations of target 3C assembly tasks are performed, and a trajectory sequence in a process of completing an operation task is recorded where H (H = 100) is the length of the trajectory sequence, denotes a target domain robot state vector of the target domain robot at time step τ in a process of performing a soft-wire buckling operation task. Since the target learning task of the target domain robot can only obtain a sparse reward r τ from the environment, that is, a reward of 1 is obtained when the task is successfully executed, and a reward of -1 is obtained when the task fails, which leads to difficulty in policy learning. Therefore, based on the source domain encoder f and the target domain encoder g learned in the foregoing, a transfer reward function at time step τ is constructed as follows:

[0070]

[0071] The learning of the target domain robot skill policy is performed in the form of maximizing the comprehensive reward function while maximizing the entropy of the target domain robot skill policy denotes an expected value calculated by sampling a transfer action vector a trans from the target domain robot skill policy to encourage the target domain robot skill policy to explore and speed up the learning process. In summary, the target domain robot skill policy is learned by the following formula:

[0072]

[0073] where γ (γ = 0.95) and are a discount factor and a weight coefficient, respectively, and γ τ denotes the τth power of the discount factor γ. That is, the source domain robot skill policy that has been well learned on the target task can be transferred to the target domain robot by means of the transfer reward function . After the target domain robot skill policy for the target 3C assembly task operation is trained, the target domain robot can execute the skill policy ​​Complete the target 3C assembly operation task.

[0074] The robot heterogeneous operation skill migration learning device provided by the second aspect embodiment of the present disclosure comprises:

[0075] The first module is configured to set a pre-training task for robot heterogeneous operation skill migration learning for 3C assembly, and implement skill policy learning of robots in a source domain and robots in a target domain on the same pre-training task based on a reinforcement learning method, to obtain a trained source domain robot pre-training task skill policy and a trained target domain robot pre-training task skill policy.

[0076] The second module is configured to control the source domain robots and the target domain robots to respectively execute the trained source domain robot pre-training task skill policy and the trained target domain robot pre-training task skill policy in a pre-training task operation environment, and record source domain robot states and target domain robot states, to obtain interactive training data.

[0077] The third module is configured to construct a source domain robot autoencoder and a target domain robot autoencoder, each of which comprises an encoder and a decoder, and train the source domain robot autoencoder and the target domain robot autoencoder based on the interactive training data by using a contrastive learning method and fusing a domain randomization method, to obtain a trained source domain encoder and a trained target domain encoder, so as to respectively map the source domain robot states and the target domain robot states to a common feature domain space.

[0078] The fourth module is configured to construct a source domain robot skill policy neural network and a target domain robot skill policy neural network, train the source domain robot skill policy neural network based on teaching data of an execution target 3C assembly task and an imitation learning method of behavior cloning, to obtain a skill policy of the source domain robot on the target 3C assembly task, construct a migration reward function by using the trained source domain encoder and the trained target domain encoder, and train the target domain robot skill policy neural network based on a reinforcement learning method by maximizing a sum of a target 3C assembly task reward function and the migration reward function, to migrate the skill policy of the source domain robot on the target 3C assembly task to the target domain robot.

[0079] In order to implement the above-mentioned embodiments, the present disclosure further provides a computer readable storage medium having a computer program stored thereon, which is executed by a processor to perform the robot heterogeneous operation skill migration learning method of the above-mentioned embodiments.

[0080] The following refers to Figure 5The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of the present disclosure. It should be noted that the electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs, desktop computers, and servers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0081] like Figure 5 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 102 or a program loaded from a storage device 108 into a random access memory (RAM) 103. The RAM 103 also stores various programs and data required for the operation of the electronic device 100. The processing unit 101, ROM 102, and RAM 103 are interconnected via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0082] Typically, the following devices can be connected to I / O interface 105: input devices 106 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, etc.; output devices 107 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 108 including, for example, magnetic tapes, hard disks, etc.; and communication devices 109. Communication device 109 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0083] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, this embodiment includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via communication device 109, or installed from storage device 108, or installed from ROM 102. When the computer program is executed by processing device 101, it performs the functions defined above in the methods of embodiments of this disclosure.

[0084] It should be noted that the computer-readable medium described above can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the disclosure, the computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such a propagated data signal can take any of a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that can be used to carry or store a program for use by or in connection with an instruction execution system, apparatus or device, other than the computer-readable storage medium. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wire, cable, optical fiber, RF (radio frequency), or any suitable combination of the foregoing.

[0085] The computer-readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device and be not assembled into the electronic device.

[0086] The computer readable medium carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to: set a pre-training task of robot heterogeneous operation skill transfer learning facing 3C assembly, implement skill policy learning of robots in a source domain and robots in a target domain on the same pre-training task based on a reinforcement learning method, and obtain a trained source domain robot pre-training task skill policy and a target domain robot pre-training task skill policy respectively; in a pre-training task operation environment, execute the trained source domain robot pre-training task skill policy and the target domain robot pre-training task skill policy respectively, and record source domain robot states and target domain robot states, and obtain interactive training data; construct a source domain robot autoencoder and a target domain robot autoencoder, each robot autoencoder includes an encoder and a decoder, and the source domain robot autoencoder and the target domain robot autoencoder are trained based on the interactive training data by using a contrast learning method and fusing a domain randomization method, to obtain a trained source domain encoder and a target domain encoder, so as to respectively map the source domain robot states and the target domain robot states to a common feature domain space; construct a source domain robot skill policy neural network and a target domain robot skill policy neural network, train the source domain robot skill policy neural network based on teaching data of an execution target 3C assembly task and an imitation learning method of behavior cloning, to obtain a skill policy of the source domain robot on the target 3C assembly task, construct a transfer reward function by using the trained source domain encoder and the target domain encoder, and train the target domain robot skill policy neural network by maximizing a sum of a target 3C assembly task reward function and the transfer reward function based on the reinforcement learning method, to transfer the skill policy of the source domain robot on the target 3C assembly task to the target domain robot.

[0087] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++, python, or conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0088] In the description of the application, reference to "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that a particular feature, structure, material, or characteristic being described is included in at least one embodiment or example of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment or example. Furthermore, the described specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples. Moreover, the usage of the terms "first", "second" or "third" does not limit the quantity or order of the specific features, structures, materials or characteristics, but rather the term "first", "second" or "third" can be used to distinguish different features, structures, materials or characteristics, which can be combined in any suitable manner. Furthermore, the singular forms "a", "an" and "the" include plural references unless the context clearly dictates otherwise.

[0089] Furthermore, the terms "first", "second", or the like, merely denote different instances of a similar feature, structure, material or characteristic, without necessarily implying any relative importance or any particular order. Thus, a feature defined with "first" or "second" can implicitly or explicitly include at least one of the features. The meaning of "a", "an" and "the" includes plural references unless the context clearly dictates otherwise.

[0090] Any process or method descriptions or blocks in flow charts or otherwise described herein represent embodiments which can be managed as one or more modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions or steps, and alternate implementations are possible. In some embodiments, the processes or methods described can be executed by one or more apparatuses or devices, either directly or after conversion to another language. Alternate implementations are possible.

[0091] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of instructions to implement logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a computer- readable storage medium or a computer-readable signal medium. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires (electrical connections), a portable computer diskette (a magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and stored in a computer memory.

[0092] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the various steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0093] Those skilled in the art can understand that all or part of the steps carried out by the above-mentioned embodiments can be completed by programs instructing related hardware, and the developed programs can be stored in a computer-readable storage medium, and the programs include one or a combination of the steps of the method embodiments when executed.

[0094] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0095] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for robot heterogeneous operation skill transfer learning, characterized in that, The method comprises the following steps: setting a pre-training task for robot heterogeneous operation skill transfer learning facing 3C assembly, and realizing skill strategy learning of robots in a source domain and robots in a target domain for the same pre-training task based on a reinforcement learning method, to obtain a source domain robot pre-training task skill strategy and a target domain robot pre-training task skill strategy; respectively executing the source domain robot pre-training task skill strategy and the target domain robot pre-training task skill strategy in a pre-training task operation environment, and recording source domain robot states and target domain robot states to obtain interaction training data; A source domain robot autoencoder and a target domain robot autoencoder are constructed. Each robot's autoencoder contains both an encoder and a decoder. Based on the interactive training data, a contrastive learning method combined with domain randomization is used to train both the source domain robot autoencoder and the target domain robot autoencoder simultaneously, resulting in trained source domain encoders and target domain encoders. These encoders map the source domain robot state and the target domain robot state to a common feature domain space, respectively. The source domain state s... S After mapping the source domain encoder f to the source domain autoencoder of the source domain robot, the source domain latent space state features f(s) are obtained. S ), target domain state s T After mapping the target domain encoder g through the target domain robot autoencoder, the latent space state features g(s) of the target domain are obtained. T By minimizing ||f(s) S )-g(s T The learning method of )||2 makes f(s) S ) and g(s T Mapping to a common feature domain space; employing a domain randomization method, noise data σ is collected from a Gaussian distribution N. S σ T The input source domain states s are respectively added to the source domain encoder f. S The input target domain state s corresponding to the target domain encoder g T To augment the distribution of input states; constructing a source domain robot skill strategy neural network and a target domain robot skill strategy neural network, training the source domain robot skill strategy neural network based on demonstration data and behavior cloning imitation learning methods for executing a target 3C assembly task, obtaining a skill strategy of the source domain robot on the target 3C assembly task, constructing a transfer reward function by using the trained source domain encoder and target domain encoder, and training the target domain robot skill strategy neural network by maximizing a sum of a target 3C assembly task reward function and the transfer reward function based on the reinforcement learning method, to transfer the skill strategy of the source domain robot on the target 3C assembly task to the target domain robot.

2. The transfer learning method of claim 1, wherein, The heterogeneous operation skill is different in skill form due to different configurations of the source domain robot and the target domain robot or different types of objects assembled by the source domain robot and the target domain robot, resulting in different dimensions of input state vectors or different dimensions of output action vectors of skill strategies required by the source domain robot and the target domain robot for executing respective tasks. 3.The method of claim 1, wherein, The pre-training task is an operation task for 3C parts. 4.The method of claim 1, wherein, The source domain robot pre-training task skill strategy and the target domain robot pre-training task skill strategy are obtained according to the following steps: The source domain robot pre-training task skill policy neural network and the target domain robot pre-training task skill policy neural network are constructed; the input of the source domain robot pre-training task skill policy neural network is a 2M+18-dimensional state vector The output is a 7-dimensional action vector a S The state vector contains the angles and angular velocities corresponding to M joints of the source domain robot, the 6-dimensional pose of the end effector of the source domain robot and the corresponding velocity, and the spatial 6-dimensional position and attitude of the 3C components to be assembled, the first 6 dimensions of the action vector a S are used to control the 6-dimensional position and attitude of the end effector of the source domain robot, and the seventh dimension is used to control the opening and closing of the end effector of the source domain robot; the input of the target domain robot pre-training task skill policy neural network is a 2L+18-dimensional state vector The output is a 7-dimensional action vector a T The state vector contains the angles and angular velocities corresponding to L joints of the robot in the target domain, the 6-dimensional pose of the end effector of the target domain robot and the corresponding velocity, and the spatial 6-dimensional position and attitude of the 3C components to be assembled, the first 6 dimensions of the action vector a T are used to control the 6-dimensional position and attitude of the end effector of the target domain robot, and the seventh dimension is used to control the opening and closing of the end effector of the target domain robot The source domain robot pre-training task skill strategy neural network and the target domain robot pre-training task skill strategy neural network initialized randomly are learned in a common pre-training task by using a reinforcement learning method, to obtain trained source domain robot pre-training task skill strategies and target domain robot pre-training task skill strategies.

5. The transfer learning method of claim 1, wherein, The interaction training data includes making the source domain robot and the target domain robot respectively execute a plurality of times the generated source domain robot pre-training task skill policy and the target domain robot pre-training task skill policy, recording the 2M+12-dimensional source domain robot state vector corresponding to each time step t in the execution process of the source domain robot pre-training task skill policy and the images of the working scene where the source domain robot is collected and the 2L+12-dimensional target domain robot state vector corresponding to each time step t in the execution process of the target domain robot pre-training task skill policy and the images of the working scene where the target domain robot is collected The source domain robot state vector contains the angles and angular velocities corresponding to the M joints of the source domain robot, and the 6-dimensional pose of the end effector of the source domain robot and the corresponding velocity contains the angles and angular velocities corresponding to the L joints of the target domain robot, and the 6-dimensional pose of the end effector of the target domain robot and the corresponding velocity 6. The transfer learning method of claim 5, wherein the set of parameters is determined based on a set of parameters of a pre-trained model. The source domain robot autoencoder is an AE S , the source domain encoder f and the source domain decoder D S , the input of the source domain encoder f is the source domain robot state vector and the image constituted by the source domain state s S , the source domain latent space state feature is output to the source domain decoder D S , and the source domain predicted state is obtained Let the target domain robot autoencoder be AE T , the target domain encoder g and the target domain decoder D T , the input of the target domain encoder g is the target domain robot state vector and the image , the target domain state s T , output the target domain latent space state feature to the target domain decoder D T , and get the target domain predicted state a source domain robot autoencoder AE S and a target domain robot autoencoder AE T The following loss function is used when training: wherein a, b are weight coefficients, N is a Gaussian distribution, representing respectively collecting source domain state s S and target domain state s T and sampling noise data s from Gaussian distribution N S and s T computed expectation value, the data experience pool B stores the interaction training data.

7. The transfer learning method of claim 1, wherein, The teaching data includes a teaching state vector and a teaching action vector The teaching state vector is recorded by teaching the source domain robot to perform a target 3C assembly task process several times at each time step t. The dimension of the source domain robot state vector is 2M+12, which includes the angles and angular velocities of the M joints of the source domain robot, and the 6-dimensional pose and corresponding velocity of the end effector. The dimension of the teaching action vector is 7, the first 6 dimensions are used to control the 6-dimensional position and attitude of the end effector of the source domain robot, and the seventh dimension is used to control the opening and closing of the end effector of the source domain robot. An input to the source domain robot skill policy neural network is the demonstration state vector An output is a 7-dimensional predicted action vector The predicted action vector The first 6 dimensions are used to control the 6-dimensional position and orientation of the source domain robot end-effector, and the 7th dimension is used to control the opening and closing of the source domain robot end-effector. When the source domain robot skill strategy neural network is learned by using a behavior cloning-based imitation learning method, the following loss function is used: wherein represents a sampled teaching state vector from a teaching pool D and a corresponding teaching action vector computed expected value, the teaching data being stored in the teaching pool D. 8.The method of claim 7, wherein, The input of the target domain robot skill strategy neural network is a transfer state vector, and the output is a 7-dimensional transfer action vector. The transfer state vector includes a target domain robot state vector and an image of a working scene where the target domain robot is located. The dimension of the target domain robot state vector is 2L+12, including angles and angular velocities corresponding to L joints, and a 6-dimensional pose of an end effector and a corresponding velocity. The first 6 dimensions of the transfer action vector are used to control a 6-dimensional position and pose of the end effector of the target domain robot, and the seventh dimension is used to control opening and closing of the end effector of the target domain robot. In the process of training the target domain robot skill policy neural network, the source domain robot and the target domain robot perform the target 3C assembly task according to the source domain robot skill policy and the target domain robot skill policy respectively, record the trajectory sequence in the process of completing the target 3C assembly task once respectively and and the state vector of the source domain robot and the target domain robot at time step τ in the process of completing the target 3C assembly task once respectively, and the following loss function is used: where γ and are discount factor and weight coefficient, respectively, γ τ denotes the τ-th power of the discount factor γ, is the transfer reward function for time step τ, r τ is the sparse reward function for the target 3C assembly task performed by the target domain robot at time step τ, is the entropy of the target domain robot skill policy, denotes sampling a transfer action vector a from the target domain robot skill policy trans the computed expectation value. 9.A method for robot heterogeneous operation skill transfer learning, characterized in that, The method comprises the following steps: The first module is configured to set a pre-training task of robot heterogeneous operation skill transfer learning for 3C assembly, and implement skill policy learning of a robot in a source domain and a robot in a target domain on the same pre-training task based on a reinforcement learning method, to obtain a source domain robot pre-training task skill policy and a target domain robot pre-training task skill policy. The second module is configured to control the source domain robot and the target domain robot to respectively execute the source domain robot pre-training task skill policy and the target domain robot pre-training task skill policy in a pre-training task operation environment, and record source domain robot states and target domain robot states, to obtain interaction training data. The third module is used to construct source domain robot autoencoders and target domain robot autoencoders. Each robot's autoencoder contains both an encoder and a decoder. Based on the interactive training data, a contrastive learning method combined with domain randomization is used to train both the source domain robot autoencoders and the target domain robot autoencoders simultaneously, resulting in trained source domain encoders and target domain encoders. These encoders map the source domain robot states and the target domain robot states to a common feature domain space, respectively. The source domain state s... S After mapping the source domain encoder f to the source domain autoencoder of the source domain robot, the source domain latent space state features f(s) are obtained. S ), target domain state s T After mapping the target domain encoder g through the target domain robot autoencoder, the latent space state features g(s) of the target domain are obtained. T By minimizing ||f(s) S )-g(s T The learning method of )||2 makes f(s) S ) and g(s T Mapping to a common feature domain space; employing a domain randomization method, noise data σ is collected from a Gaussian distribution N. S σ T The input source domain states s are respectively added to the source domain encoder f. S The input target domain state s corresponding to the target domain encoder g T To augment the distribution of input states; The fourth module is configured to construct a source domain robot skill policy neural network and a target domain robot skill policy neural network, train the source domain robot skill policy neural network based on teaching data of a target 3C assembly task and an action cloning imitation learning method, to obtain a skill policy of the source domain robot on the target 3C assembly task, construct a transfer reward function by using the trained source domain encoder and target domain encoder, and train the target domain robot skill policy neural network based on a reinforcement learning method by maximizing a sum of a target 3C assembly task reward function and the transfer reward function, to transfer the skill policy of the source domain robot on the target 3C assembly task to the target domain robot. The computer readable storage medium stores computer instructions for causing the computer to execute the robot heterogeneous operation skill transfer learning method in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, ​

Images