Robot imitation learning method, device, computer equipment and readable storage medium

Abstract

The embodiments of the present application provide a robot imitation learning method, apparatus, computer equipment, and readable storage medium. The method includes: obtaining demonstration data in chronological order and inputting it into the first encoder of the target model to extract high-dimensional features to obtain a first target feature; obtaining action instruction information corresponding to the demonstration data, decomposing the first target feature into multiple action blocks according to the action instruction information through a conditional variational autoencoder, and reconstructing the features of the multiple action blocks in sequence according to the action instruction information to obtain multiple corresponding first sub-features, splicing the multiple first sub-features to obtain a second target feature corresponding to the first target feature; predicting the second target feature through the first decoder of the target model to obtain the joint position parameters of the robot's joint parts at the next moment; and controlling the robot to imitate according to the joint position parameters. In this way, the accuracy of the robot's imitation learning can be improved.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a robot imitation learning method, apparatus, computer equipment, and readable storage medium. Background Art

[0002] Imitation learning is a machine learning method whose goal is to enable robots or intelligent agents to learn to perform tasks by analyzing and imitating the behavior of operators. For example, the imitation scenarios for robots to perform imitation learning include operational tasks (such as grasping, carrying, assembly, etc.), interactive tasks (such as imitating the behavior of operators to interact), and so on.

[0003] In related technologies, when robots perform imitation tasks, they typically use a trained model to analyze and imitate demonstration data. However, due to the large amount of information in the demonstration data, the model may be unable to distinguish important features during feature extraction, and may instead extract numerous features irrelevant to the imitation task. These irrelevant features interfere with the model's correct recognition, resulting in a decrease in the accuracy of the robot's imitation learning. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose a robot imitation learning method, device, computer equipment and readable storage medium, which can improve the accuracy of robot imitation learning.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application provides a robot imitation learning method, the method comprising:

[0006] Acquire demonstration data in chronological order, and input the demonstration data into a first encoder of a preset target model to extract high-dimensional features to obtain a first target feature;

[0007] Obtaining action indication information corresponding to the demonstration data, decomposing the first target feature into a plurality of action blocks according to the action indication information using a conditional variational autoencoder, sequentially reconstructing features of the plurality of action blocks according to the action indication information to obtain a plurality of corresponding first sub-features, and splicing the plurality of first sub-features in chronological order to obtain a second target feature corresponding to the first target feature;

[0008] Predicting the second target feature by the first decoder of the target model to obtain joint position parameters of the robot's joints at the next moment;

[0009] The robot is controlled to perform imitation according to the joint position parameters.

[0010] Accordingly, a second aspect of the embodiments of the present application provides a robot imitation learning device, comprising:

[0011] An extraction module is used to obtain demonstration data in chronological order and input the demonstration data into a first encoder of a preset target model to extract high-dimensional features to obtain a first target feature;

[0012] a reconstruction module, configured to obtain action indication information corresponding to the demonstration data, decompose the first target feature into a plurality of action blocks according to the action indication information through a conditional variational autoencoder, sequentially reconstruct features of the plurality of action blocks according to the action indication information to obtain a plurality of corresponding first sub-features, and concatenate the plurality of first sub-features in chronological order to obtain a second target feature corresponding to the first target feature;

[0013] a prediction module, configured to predict the second target feature by using the first decoder of the target model to obtain joint position parameters of the robot's joints at a next moment;

[0014] The imitation module is used to control the robot to imitate according to the joint position parameters.

[0015] In some embodiments, the reconstruction module is further configured to:

[0016] For each action block, obtaining, by a second encoder of a conditional variational autoencoder, a posterior distribution corresponding to the action block based on the action indication information and the action block;

[0017] Sampling the posterior distribution to obtain corresponding latent variables;

[0018] The second decoder of the conditional variational autoencoder is used to reconstruct the features of the multiple action blocks in a high-dimensional space in chronological order based on the latent variables and the action indication information to obtain corresponding multiple first sub-features.

[0019] In some embodiments, the reconstruction module is further configured to:

[0020] Obtaining a preset prior distribution, and determining a noise vector from the prior distribution;

[0021] Obtaining a mean and a variance of the posterior distribution, and determining a first product by the variance and the noise vector;

[0022] A corresponding latent variable is obtained according to the sum of the mean and the first product.

[0023] In some embodiments, the robot imitation learning device further includes an updating module for:

[0024] Acquiring a perspective image of the robot;

[0025] The demonstration data is updated based on the perspective image and the joint position parameters to obtain new demonstration data.

[0026] In some embodiments, the robot imitation learning device further includes a training module for:

[0027] Acquire sample demonstration data in chronological order, and input the sample demonstration data into a first encoder of a preset model to extract high-dimensional features to obtain a first sample target feature;

[0028] Obtaining sample action indication information corresponding to the sample demonstration data, decomposing the first sample target feature into a plurality of sample action blocks according to the sample action indication information using a conditional variational autoencoder, sequentially reconstructing features of the plurality of sample action blocks according to the sample action indication information to obtain a plurality of corresponding first sample sub-features, and splicing the plurality of first sample sub-features in chronological order to obtain a second sample target feature corresponding to the first sample target feature;

[0029] Determining a first loss of the preset model based on the first sample target feature and the second sample target feature;

[0030] Predicting the second sample target feature by the first decoder of the preset model to obtain sample joint position parameters of the robot's joint parts at the next moment;

[0031] Obtaining label joint position parameters of the robot's joints that need to be adjusted at a next moment, and determining a second loss of the preset model based on a difference between the sample joint position parameters and the label joint position parameters;

[0032] Constructing a target loss based on the first loss and the second loss;

[0033] The preset model is trained based on the target loss until a preset model convergence condition is reached to obtain a target model.

[0034] In some embodiments, the training module is further configured to:

[0035] For each sample action block, obtaining, by a second encoder of the conditional variational autoencoder, a sample posterior distribution corresponding to the sample action block based on the sample action indication information and the sample action block;

[0036] Sampling the sample posterior distribution to obtain corresponding sample latent variables;

[0037] The second decoder of the conditional variational autoencoder is used to reconstruct features of the multiple sample action blocks in a high-dimensional space in chronological order based on the sample latent variables and the sample action indication information to obtain corresponding multiple first sample sub-features.

[0038] In some embodiments, the training module is further configured to:

[0039] Obtaining a sample prior distribution of the conditional variational autoencoder;

[0040] Determining a first sub-loss according to a difference between the sample prior distribution and the sample posterior distribution;

[0041] determining a second sub-loss according to a difference between the second sample target feature and the first sample target feature;

[0042] A first loss of the preset model is determined based on the first sub-loss and the second sub-loss.

[0043] Correspondingly, the third aspect of the embodiments of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the robot imitation learning method described in any one of the embodiments of the first aspect of the present application when executing the computer program.

[0044] Correspondingly, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the robot imitation learning method described in any one of the embodiments of the first aspect of the present application.

[0045] The embodiment of the present application obtains demonstration data in chronological order and inputs the demonstration data into a first encoder of a preset target model for extracting high-dimensional features to obtain a first target feature; obtains action indication information corresponding to the demonstration data, decomposes the first target feature into multiple action blocks according to the action indication information through a conditional variational autoencoder, and reconstructs the multiple action blocks in sequence according to the action indication information to obtain corresponding multiple first sub-features, splices the multiple first sub-features in chronological order to obtain a second target feature corresponding to the first target feature; predicts the second target feature through a first decoder of the target model to obtain joint position parameters of the robot's joint at the next moment; and controls the robot to imitate according to the joint position parameters. In this way, after extracting the first target feature, the conditional variational autoencoder can be used to decompose the high-dimensional features into multiple action blocks according to the action indication information, so that the conditional variational autoencoder can focus on sub-features related to a specific action, enhance the interpretability of the model, and predict the joint position parameters of the reconstructed second target feature at the next moment through the first decoder to improve the coherence and timing of the imitation action, avoid error accumulation, and thus improve the accuracy of the robot's imitation learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of the architecture of the robot imitation learning system provided in an embodiment of the present application;

[0047] Figure 2 This is a flowchart of the robot imitation learning method provided by an embodiment of the present application;

[0048] Figure 3 This is an overall flow chart of the robot imitation learning method provided by the embodiment of the present application;

[0049] Figure 4 This is a schematic diagram of the functional modules of the robot imitation learning device provided in an embodiment of the present application;

[0050] Figure 5 This is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0052] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0054] Imitation learning is a machine learning method whose goal is to enable robots or intelligent agents to learn to perform tasks by analyzing and imitating the behavior of operators. For example, the imitation scenarios for robots to perform imitation learning include operational tasks (such as grasping, carrying, assembly, etc.), interactive tasks (such as imitating the behavior of operators to interact), and so on.

[0055] In related technologies, when robots perform imitation tasks, they typically use a trained model to analyze and imitate demonstration data. However, due to the large amount of information in the demonstration data, the model may be unable to distinguish important features during feature extraction, and may instead extract numerous features irrelevant to the imitation task. These irrelevant features interfere with the model's correct recognition, resulting in a decrease in the accuracy of the robot's imitation learning.

[0056] Based on this, the embodiments of the present application provide a robot imitation learning method, apparatus, computer equipment and readable storage medium to improve the accuracy of the robot's imitation learning.

[0057] The robot imitation learning method, apparatus, computer device, and readable storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the robot imitation learning system in the embodiments of the present application is described.

[0058] This application embodiment provides a robot imitation learning system, please refer to Figure 1 , in some embodiments, includes a terminal 11, a server 12 and a robot 13.

[0059] For example, terminal 11 can be a user interaction device, typically used by an operator to control robot 13. For example, terminal 11 can be a personal computer or tablet computer equipped with control interface software, a smartphone, a virtual reality device, etc. Terminal 11 can collect human demonstration data, including the joint positions and viewpoint images of robot 13, through cameras, sensors, and other devices, and display the real-time operation status of robot 13. Furthermore, terminal 11 can also transmit data to server 12 and receive instructions or results returned by server 12.

[0060] Exemplarily, the server side 12 can be a cloud server, a local server, etc. The server side can receive instructions sent by the terminal 11, and run the target model according to these instructions to perform action segmentation, feature reconstruction and action prediction, etc., and generate the next action instructions of the robot 13 based on the results of the model prediction.

[0061] Furthermore, the robot 13 can perform specific action tasks, such as grasping, carrying, assembling, etc., according to the instructions received from the server 12, and feed back the status information (such as joint position, environmental perception data, etc.) during the execution process to the terminal or server for real-time adjustment and optimization.

[0062] In some embodiments, the robot 13 may also have an integrated computing architecture that includes the functions of the terminal 11 and the server 12, without the need for additional terminal 11 and server 12. Specifically, the robot 13 may collect human demonstration data through built-in cameras and sensors, process the collected data to obtain specific joint position parameters, and transmit the predicted joint position parameters as instructions to the actuator to perform the predicted action.

[0063] The robot imitation learning method in the embodiments of the present application can be illustrated by the following embodiments.

[0064] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0065] In the embodiment of the present application, the robot imitation learning device will be described from the perspective of the robot imitation learning device, which can be integrated into a computer device. Figure 2 , Figure 2 This is a flowchart of the steps of the robot imitation learning method provided in an embodiment of the present application. In this embodiment of the present application, the robot imitation learning device is specifically integrated into a terminal or server as an example. When the processor on the terminal or server executes the program instructions corresponding to the robot imitation learning method, the specific process is as follows:

[0066] Step 101: Acquire demonstration data in chronological order, and input the demonstration data into a first encoder of a preset target model to extract high-dimensional features to obtain first target features.

[0067] In some embodiments, in order to capture the global dependencies in the input demonstration data so that the target model can understand the input data more comprehensively, the first encoder of the target model can extract high-dimensional features of the demonstration data to obtain the first target features, so as to improve the accuracy and fluency of the imitation learning task.

[0068] The demonstration data may be data obtained by preprocessing the robot's joint position parameters, view images, or other sensor data.

[0069] The target model may be a trained neural network model, which is composed of multiple components, including at least a first encoder, a conditional variational autoencoder, and a first decoder.

[0070] The first encoder may be a Transformer encoder, which is used to extract a high-dimensional feature representation, that is, a first target feature, from the demonstration data.

[0071] The first target feature may be a sequence of high-dimensional feature representations extracted from the demonstration data by the first encoder.

[0072] For example, when obtaining demonstration data, data of technicians performing tasks at different time steps can be collected from cameras, sensors and other devices in chronological order to facilitate robot imitation and prediction, and the robot's joint position parameters and view images can be recorded to obtain raw data. Afterwards, the raw data can be standardized to ensure that the data has zero mean and unit variance to obtain demonstration data. For example, assuming the raw data is X = {x1, x2, ..., x T},in, Represents the d-dimensional feature vector at time step t, and the original data is standardized to obtain the demonstration data X′=

[0073] ″′

[0074] {x1,x2,...,x T The process of} is as follows:

[0075]

[0076] Among them, μ represents the mean of the original data, and σ represents the standard deviation of the original data.

[0077] Furthermore, after obtaining the demonstration data, the demonstration data X′={x1′,x2′,...,x′ T} is input into the first encoder of the preset target model, that is, the Transformer encoder, to extract high-dimensional features and obtain the first target feature H = {h1,h2,...,h T},in, represents the high-dimensional feature representation at time step t.

[0078] Exemplarily, the process of the first encoder calculating and obtaining the first target feature is as follows:

[0079] H=TransformerEncoder(X′);

[0080] Each layer of the first encoder consists of a multi-head self-attention mechanism and a feedforward neural network. The specific calculation process is as follows:

[0081] H l =LayerNorm(H (l-1) +MultiHeadAttention(H (l-1) ));

[0082] H l =LayerNorm(H (l) +FeedForward(H (l) ));

[0083] Among them, H (l) Represents the output of the lth layer, MultiHeadAttention represents the multi-head self-attention mechanism, FeedForward represents the feedforward neural network, and LayerNorm represents layer normalization.

[0084] Combined with the above process formula for calculating the first target feature, assuming that the first encoder has L layers, the calculation process of each layer is as follows: for the demonstration data H (l-1) , calculate the output of the multi-head self-attention mechanism MultiHeadAttention(H (l-1) ), the multi-head attention mechanism will H (l-1)Divide it into multiple heads, perform self-attention calculation on each head, then concatenate the results and perform linear transformation to obtain the output of the multi-head self-attention mechanism. (l-1) Add them together to get H l Furthermore, the output of the feedforward neural network can be obtained and compared with H l After L layers of calculation, the output of the last layer of the first encoder is the first target feature.

[0085] By using the first encoder to extract high-dimensional feature representations from preprocessed demonstration data, the algorithm not only preserves temporal information but also generates rich contextual feature representations, known as first target features. These first target features more comprehensively reflect the complexity of action sequences, providing more precise guidance in subsequent action segmentation and prediction tasks, significantly improving the accuracy and efficiency of robot imitation learning.

[0086] Step 102: obtain action indication information corresponding to the demonstration data, decompose the first target feature into multiple action blocks according to the action indication information through a conditional variational autoencoder, and reconstruct the features of the multiple action blocks in sequence according to the action indication information to obtain multiple corresponding first sub-features, and splice the multiple first sub-features in chronological order to obtain a second target feature corresponding to the first target feature.

[0087] In some embodiments, in order to improve prediction accuracy and reduce cumulative errors, the first target feature can be divided into multiple action blocks according to the action indication information, and the prediction of each action block is relatively independent, so as to facilitate more refined processing of each subtask. In addition, by introducing conditional variables and generating latent space representations of each action block through conditional variational autoencoders, the target model's ability to understand the first target feature can be enhanced, thereby improving the accuracy of action prediction.

[0088] The action indication information can be additional information or features used to influence the generation of the second target feature. The action indication information is a conditional variable, which can be a label, a data-specific feature, or other feature. It is used to enable the second encoder of the conditional variational autoencoder to consider specific context or conditions when generating low-dimensional data. The action indication information provides additional contextual information to help the target model better understand and generate the second target feature that meets the specific conditions.

[0089] Among them, the conditional variational autoencoder (CVAE) can be a generative model, which introduces conditional variables so that the generated second target feature depends not only on the latent variable but also on the given conditions.

[0090] Action blocks can be independent action units, each representing a subtask or an action phase, such as "grasping," "carrying," "placing," etc. By dividing actions into blocks, each subtask can be processed more finely, improving prediction accuracy and reducing cumulative error.

[0091] The first sub-feature may be a feature representation obtained by reconstructing the action block. The first sub-feature retains key information of each action block and considers the influence of action indication information during the reconstruction process.

[0092] The second target feature may be a new high-dimensional feature representation obtained by concatenating multiple first sub-features in chronological order or mapping them in chronological order.

[0093] In some embodiments, a blocking mechanism can be introduced into the second encoder of the conditional variational autoencoder, so that the second encoder can decompose the first target feature into multiple action blocks according to the action indication information, and a merging mechanism can be introduced into the second decoder of the conditional variational autoencoder, so that the first sub-features generated by each action block can be spliced ​​into a second target feature in chronological order.

[0094] In some embodiments, the first target feature can be divided into multiple action blocks according to the action block algorithm, and then input into the conditional variational autoencoder in chronological order to obtain the first sub-feature corresponding to each action block, and the multiple first sub-features can be spliced ​​in chronological order to obtain the corresponding second target feature.

[0095] Furthermore, when segmenting the first target feature, the first target feature can be segmented based on the time steps corresponding to each feature in the first target feature to obtain fixed-length action blocks. Alternatively, the action blocks can be dynamically segmented based on the action instruction information. For example, if the action instruction information of the imitation learning task is "grab," "carry," or "place," each action instruction information can correspond to an action block. Alternatively, the first target feature can be segmented based on occupied space, so that each resulting action block has the same occupied space.

[0096] For example, the first target feature and action indication information can be input into a conditional variational autoencoder, and the first target feature can be task-decomposed based on the action indication information to obtain multiple action blocks. For each action block, the second encoder of the conditional variational autoencoder can map each action block to a latent space in chronological order, generate a posterior distribution of the corresponding latent variable, and sample from the posterior distribution to obtain the latent variable.

[0097] Furthermore, reconstruction can be performed based on the latent variables and action indication information to generate a first sub-feature for each action block. Furthermore, all first sub-features can be concatenated in chronological order to obtain a second target feature that is a reconstruction of the first target feature. This allows for smoother and more natural transitions between actions by reconstructing features of multiple action blocks in chronological order, while effectively preventing the accumulation of errors.

[0098] By decomposing the first target feature into multiple independent action blocks, each subtask can be processed more finely. Each action block is reconstructed through the second encoder and second decoder of the conditional variational autoencoder. The generated first sub-feature retains the key information. Finally, the first sub-features are spliced ​​in chronological order to obtain the second target feature. This block division and reconstruction method not only makes the transition between actions smoother and more natural, but also effectively prevents the accumulation of errors, thereby improving the overall performance and robustness of the system.

[0099] In some embodiments, to more finely process each subtask, a posterior distribution can be generated based on the action indication information and the action block, and latent variables can be sampled from the posterior distribution. Feature reconstruction can then be performed accordingly. This can refine the granularity of feature processing and make the extraction of important features more targeted and accurate. For example, the "reconstructing features of multiple action blocks in sequence according to the action indication information to obtain corresponding multiple first sub-features" in step 102 may include:

[0100] (102.1) For each action block, obtain the posterior distribution corresponding to the action block based on the action indication information and the action block through the second encoder of the conditional variational autoencoder;

[0101] (102.2) Sampling the posterior distribution to obtain the corresponding latent variable;

[0102] (102.3) Through the second decoder of the conditional variational autoencoder, based on the latent variables and the action indication information, the multiple action blocks are reconstructed in a high-dimensional space in chronological order to obtain the corresponding multiple first sub-features.

[0103] The second encoder can be part of a conditional variational autoencoder, which is used to map the action blocks and action indication information into a latent space and generate a posterior distribution of the latent variables. The second encoder is usually composed of a multi-layer neural network, such as a fully connected layer or a convolutional layer.

[0104] The posterior distribution may be the posterior distribution of the latent variable output by the second encoder in the conditional variational autoencoder. The posterior distribution includes a mean and a variance, and is used to describe the distribution of the latent variable.

[0105] Among them, the latent variable can be a specific numerical representation sampled from the posterior distribution, and the latent variable is a point in the latent space, representing a compressed representation of the action block and action indication information.

[0106] The second decoder can be part of a conditional variational autoencoder, configured to generate a reconstructed feature representation, i.e., the first sub-feature, from the latent variables and the action indication information. The second decoder is also composed of a multi-layer neural network, such as a fully connected layer or a convolutional layer.

[0107] For example, if the robot needs to complete the operation of "grabbing a cup and putting it on the table", the action instruction information includes three action labels: grabbing, carrying, and placing. After obtaining the first target feature H corresponding to the demonstration data, the first target feature and the action instruction information can be input into the conditional variational autoencoder. The conditional variational autoencoder can first decompose H into three action blocks according to the action instruction information, namely, action block h1 (grabbing), action block h2 (carrying), and action block h3 (placing).

[0108] Furthermore, the action blocks can be input into the second encoder in chronological order in combination with the action indication information. Taking the action block h1 as an example, the second encoder can output the posterior distribution μ1 and μ2 of the latent variables of the corresponding action block. Among them, μ1 represents the mean of the posterior distribution of the latent variable z1, represents the variance of the posterior distribution of the latent variable z1. Then the posterior distribution can be expressed as q φ (z|h1,c), where c represents action instruction information.

[0109] Furthermore, we can get the posterior distribution q φ (z|h1,c) is sampled to obtain the latent variable z1. Specifically, the latent variable z1 can be sampled from the posterior distribution using the reparameterization technique. The latent variable z1 and the action indication information can then be input into the second decoder of the conditional variational autoencoder for reconstruction, obtaining the first sub-feature corresponding to the action block h1.

[0110] By using a conditional variational autoencoder to reconstruct the features of each action block, the target model can better understand and generate data that conforms to the action instruction information, thereby improving the robustness and generalization ability of the target model.

[0111] In some embodiments, to ensure the randomness and diversity of the latent variables and to make them conform to the statistical characteristics of the input data, a noise vector can be sampled from a preset standard normal prior distribution and scaled and offset based on the mean and variance of the posterior distribution to obtain the latent variables, thereby improving the robustness and generalization ability of the generative model. For example, (102.2) may include:

[0112] (102.2.1) Obtain a preset prior distribution and determine a noise vector from the prior distribution;

[0113] (102.2.2) Obtain the mean and variance of the posterior distribution and determine the first product using the variance and the noise vector;

[0114] (102.2.3) According to the sum of the mean and the first product, the corresponding latent variable is obtained.

[0115] The prior distribution can be the assumed distribution of the latent variable before any data is observed. In the conditional variational autoencoder, it is usually assumed that the prior distribution is a standard normal distribution. Where I is the identity matrix. The prior distribution provides a benchmark so that the latent variables can maintain a certain degree of randomness and diversity.

[0116] The noise vector can be a random vector sampled from a prior distribution to introduce randomness.

[0117] Among them, the mean can be the posterior distribution q φ The expected value and mean of (z|h,c) represent the central position of the latent variable given the action block and action instruction information.

[0118] The variance may be the variance of the posterior distribution, which represents the uncertainty or degree of variation of the latent variable given the action block and action instruction information.

[0119] The first product may be a result obtained by element-wise multiplication of the variance and the noise vector, and is used to adjust the scale of the noise vector to match the variance of the posterior distribution.

[0120] For example, assume that the prior distribution is a standard normal distribution Next, based on the dimension of the latent variable, a noise vector ∈ can be sampled from a standard normal distribution with the same dimension as the latent variable. Furthermore, the mean and variance of the posterior distribution output by the second encoder can be obtained, and the standard deviation of the posterior distribution can be obtained from the variance. The standard deviation is then element-wise multiplied by the noise vector to adjust the noise scale, resulting in a first product. The mean and the first product can then be added to obtain the latent variable z1.

[0121] Through the above methods, a variety of latent variables can be generated, ensuring that the generated latent variables not only conform to the statistical characteristics of the action blocks and action instruction information, but also have a certain degree of variability, thereby improving the accuracy of the target model in predicting actions.

[0122] Step 103 : predicting the second target feature through the first decoder of the target model to obtain the joint position parameters of the robot's joints at the next moment.

[0123] In some embodiments, in order to accurately predict the robot joint position parameters, the second target feature can be predicted by the first decoder of the target model to obtain the joint position parameters at the next moment, thereby enabling the robot to perform complex tasks more flexibly and accurately.

[0124] Among them, the first decoder can be a Transformer decoder, which is responsible for predicting the joint position parameters at the next moment based on the latent space representation.

[0125] The robot may be a robot with a physical form that can interact with the environment in the real world, such as a handling robot, an arm-operating robot, etc. For example, the robot's mechanical arm may be obtained by 3D printing to save assembly costs.

[0126] Joints are the movable parts of a robot's body. They can be driven by motors or servo systems, enabling the robot to perform various actions. For example, for a simple robotic arm, the joints might include the shoulder, elbow, and wrist.

[0127] Joint position parameters refer to the specific position and posture of each joint in a robot. Joint position parameters are typically expressed as angles or displacements and are used to control the robot's motion. For a robotic arm, these parameters might include the angles of the shoulder, elbow, and wrist joints.

[0128] For example, the second target feature may be Y={y1,y2,...,y T}in, Represents the d-dimensional target joint position at time step t. The second target feature is predicted by the first decoder of the target model to obtain the joint position parameters of the robot's joint at the next moment. The calculation process is as follows:

[0129] Y l =LayerNorm(Y (l-1) +MultiHeadAttention(Y (l-1) ));

[0130] Y l =LayerNorm(Y (l) +EnconderDecoderAttention(Y (l) , H));

[0131] Y l =LayerNorm(Y (l) +FeedForward(Y (l) ));

[0132] Among them, Y (l) Represents the output of the lth layer, MultiHeadAttention represents the multi-head self-attention mechanism, EnconderDecoderAttention represents the encoder-decoder attention mechanism, FeedForward represents the feedforward neural network, and LayerNorm represents layer normalization.

[0133] Exemplarily, the first decoder may include a multi-layer self-attention mechanism, an encoder-decoder attention mechanism, and a feedforward neural network. (l-1) represents the input to layer l. The multi-head self-attention mechanism in the first decoder can be used to capture dependencies within the decoder. The encoder-decoder attention mechanism can be used to capture the dependencies between the second target feature and the current state of the first decoder. Feedforward neural networks can be used for nonlinear transformations. Layer normalization can be used to normalize the output of each layer, improving the stability of the target model. After multiple layers of processing, the output of the last layer of the first decoder is passed to a fully connected layer, which maps the hidden state of the first decoder to the robot's joint position parameters.

[0134] Through the above method, the first decoder can generate the joint position parameters at the next moment, so as to facilitate the subsequent control of the robot to perform corresponding actions.

[0135] Step 104: Control the robot to perform imitation according to the joint position parameters.

[0136] In some embodiments, in order to achieve high-precision task execution, the robot can be controlled to perform imitation according to joint position parameters to enhance the robot's performance in complex and dynamic environments.

[0137] Exemplarily, the joint position parameters can be the specific position and posture of each joint of the robot, which can be an angle (for rotational joints) or a linear displacement (for translational joints). After sending the joint position parameters to the robot's control system, the robot can predict the motion trajectory based on the joint position parameters and perform corresponding actions by moving the base or controlling the robotic arm.

[0138] The embodiment of the present application obtains demonstration data in chronological order and inputs the demonstration data into a first encoder of a preset target model for extracting high-dimensional features to obtain a first target feature; obtains action indication information corresponding to the demonstration data, decomposes the first target feature into multiple action blocks according to the action indication information through a conditional variational autoencoder, and reconstructs the multiple action blocks in sequence according to the action indication information to obtain corresponding multiple first sub-features, splices the multiple first sub-features in chronological order to obtain a second target feature corresponding to the first target feature; predicts the second target feature through a first decoder of the target model to obtain joint position parameters of the robot's joint at the next moment; and controls the robot to imitate according to the joint position parameters. In this way, after extracting the first target feature, the conditional variational autoencoder can be used to decompose the high-dimensional features into multiple action blocks according to the action indication information, so that the conditional variational autoencoder can focus on sub-features related to a specific action, enhance the interpretability of the model, and predict the joint position parameters of the reconstructed second target feature at the next moment through the first decoder to improve the coherence and timing of the imitation action, avoid error accumulation, and thus improve the accuracy of the robot's imitation learning.

[0139] In some embodiments, in order to improve the accuracy and success rate of task execution, the robot's operation can be continuously adjusted and optimized based on real-time visual feedback, so that the prediction of the joint position parameters at the next moment is more accurate, thereby improving the robot's imitation ability. For example, after step 104, the following steps may also be included:

[0140] (A.1) Obtain the robot's perspective image;

[0141] (A.2) Update the demonstration data based on the view image and joint position parameters to obtain new demonstration data.

[0142] The perspective image can be an image of the environment captured by one or more cameras installed on the robot. The perspective image can be an RGB image, a depth image, or other types of visual data, used to provide visual information about the robot's surroundings.

[0143] For example, multiple cameras may be used to capture images from different perspectives to provide more comprehensive environmental information.

[0144] Specifically, the current position parameters of each joint can be obtained from the robot's control system, or the joint position parameters can be directly obtained when the joint position parameters corresponding to the second target feature are generated by the first decoder, and the perspective image and the joint position parameters are fused to form a multimodal data set to provide more comprehensive environment and robot status information.

[0145] Furthermore, multimodal fusion techniques (such as feature-level fusion, decision-level fusion, etc.) can be used to integrate data from different sources, and the fused data can be used as new demonstration data, so that the target model can re-predict the joint position parameters at the next moment based on the new demonstration data, thereby improving the consistency and accuracy of the robot's actions.

[0146] Through the above methods, the consistency and accuracy of the robot's movements can be improved, and the accumulation of errors can be avoided. At the same time, the robot's performance in complex and dynamic environments, as well as the efficiency and accuracy of task execution can be significantly improved.

[0147] In some embodiments, in order to enable the model to accurately predict the joint position parameters of the robot at the next moment, thereby achieving efficient, accurate and consistent task execution, the preset model can be trained to obtain a target model so that the model can predict the correct joint position parameters at the next moment. Exemplarily, the target model is trained in the following way:

[0148] (B.1) acquiring sample demonstration data in chronological order, and inputting the sample demonstration data into a first encoder of a preset model to extract high-dimensional features, thereby obtaining a first sample target feature;

[0149] (B.2) obtaining sample action indication information corresponding to the sample demonstration data, decomposing the first sample target feature into a plurality of sample action blocks according to the sample action indication information using a conditional variational autoencoder, sequentially reconstructing features of the plurality of sample action blocks according to the sample action indication information to obtain a plurality of corresponding first sample sub-features, and concatenating the plurality of first sample sub-features in chronological order to obtain a second sample target feature corresponding to the first sample target feature;

[0150] (B.3) determining a first loss of a preset model based on the first sample target feature and the second sample target feature;

[0151] (B.4) predicting the second sample target feature using the first decoder of the preset model to obtain sample joint position parameters of the robot's joint parts at the next moment;

[0152] (B.5) obtaining label joint position parameters of the robot's joint parts that need to be adjusted at the next moment, and determining a second loss of the preset model based on the difference between the sample joint position parameters and the label joint position parameters;

[0153] (B.6) constructing a target loss based on the first loss and the second loss;

[0154] (B.7) The preset model is trained based on the target loss until the preset model convergence condition is reached to obtain the target model.

[0155] Among them, the sample demonstration data can be data collected and saved from the operator or in advance, including joint position parameters, view images and other sensor data. These data are used to train the preset model after preprocessing (such as standardization).

[0156] The preset model may be a pre-designed neural network model, and the preset model may include a first encoder, a conditional variational autoencoder, and a first decoder.

[0157] Among them, the first sample target feature can be a high-dimensional feature representation extracted from the sample demonstration data by the first encoder of the preset model, which is used to capture the global dependencies and complex patterns in the sample demonstration data.

[0158] The sample action indication information is additional information or features used to influence the generation result during the generation of the second sample target feature. The sample action indication information can be a label, a unique feature of the data, etc., used to help the preset model better understand and generate features that meet specific conditions.

[0159] The sample action blocks may be independent action units, and each action block represents a subtask or an action stage (eg, "grabbing," "carrying," "placing," etc.).

[0160] The first sample sub-feature may be a feature representation obtained by reconstructing the sample action block. The first sample sub-feature retains key information of each action block and considers the influence of the sample action indication information during the reconstruction process.

[0161] The second sample target feature can be a new high-dimensional feature representation obtained by splicing or mapping multiple first sample sub-features in chronological order. The second sample target feature provides a more comprehensive and coherent feature representation, which is helpful for subsequent action prediction.

[0162] The first loss may be a loss function calculated based on the difference between the first sample target feature and the second sample target feature. The first loss measures the KL divergence loss of the second encoder of the conditional variational autoencoder and the reconstruction loss of the second decoder.

[0163] The sample joint position parameters may be joint position parameters obtained by predicting the second sample target features through the first decoder of the preset model, and are used to control the movement of the robot.

[0164] Among them, the label joint position parameters can be the real joint position parameters of the sample demonstration data, which are used for supervised learning.

[0165] Among them, the second loss can be a loss function calculated based on the difference between the sample joint position parameters and the label joint position parameters, which is used to measure the difference between the sample joint position parameters predicted by the preset model and the true value, ensuring that the preset model can accurately predict the joint position.

[0166] The target loss may be a total loss function constructed based on the sum of the first loss and the second loss.

[0167] It can be understood that the process of inputting the sample demonstration data into the first encoder of the preset model to extract high-dimensional features and obtain the first sample target features is similar to the process of inputting the demonstration data into the first encoder of the target model to extract high-dimensional features and obtain the first target features. For details, please refer to the above embodiment and will not be repeated here.

[0168] In some embodiments, a blocking mechanism can be introduced into the second encoder of the conditional variational autoencoder, so that the second encoder can decompose the first sample target feature into multiple sample action blocks according to the sample action indication information, and a merging mechanism can be introduced into the second decoder of the conditional variational autoencoder, so that the first sample sub-features generated by each sample action block can be spliced ​​into a second sample target feature in chronological order.

[0169] In some embodiments, the first sample target feature can be divided into multiple sample action blocks according to the action block algorithm, and then input into the conditional variational autoencoder in chronological order to obtain the first sample sub-feature corresponding to each sample action block, and the multiple first sample sub-features can be spliced ​​in chronological order to obtain the corresponding second sample target feature.

[0170] Furthermore, when the first sample target feature is segmented, the first sample target feature can be segmented based on the time step corresponding to each feature in the first sample target feature to obtain sample action blocks of fixed length. Alternatively, the sample action blocks can be dynamically segmented based on the sample action indication information. For example, if the sample action indication information of the imitation learning task is "grab," "carry," or "place," each sample action indication information can correspond to a sample action block. Alternatively, the first sample target feature can be segmented based on occupied space, so that each sample action block obtained by segmentation has the same occupied space.

[0171] For example, the first sample target feature and the sample action indication information can be input into a conditional variational autoencoder, and the first sample target feature can be task-decomposed based on the sample action indication information to obtain multiple sample action blocks. For each sample action block, the second encoder of the conditional variational autoencoder can map each sample action block to a latent space in chronological order, generate a posterior distribution of the corresponding latent variable, and sample from the posterior distribution to obtain the latent variable.

[0172] Furthermore, reconstruction can be performed based on the latent variables and the sample action indication information to generate a first sample sub-feature after reconstruction of each sample action block. Furthermore, all first sample sub-features can be concatenated in chronological order to obtain a second sample target feature after reconstruction of the first sample target feature. In this way, by reconstructing features of multiple sample action blocks in chronological order, the transition between actions becomes smoother and more natural, while effectively preventing the accumulation of errors.

[0173] Exemplarily, the sample prior distribution of the conditional variational autoencoder can be obtained, and based on the difference between the sample prior distribution and the sample posterior distribution, the first sub-loss, that is, the KL divergence loss of the second encoder, is determined; and based on the difference between the second sample target feature and the first sample target feature, the second sub-loss, that is, the reconstruction loss of the second decoder is determined; based on the first sub-loss and the second sub-loss, the first loss of the preset model is determined, so that the conditional variational autoencoder can generate samples similar to the input data (first sample target feature) under given conditions, and maintain the smoothness and diversity of the latent variable distribution.

[0174] Specifically, the first loss corresponding to the conditional variational autoencoder is It can be calculated by the following formula:

[0175]

[0176] Among them, -KL(q(z t |h t )‖p(z t)) represents the first sub-loss, KL(q(z t |h t )‖p(z t )) is the potential distribution q(z) output by the second encoder t |h t ) and the KL divergence between the sample prior distribution, q(z t |h t ) is the sample posterior distribution of the sample latent variable, p(z t ) is the sample prior distribution of the sample latent variable, represents the second child loss, h t is the first sample target feature, p(h t |z t ) is the latent variable z of a given sample t When generating h t The probability distribution of .

[0177] It can be understood that by minimizing the first loss of the conditional variational autoencoder, a preset model can be obtained that can not only learn a structured latent space, but also achieve high-quality reconstruction and generate specific types of data under given conditions.

[0178] It can be understood that when the first sample feature needs to be divided into multiple sample action blocks, it is only necessary to calculate the sub-loss for each sample sub-action block according to the above formula, and then add each sub-loss to obtain the first loss.

[0179] It should be noted that the solution of predicting the second sample target feature through the first decoder of the preset model to obtain the sample joint position parameters of the robot's joint parts at the next moment is similar to the above embodiment of predicting the second target feature through the first decoder of the target model to obtain the joint position parameters of the robot's joint parts at the next moment. Please refer to the above for details and will not be repeated here.

[0180] Furthermore, after calculating the sample joint position parameters, the second loss can be calculated based on the difference between the sample joint position parameters and the label joint position parameters. The second loss can be calculated by the mean square error or L1 loss. Taking L1 loss as an example, the second loss can be calculated as follows:

[0181]

[0182] Among them, y t represents the sample joint position parameters predicted at time step t, Denotes the label joint position parameters at time step t, and T denotes the sum of the time steps of the first sample target feature. By minimizing the second loss, the accuracy of the preset model's prediction of the sample joint position parameters can be improved.

[0183] Furthermore, the target loss can be obtained by adding the first loss and the second loss, which is and Adding them together, we can get the target loss.

[0184] Furthermore, the model convergence condition may be that the preset model has been trained a preset number of times, such as 100 times, 400 times, etc. Alternatively, the model convergence condition may be that the target loss is less than the preset loss for more than a threshold number of times, such as the target loss is less than the preset loss for more than 10 times, etc. The preset loss can be set according to actual conditions and is not specifically limited here.

[0185] It is understandable that when the number of times the preset model is trained reaches the preset model convergence condition, the training can be stopped to obtain the target model.

[0186] By processing the first sample target features in blocks and then concatenating the first sample sub-features in chronological order during reconstruction, it helps reduce the accumulation of errors in long time series and effectively enhances the smoothness and naturalness of transitions between actions. Furthermore, the target loss formed by combining the first loss (involving latent space regularization and reconstruction quality) with the second loss (focusing on joint position parameter prediction) jointly drives the model training process, which is conducive to obtaining a target model that can both accurately predict joint positions and generate smooth and natural action sequences.

[0187] In some embodiments, in order to process each subtask more finely, a sample posterior distribution can be generated based on the sample action indication information and the sample action block, and sample latent variables can be sampled therefrom to obtain feature reconstruction. In this way, the processing granularity of the features can be refined, making the extraction of important features more targeted and accurate. For example, in (B.2), "feature reconstruction of multiple sample action blocks in sequence according to the sample action indication information to obtain corresponding multiple first sample sub-features" may include:

[0188] (B.2.1) for each sample action block, obtaining a sample posterior distribution corresponding to the sample action block based on the sample action indication information and the sample action block using the second encoder of the conditional variational autoencoder;

[0189] (B.2.2) Sampling the sample posterior distribution to obtain the corresponding sample latent variable;

[0190] (B.2.3) Through the second decoder of the conditional variational autoencoder, based on the sample latent variables and the sample action indication information, the features of the multiple sample action blocks are reconstructed in the high-dimensional space in chronological order to obtain the corresponding multiple first sample sub-features.

[0191] The sample posterior distribution may be a sample posterior distribution of a sample latent variable output by the second encoder in the conditional variational autoencoder. The sample posterior distribution includes a mean and a variance, and is used to describe the distribution of the sample latent variable.

[0192] The sample latent variable may be a specific numerical representation sampled from the sample posterior distribution. The sample latent variable is a point in the latent space, representing a compressed representation of the sample action block and the sample action indication information.

[0193] The second decoder can be part of a conditional variational autoencoder, configured to generate a reconstructed feature representation, i.e., the first sample sub-feature, from the sample latent variable and the sample action indication information. The second decoder is also composed of a multi-layer neural network, such as a fully connected layer or a convolutional layer.

[0194] For example, if the robot needs to complete the operation of "grabbing a cup and putting it on the table", the sample action instruction information includes three action labels: grabbing, carrying, and placing. After obtaining the first target feature H corresponding to the demonstration data, the first target feature and the sample action instruction information can be input into the conditional variational autoencoder. The conditional variational autoencoder can first decompose H into three sample action blocks according to the sample action instruction information, namely, sample action block h1 (grabbing), sample action block h2 (carrying), and sample action block h3 (placing).

[0195] Furthermore, the sample action blocks can be input into the second encoder in chronological order in combination with the sample action indication information. Taking the sample action block h3 as an example, the second encoder can output the sample posterior distribution μ1 and μ2 of the sample latent variable of the corresponding sample action block. Among them, μ1 represents the mean of the sample posterior distribution of the sample latent variable z1, represents the variance of the sample posterior distribution of the sample latent variable z1. Then the sample posterior distribution can be expressed as q φ (z|h1,c), where c represents the sample action indication information.

[0196] Furthermore, we can get the sample posterior distribution q φ (z|h1,c) is sampled to obtain the sample latent variable z1. Specifically, the sample latent variable z1 can be sampled from the sample posterior distribution using the reparameterization technique. The sample latent variable z1 and the sample action indication information can then be input into the second decoder of the conditional variational autoencoder for reconstruction to obtain the first sample sub-feature corresponding to the sample action block h1.

[0197] By using the conditional variational autoencoder to reconstruct the features of each sample action block, the target model can better understand and generate data that conforms to the sample action indication information, thereby improving the robustness and generalization ability of the preset model.

[0198] In some embodiments, to continuously optimize model performance, the prior distribution of samples can be obtained, and with the help of encoder and decoder optimization, the distribution differences and feature reconstruction of samples can be evaluated to better learn the potential representation of the data and achieve accurate sample reconstruction. For example, (B.3) may include:

[0199] (B.3.1) Obtain the sample prior distribution of the conditional variational autoencoder;

[0200] (B.3.2) Determine the first sub-loss based on the difference between the sample prior distribution and the sample posterior distribution;

[0201] (B.3.3) determining a second sub-loss based on a difference between the second sample target feature and the first sample target feature;

[0202] (B.3.4) Based on the first sub-loss and the second sub-loss, determine a first loss of a preset model.

[0203] The sample prior distribution can be a known prior distribution, which represents the distribution in the latent space without any input sample information.

[0204] Among them, the first sub-loss can be a KL divergence loss, which is used to measure the difference between the sample potential distribution output by the second encoder and the sample prior distribution, that is, the difference between the sample posterior distribution and the sample prior distribution.

[0205] The second sub-loss may be a reconstruction loss, which is used to measure the difference between the generated second sample target feature and the first sample target feature.

[0206] Exemplarily, the first loss corresponding to the conditional variational autoencoder is It can be calculated by the following formula:

[0207]

[0208] Among them, -KL(q(z t |h t )‖p(z t )) represents the first sub-loss, KL(q(z t |h t )‖p(z t )) is the potential distribution q(z) output by the second encoder t |h t) and the KL divergence between the sample prior distribution, q(z t |h t ) is the sample posterior distribution of the sample latent variable, p(z t ) is the sample prior distribution of the sample latent variable, represents the second child loss, h t is the first sample target feature, p(h t |z t ) is the latent variable z of a given sample t When generating h t The probability distribution of .

[0209] It can be understood that by minimizing the first loss of the conditional variational autoencoder, a preset model can be obtained that can not only learn a structured latent space, but also achieve high-quality reconstruction and generate specific types of data under given conditions.

[0210] In some embodiments, when the first sample feature needs to be divided into multiple sample action blocks, it is only necessary to calculate the sub-loss for each sample sub-action block according to the above formula, and then add each sub-loss to obtain the first loss.

[0211] In some embodiments, different adjustment weights may be set for the first sub-loss and the second sub-loss to enable focused training of the preset model. For example, a first weight (e.g., 0.4) for adjusting the first sub-loss and a second weight (e.g., 0.6) for adjusting the second sub-loss may be obtained. The first sub-loss may be adjusted based on the first weight to obtain a first target sub-loss, and the second sub-loss may be adjusted based on the second weight to obtain a second target sub-loss. The first loss of the preset model may be determined based on the sum of the first target sub-loss and the second target sub-loss.

[0212] By defining the sample prior distribution and minimizing the KL divergence loss (first sub-loss), the preset model can learn a structured latent space in which similar samples are close to each other in the latent space, which is conducive to data clustering and dimensionality reduction; and by introducing the reconstruction loss (second sub-loss), the accuracy of data reconstruction by the preset model can be improved.

[0213] Please refer to Figure 3 , combined with Figure 3, an overall embodiment of the robot imitation learning method in the present application is introduced. Exemplarily, the demonstration data can be obtained by collecting the original data demonstrated by the operator and preprocessing the data, that is, standardizing and enhancing the original data. Furthermore, the first encoding of the target model can be used to extract the high-dimensional feature representation of the demonstration data to obtain the first target feature. Afterwards, the first target feature is divided into blocks according to the conditional variational autoencoder in the target model to generate the representation of the first target feature in the latent space, that is, the second target feature. Furthermore, the first decoder of the target model can be used to predict the joint position parameters of the next step from the second target feature in the latent space, and generate a motion trajectory based on the joint position parameters to control the robot to imitate.

[0214] The embodiment of the present application obtains demonstration data in chronological order and inputs the demonstration data into a first encoder of a preset target model for extracting high-dimensional features to obtain a first target feature; obtains action indication information corresponding to the demonstration data, decomposes the first target feature into multiple action blocks according to the action indication information through a conditional variational autoencoder, and reconstructs the multiple action blocks in sequence according to the action indication information to obtain corresponding multiple first sub-features, splices the multiple first sub-features in chronological order to obtain a second target feature corresponding to the first target feature; predicts the second target feature through a first decoder of the target model to obtain joint position parameters of the robot's joint at the next moment; and controls the robot to imitate according to the joint position parameters. In this way, after extracting the first target feature, the conditional variational autoencoder can be used to decompose the high-dimensional features into multiple action blocks according to the action indication information, so that the conditional variational autoencoder can focus on sub-features related to a specific action, enhance the interpretability of the model, and predict the joint position parameters of the reconstructed second target feature at the next moment through the first decoder to improve the coherence and timing of the imitation action, avoid error accumulation, and thus improve the accuracy of the robot's imitation learning.

[0215] See also Figure 4 The present application also provides a robot imitation learning device that can implement the above-mentioned robot imitation learning method. The robot imitation learning device includes:

[0216] An extraction module 41 is configured to obtain demonstration data in chronological order and input the demonstration data into a first encoder of a preset target model to extract high-dimensional features to obtain first target features;

[0217] The reconstruction module 42 is configured to obtain action indication information corresponding to the demonstration data, decompose the first target feature into multiple action blocks according to the action indication information through a conditional variational autoencoder, sequentially reconstruct features of the multiple action blocks according to the action indication information to obtain multiple corresponding first sub-features, and concatenate the multiple first sub-features in chronological order to obtain a second target feature corresponding to the first target feature;

[0218] A prediction module 43 is configured to predict the second target feature by using the first decoder of the target model to obtain joint position parameters of the robot's joints at the next moment;

[0219] The imitation module 44 is used to control the robot to imitate according to the joint position parameters.

[0220] The specific implementation of the robot imitation learning device is basically the same as the specific embodiment of the robot imitation learning method described above, and will not be repeated here. Under the premise of meeting the requirements of the embodiment of this application, the robot imitation learning device can also be provided with other functional modules to implement the robot imitation learning method in the above embodiment.

[0221] The present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-described robot imitation learning method when executing the computer program. The computer device can be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0222] See also Figure 5 , Figure 5 The hardware structure of a computer device according to another embodiment is shown. The computer device includes:

[0223] The processor 51 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0224] The memory 52 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 52 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program codes are stored in the memory 52 and are called by the processor 51 to execute the robot imitation learning method of the embodiments of this application.

[0225] Input / output interface 53, used to implement information input and output;

[0226] Communication interface 54, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, Wi-Fi, Bluetooth, etc.);

[0227] bus 55 , which transmits information between the various components of the device (e.g., processor 51 , memory 52 , input / output interface 53 , and communication interface 54 );

[0228] The processor 51 , the memory 52 , the input / output interface 53 and the communication interface 54 are connected to each other in communication within the device via a bus 55 .

[0229] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned robot imitation learning method.

[0230] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0231] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0232] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0233] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0234] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0235] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0236] It should be understood that in this application, "at least one (item)" and "several" refer to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0237] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the above units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0238] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0239] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0240] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0241] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A robot imitation learning method, characterized in that: The method comprises: Acquire demonstration data in chronological order, and input the demonstration data into a first encoder of a preset target model to extract high-dimensional features to obtain a first target feature; Obtaining action indication information corresponding to the demonstration data, decomposing the first target feature into a plurality of action blocks according to the action indication information using a conditional variational autoencoder, sequentially reconstructing features of the plurality of action blocks according to the action indication information to obtain a plurality of corresponding first sub-features, and splicing the plurality of first sub-features in chronological order to obtain a second target feature corresponding to the first target feature; Predicting the second target feature by the first decoder of the target model to obtain joint position parameters of the robot's joints at the next moment; The robot is controlled to perform imitation according to the joint position parameters.

2. The robot imitation learning method according to claim 1, characterized in that: The step of sequentially reconstructing features of the plurality of action blocks according to the action indication information to obtain corresponding plurality of first sub-features includes: For each action block, obtaining, by a second encoder of a conditional variational autoencoder, a posterior distribution corresponding to the action block based on the action indication information and the action block; Sampling the posterior distribution to obtain corresponding latent variables; The second decoder of the conditional variational autoencoder is used to reconstruct the features of the multiple action blocks in a high-dimensional space in chronological order based on the latent variables and the action indication information to obtain corresponding multiple first sub-features.

3. The robot imitation learning method according to claim 2, characterized in that: The sampling of the posterior distribution to obtain the corresponding latent variables includes: Obtaining a preset prior distribution, and determining a noise vector from the prior distribution; Obtaining a mean and a variance of the posterior distribution, and determining a first product by the variance and the noise vector; A corresponding latent variable is obtained according to the sum of the mean and the first product.

4. The robot imitation learning method according to claim 1, characterized in that: After controlling the robot to imitate according to the joint position parameters, the method further comprises: Acquiring a perspective image of the robot; The demonstration data is updated based on the perspective image and the joint position parameters to obtain new demonstration data.

5. The robot imitation learning method according to claim 1, characterized in that: The target model is trained in the following way: Acquire sample demonstration data in chronological order, and input the sample demonstration data into a first encoder of a preset model to extract high-dimensional features to obtain a first sample target feature; Obtaining sample action indication information corresponding to the sample demonstration data, decomposing the first sample target feature into a plurality of sample action blocks according to the sample action indication information using a conditional variational autoencoder, sequentially reconstructing features of the plurality of sample action blocks according to the sample action indication information to obtain a plurality of corresponding first sample sub-features, and splicing the plurality of first sample sub-features in chronological order to obtain a second sample target feature corresponding to the first sample target feature; Determining a first loss of the preset model based on the first sample target feature and the second sample target feature; Predicting the second sample target feature by the first decoder of the preset model to obtain sample joint position parameters of the robot's joint parts at the next moment; Obtaining label joint position parameters of the robot's joints that need to be adjusted at a next moment, and determining a second loss of the preset model based on a difference between the sample joint position parameters and the label joint position parameters; Constructing a target loss based on the first loss and the second loss; The preset model is trained based on the target loss until a preset model convergence condition is reached to obtain a target model.

6. The robot imitation learning method according to claim 5, characterized in that: The step of sequentially reconstructing features of the plurality of sample action blocks according to the sample action indication information to obtain corresponding plurality of first sample sub-features includes: For each sample action block, obtaining, by a second encoder of the conditional variational autoencoder, a sample posterior distribution corresponding to the sample action block based on the sample action indication information and the sample action block; Sampling the sample posterior distribution to obtain corresponding sample latent variables; The second decoder of the conditional variational autoencoder is used to reconstruct features of the multiple sample action blocks in a high-dimensional space in chronological order based on the sample latent variables and the sample action indication information to obtain corresponding multiple first sample sub-features.

7. The robot imitation learning method according to claim 5, characterized in that: The determining the first loss of the preset model based on the first sample target feature and the second sample target feature includes: Obtaining a sample prior distribution of the conditional variational autoencoder; Determining a first sub-loss according to a difference between the sample prior distribution and the sample posterior distribution; determining a second sub-loss according to a difference between the second sample target feature and the first sample target feature; A first loss of the preset model is determined based on the first sub-loss and the second sub-loss.

8. A robot imitation learning device, characterized in that: The device comprises: An extraction module is used to obtain demonstration data in chronological order and input the demonstration data into a first encoder of a preset target model to extract high-dimensional features to obtain a first target feature; a reconstruction module, configured to obtain action indication information corresponding to the demonstration data, decompose the first target feature into a plurality of action blocks according to the action indication information through a conditional variational autoencoder, sequentially reconstruct features of the plurality of action blocks according to the action indication information to obtain a plurality of corresponding first sub-features, and concatenate the plurality of first sub-features in chronological order to obtain a second target feature corresponding to the first target feature; a prediction module, configured to predict the second target feature by using the first decoder of the target model to obtain joint position parameters of the robot's joints at a next moment; The imitation module is used to control the robot to imitate according to the joint position parameters.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the robot imitation learning method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the robot imitation learning method according to any one of claims 1 to 7 is implemented.

Images