A latent action-based robot arm control method

Abstract

The application discloses a mechanical arm control method based on potential action, wherein the method comprises the following steps: acquiring video data and performing preprocessing; constructing a potential action extraction module and a forward model; training the potential action extraction module and the forward model by using the preprocessed video data; constructing a potential strategy model, and training the potential strategy model in combination with the trained forward model; training a specific action model based on the potential strategy model; given a task description, acquiring a current observation image of the mechanical arm, inputting the trained potential strategy model, and obtaining a potential action vector; inputting the task description, the current observation image and the potential action vector into the trained specific action model, and obtaining a specific action to be executed by the mechanical arm. By introducing the potential strategy model, the application utilizes the commonness between tasks to make the mechanical arm quickly adapt to new tasks and dynamic environment, thereby improving the efficiency and success rate of task execution.

Description

Technical Field

[0001] This invention relates to the field of robotic arm operation technology, and in particular to a robotic arm control method based on potential motion. Background Technology

[0002] Existing robotic arm manipulation technologies typically rely on precise programming or complex sensor systems to execute tasks. While these methods can accomplish specific operational tasks, they often lack sufficient flexibility and adaptability when facing changing environments and tasks. For example, in industrial production lines, robotic arms usually require precise programming for specific tasks. Once task requirements or the environment change, the robotic arm may not be able to adapt quickly, leading to decreased efficiency or even task failure. Furthermore, traditional robotic arm learning methods require large amounts of high-quality labeled data and complex model training processes, which not only increases costs but also limits the widespread deployment of robotic arms in practical applications.

[0003] In recent years, with the development of deep learning technology, imitation learning has become an effective method for controlling robotic arms. Imitation learning is a technique that trains an intelligent agent by observing and imitating the behavior of experts. Its core idea is to use expert demonstration data to allow the agent to learn how to take appropriate actions in different states to complete specific tasks. By analyzing the action sequences and decision-making processes of humans performing tasks, robotic arms can perform complex tasks without explicit programming instructions. This method reduces the reliance on precise programming to some extent, allowing robotic arms to adapt to complex tasks more naturally.

[0004] However, existing imitation learning methods still have the following shortcomings:

[0005] 1) Reliance on large amounts of high-quality expert demonstration data: Existing imitation learning methods typically require a large amount of high-quality expert demonstration data to train the model. This data not only needs to be accurately labeled, but also needs to cover a variety of possible task scenarios and operational steps. However, obtaining such high-quality data is costly and often difficult to achieve in practical applications. For example, in some highly specialized or extreme environments, the collection of expert demonstration data may be constrained by operational risks, technical limitations, or insufficient resources. Furthermore, even if data can be obtained, its diversity and coverage may be insufficient, leading to limitations in the model's performance in real-world applications.

[0006] 2) Insufficient generalization ability: Existing imitation learning methods often exhibit poor generalization ability when faced with new tasks or environments. Since model training relies on demonstration data in specific scenarios, the model may fail to adapt effectively when task requirements or environmental conditions change. For example, in industrial settings, robotic arms may need to handle objects of different shapes, sizes, or materials, and existing methods often require retraining or adjusting the model to cope with these changes, leading to inefficiency. This lack of generalization ability limits the application potential of robotic arms in diverse tasks and dynamic environments. Summary of the Invention

[0007] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a robotic arm control method, device and medium based on potential actions.

[0008] The first technical solution adopted in this invention is:

[0009] A robotic arm manipulation method based on potential motion includes the following steps:

[0010] Select video clips of humans performing tasks from a first-person perspective and combine them with videos of robotic arm operations for data preprocessing;

[0011] A latent action extraction module and a forward model are constructed. The latent action extraction module is used to extract the latent actions from the current observation image to the future observation image based on the current observation image and the future observation image. The forward model is used to generate the future observation image based on the current observation image and the extracted latent actions.

[0012] The latent action extraction module and the forward model are trained using preprocessed video data;

[0013] A latent policy model is constructed and trained by combining it with a trained forward model; the latent policy model is used to predict potential actions from the currently observed image and the specified task.

[0014] Based on the latent policy model, a specific action model is trained; the specific action model is used to take the latent action output by the latent policy model as the sub-task target, and combine it with the current observation to output the specific robotic arm action.

[0015] Given a task description, obtain the current observation image of the robotic arm, input it into the trained latent policy model to obtain the latent action vector; input the task description, the current observation image and the latent action vector into the trained specific action model to obtain the specific action to be executed by the robotic arm.

[0016] Furthermore, the latent action extraction module includes multiple attention layers, each consisting of a self-attention layer and a feedforward layer, used to extract high-level feature representations from the input data; after the attention layers, a codebook containing multiple learnable vectors is designed, each vector representing a unique latent action, used to map the extracted features to a specific action space.

[0017] In the first attention layer, the input is the concatenation of the feature vectors obtained by encoding the current and future observation images through the image encoder. The input of each subsequent attention layer is the output of the previous attention layer. Through layer-by-layer propagation and optimization, more abstract and task-related potential action vectors are gradually extracted. Then, the dimensions of the feature vectors and potential action vectors are aligned by a multilayer perceptron. Finally, the potential action vector that is closest to the feature vector in the codebook is selected as the potential action.

[0018] Furthermore, the image encoder is implemented using a pre-trained Vision Transformer (ViT);

[0019] The calculation process of the potential action extraction module is as follows:

[0020]

[0021] F i+1 =FFN(SelfAtten(F ii )+F i )

[0022] z e =MLP(F 12 )

[0023]

[0024] In the formula, I t and I t+1 These represent the current and future observation images, respectively, with ViT representing the image encoder. This indicates the concatenation operation, F represents the feature vector output by the network, FFN represents the feedforward layer, SelfAttenuation represents the self-attention layer, MLP represents the multilayer perceptron, and z e The feature vector is aligned with the latent action vector, where C represents the codebook, and c j Let ||·||2 represent the learnable latent action vectors in the codebook, ||·||2 represent the Euclidean norm, and z represent the latent action vectors extracted by the latent action extraction module.

[0025] Furthermore, the forward model includes multiple attention layers and a downsampling layer. Each attention layer consists of a cross attention layer, a self attention layer, and a feedforward layer, used to predict future observation images from the current observation image and potential actions.

[0026] In the first cross-attention layer, the input consists of the feature vector obtained from the current observed image after passing through the image encoder and the potential action vector extracted from the potential action extraction module. After passing through the cross-attention layer, a fused feature vector is generated and then input into the self-attention layer. After passing through the feedforward layer, the feature vector and the potential action vector are input into the next cross-attention layer, and the above process is repeated. After passing through multiple attention layers, the obtained feature vector is finally downsampled and recombined to obtain the predicted future observed image.

[0027] Furthermore, the computation process of the forward model is as follows:

[0028] F0 = ViT(I) t )

[0029] F i+1 =FFn(SelfAtten(CrossAtten(F i ,z e +sg[zz e ])+F i )

[0030]

[0031] In the formula, I t The input represents the currently observed image, ViT represents the image encoder, F represents the feature vector output by the network, FFN represents the feedforward layer, SelfAtten represents the self-attention layer, CrossAtten represents the cross-attention layer, sg[·] represents the gradient truncation operation, and DownSampled represents the downsampling layer. This represents the future observation image predicted by the forward model.

[0032] Furthermore, the step of training the latent action extraction module and the forward model using the preprocessed video data includes:

[0033] Extract the current observation image and future observation image from the preprocessed video data and input them as training data into the potential action extraction module;

[0034] The latent action extraction module extracts latent action vectors from the current observation image and the future observation image. These vectors contain semantic information about the action that transitions from the current observation to the future observation.

[0035] The extracted latent action vectors are used together with the current observed image as input to the forward model. The forward model combines the detailed features of the current observed image with the high-level semantic information of the latent actions to predict future observed images.

[0036] The loss function is calculated based on the predicted future observation images and the actual future observation images, and the parameters of the potential action extraction module and the forward model are optimized.

[0037] The formula for calculating the loss function is as follows:

[0038]

[0039] L total =L recon +L embedding +αL commit

[0040] In the formula, L recon Indicates the image reconstruction loss, MSE represents the mean squared error, and I represents the mean squared error. t+1 Represents a true future observation image. L represents the future observation image predicted by the forward model. embedding L represents the encoding loss used for training the codebook, where α is the corresponding weight coefficient. commit This indicates that the output should be closer to the codebook to focus on error. total It is the final loss function.

[0041] Furthermore, the latent policy model includes multiple attention layers, each consisting of a cross attention layer, a self attention layer, and a feedforward layer, for predicting potential actions from the currently observed image and the specified task;

[0042] In the first cross-attention layer, the input consists of a vector obtained by the task description through a text encoder and a feature vector obtained by the current observed image through an image encoder. After passing through the cross-attention layer, a fused feature vector is generated and then input into the self-attention layer. After passing through the feedforward layer, the feature vector obtained and the latent action vector are input into the next cross-attention layer, and the above process is repeated. After passing through multiple attention layers, the latent action is output.

[0043] The latent actions output by the latent policy model and the current observed image are used as inputs to the trained forward model. The forward model outputs the predicted future observed image. The loss function is calculated based on the predicted future observed image and the actual future observed image, and the network parameters of the latent policy model are trained.

[0044] Furthermore, the text is encoded as a CLIP model, and its parameters are frozen during training;

[0045] The computation process of the potential strategy model is as follows:

[0046] F0 = CLIP(T)

[0047] F i+1 =FFN(SelfAtten(CrossAtten(F i ,ViT(I t )+F i )

[0048] z e =MLP(F 12 )

[0049]

[0050] In the formula, T represents the input task text description, CLIP represents the text encoder, F represents the feature vector output by the network, FFN represents the feedforward layer, SelfAttenuation represents the self-attention layer, MLP represents the multilayer perceptron, and z e The feature vector is aligned with the latent action vector, where C represents the codebook, and c j Let ||·||2 represent the latent action vector in the learned codebook, ||·||2 represent the Euclidean norm, and z represent the latent action vector obtained from the output of the latent policy model.

[0051] The latent policy model obtains latent action vectors, which are then used together with the current observed image as input to the forward model to obtain the predicted future observed image.

[0052] F′0=ViT(I t )

[0053] F′ i+1 =FFN(SelfAtten(CrossAtten(F′ i ,z e +sg[zz e ])+F′ i )

[0054]

[0055] In the formula, I t The input represents the currently observed image, ViT represents the image encoder, F′ represents the feature vector output by the network, FFN represents the feedforward layer, SelfAtten represents the self-attention layer, CrossAtten represents the cross-attention layer, sg[·] represents the gradient cutoff operation, and z e The feature vector is aligned with the latent action vector, z represents the latent action vector output by the latent policy model, and DownSampled represents the downsampling layer. This represents the future observation image predicted by the forward model;

[0056] The calculation process of the loss function is as follows:

[0057]

[0058] L total =L recon +L commit

[0059] In the formula, L recon Indicates the image reconstruction loss, MSE represents the mean squared error, and I represents the mean squared error. t+1 Represents a true future observation image. L represents the future observation image predicted by the forward model. commit This indicates that the output should be closer to the codebook to focus on error. total It is the final loss function.

[0060] Furthermore, the specific action model includes multiple attention layers, each consisting of a self-attention layer and a feedforward layer. The potential action output by the latent policy model is used as the sub-task target, and the specific robotic arm action is output in combination with the current observation.

[0061] In the first attention layer, the input is a concatenation of the feature vector obtained by encoding the current observed image through the image encoder, the vector obtained by encoding the task description through the text encoder, and the potential action vector output by the potential policy model; the input of each subsequent attention layer is the output of the previous attention layer; finally, the specific action is predicted by the multilayer perceptron.

[0062] The loss function is calculated based on the predicted specific action and the actual specific action, and the network parameters of the specific action model are trained.

[0063] Furthermore, the image encoder is implemented using a pre-trained Vision Transformer, and the text encoder is implemented using a pre-trained CLIP model, with the parameters of both frozen during training.

[0064] The calculation process of the specific action model is as follows:

[0065]

[0066] F i+1 =FFN(SelfAtten(F ii )+F i )

[0067]

[0068] In the formula, It This represents the currently observed input image, where ViT stands for image encoder. The concatenation operation is represented by T, the input task text description is represented by CLIP, the text encoder is represented by F, the feature vector output by the network is represented by FFN, the self-attention layer is represented by SelfAttention, and MLP is represented by Multilayer Perceptron. This represents the specific motion (x, y, z, w) of the robotic arm predicted by the specific motion model. x ,w y ,w z ,g), where x,y,z represent the displacements of the robotic arm's end effector along the x,y,z axes, and w x ,w y ,w z This indicates the rotation angle of the robotic arm's end effector on the x, y, and z axes, while g represents the opening and closing of the gripper.

[0069] The calculation process of the loss function is as follows:

[0070]

[0071] In the formula, L recon Let represent the action prediction loss, MSE represent the mean squared error, and 'a' represent the actual action. This indicates the specific action predicted by the specific action model.

[0072] The second technical solution adopted in this invention is:

[0073] An electronic device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a robotic arm manipulation method based on potential actions as described above.

[0074] The fourth technical solution adopted in this invention is:

[0075] A computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a potential motion-based robotic arm manipulation method as described above.

[0076] The fifth technical solution adopted in this invention is:

[0077] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method for manipulating a robotic arm based on potential actions.

[0078] The beneficial effects of this invention are as follows: By introducing a latent policy model and utilizing the commonalities among potential motion capture tasks, this invention enables the robotic arm to quickly adapt to new tasks and dynamic environments, thereby improving the efficiency and success rate of task execution. Furthermore, by combining training with large-scale online video data, this invention reduces reliance on high-quality expert demonstration data, lowers the cost and difficulty of data acquisition, and simultaneously broadens the diversity of data sources. Attached Figure Description

[0079] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0080] Figure 1 This is a flowchart of a robotic arm control method based on potential actions in an embodiment of the present invention;

[0081] Figure 2 This is a schematic diagram of the training framework disclosed in the embodiments of the present invention;

[0082] Figure 3 This is a schematic diagram of the potential action extraction module and the forward model structure in an embodiment of the present invention;

[0083] Figure 4 This is a schematic diagram of the potential strategy model and specific action model structure in an embodiment of the present invention. Detailed Implementation

[0084] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0085] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0086] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0087] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0088] To address the aforementioned problems, this invention proposes a robotic arm control scheme based on latent actions, primarily solving the issues of existing robotic arm control algorithms' reliance on large amounts of high-quality expert demonstration data and insufficient generalization. Existing technologies typically rely on precise expert demonstration data, which is not only costly and difficult to acquire but also challenging to collect in highly specialized or extreme environments. Furthermore, although large-scale online video data is readily available, its significant differences from robotic arm operation scenarios prevent direct transfer to imitation learning, leading to challenges of data scarcity and insufficient generalization in practical applications. This invention, by introducing the concept of latent actions and combining large-scale online video data with robotic arm expert demonstration data, significantly reduces the reliance on high-quality expert demonstration data while simultaneously improving the robotic arm's adaptability to new tasks and dynamic environments.

[0089] Example 1

[0090] like Figure 1 As shown, this embodiment provides a robotic arm control method based on potential actions, including the following steps:

[0091] S1. Select relevant video clips of human first-person perspective task execution from large-scale network data, and perform data preprocessing by combining them with robotic arm operation videos.

[0092] In some embodiments, the large-scale network data is the publicly available video dataset Something-Something, which specifically collects videos of humans performing various tasks from a first-person perspective, accompanied by text descriptions. The robotic arm operation videos are selected from expert demonstration data collected in the Calvin simulation environment, which includes video of task execution and corresponding text descriptions, as well as recording the specific movements of the robotic arm. Videos with clear visuals are selected from the video dataset, and the images are uniformly cropped to 224×224 pixels.

[0093] S2. Construct a latent action extraction module and a forward model; wherein, the latent action extraction module is used to extract the latent actions from the current observation image to the future observation image based on the current observation image and the future observation image, and the forward model is used to generate the future observation image based on the current observation image and the extracted latent actions.

[0094] In some embodiments, the latent action extraction module is constructed as follows:

[0095] See Figure 3 In this embodiment, the potential action extraction module is as follows: Figure 3 As shown in the dashed box in the lower right corner, the core structure of this module includes 12 attention layers. Each attention layer consists of a self-attention layer and a feedforward layer, used to extract high-level feature representations from the input data. Following the attention layers, a codebook containing 128 learnable vectors is designed, each representing a unique latent action, used to map the extracted features to a specific action space. In the first attention layer, the input is a concatenation of feature vectors obtained from the current and future observation images after encoding by an image encoder. In this embodiment, a pre-trained Vision Transformer (ViT) is used as the image encoder, and its parameters are frozen during training. The input of each subsequent attention layer is the output of the previous attention layer. Through layer-by-layer propagation and optimization, more abstract and task-relevant latent action features are gradually extracted. Then, a multilayer perceptron is used to align the dimensions of the feature vectors and the latent action vectors. Finally, the model selects the latent action vector in the codebook that is closest to the feature vector as the latent action output.

[0096] In summary, the computation process of the potential action extraction module is as follows:

[0097]

[0098] F i+1 =FFN(SelfAtten(F i )+F i )

[0099] z e=MLP(F 12 )

[0100]

[0101] In the formula, I t and I t+1 These represent the current and future observation images, respectively, with ViT representing the image encoder. This indicates the concatenation operation, F represents the feature vector output by the network, FFN represents the feedforward layer, SelfAttenuation represents the self-attention layer, MLP represents the multilayer perceptron, and z e The feature vector is aligned with the latent action vector, where C represents the codebook, and c j Let ||·||2 represent the learnable latent action vectors in the codebook, ||·||2 represent the Euclidean norm, and z represent the latent action vectors extracted by the latent action extraction module.

[0102] In some embodiments, the forward model is constructed as follows:

[0103] See Figure 3 The forward model in this embodiment is as follows: Figure 3 As shown in the dashed box above, the core structure of the forward model consists of 24 attention layers and one downsampling layer. Each attention layer comprises a cross-attention layer, a self-attention layer, and a feedforward layer, used to predict future observation images from the current observation image and potential actions. In the first cross-attention layer, the input is the feature vector obtained from the current observation image after passing through the image encoder and the potential action vector extracted from the potential action extraction module. After passing through the cross-attention layer, a fused feature vector is generated and then input into the self-attention layer. This is then transformed by the feedforward layer, and the resulting feature vector, along with the potential action vector, is input into the next cross-attention layer, repeating the above process. The feature vector obtained after passing through 24 attention layers is finally downsampled by a factor of 5 and reconstructed into a 224×224 image, which is output as the predicted future observation image.

[0104] In summary, the computation process of the forward model is as follows:

[0105] F0 = ViT(I) t )

[0106] F i+1 =FFN(SelfAtten(CrossAtten(F i ,z e +sg[zz e ])+F i )

[0107]

[0108] In the formula, I t The input represents the currently observed image, ViT represents the image encoder, F represents the feature vector output by the network, FFN represents the feedforward layer, SelfAtten represents the self-attention layer, CrossAtten represents the cross-attention layer, sg[·] represents the gradient truncation operation, and DownSampled represents the downsampling layer. This represents the future observation image predicted by the forward model.

[0109] S3. Using the preprocessed video data, train the latent action extraction module and the forward model to obtain a certain number of latent action vectors with high-level semantic information.

[0110] See Figure 2 The action extraction module and the forward model training process in this embodiment are as follows: Figure 2 As shown in the left half, firstly, the current observation image and future observation image are extracted from the preprocessed video clip and input as training data into the action extraction module. The action extraction module extracts latent action vectors from the current and future observation images through a multi-layer attention mechanism and codebook mapping. These vectors contain semantic information about the action transitioning from the current observation to the future observation. Subsequently, the extracted latent action vectors, along with the current observation image, are used as input to the forward model. The forward model, through multi-layer cross-attention and self-attention mechanisms, combines the detailed features of the current observation image with the high-level semantic information of the latent actions to predict the future observation image. Then, based on the predicted and actual future observation images, the loss function and gradient are calculated, and the Adam backpropagation algorithm is used for training to train the network parameters of the action extraction module and the forward model. The formula for calculating the loss function is as follows:

[0111]

[0112]

[0113] L total =L recon +L embedding +αL commit

[0114] In the formula, L recon Indicates the image reconstruction loss, MSE represents the mean squared error, and I represents the mean squared error. t+1 Represents a true future observation image. L represents the future observation image predicted by the forward model. embedding Let L represent the encoding loss used for training the codebook, and α be its corresponding weight coefficient, which is set to 0.25 in this embodiment. commit This indicates that the output should be closer to the codebook to focus on error. totalIt is the final loss function.

[0115] S4. Construct a latent policy model and train the latent policy model by combining it with the trained forward model; the latent policy model is used to predict potential actions from the currently observed image and the specified task.

[0116] Specifically, a latent policy model is trained using preprocessed video data. This model predicts potential actions based on current observations, representing a high-level task decomposition and action guidance. (See also...) Figure 4 The potential strategy model in this embodiment is as follows: Figure 4 As shown in the dashed box below, the core structure of the latent policy model consists of 12 attention layers. Each attention layer comprises a cross-attention layer, a self-attention layer, and a feedforward layer, used to predict latent actions from the currently observed image and the specified task. In the first cross-attention layer, the input is a vector obtained by a text encoder for the task description and a feature vector obtained by an image encoder for the currently observed image. In this embodiment, the text encoding is a pre-trained CLIP model, and its parameters are frozen during training. After passing through the cross-attention layer, a fused feature vector is generated and input into the self-attention layer. Then, the feature vector obtained by the feedforward layer transformation and the latent action vector are input into the next cross-attention layer, repeating the above process. After 12 attention layers, the latent action is output. Subsequently, the latent action output by the model is compared with the current one. Then, the loss function and gradient are calculated based on the predicted future observed image and the actual future observed image, and the Adam backpropagation algorithm is used for training to train the network parameters of the latent policy model.

[0117] In summary, the computation process of the latent policy model is as follows:

[0118] F0 = CLIP(T)

[0119] F i+1 =FFN(SelfAtten(CrossAtten(F i ,ViT(I t )+F i )

[0120] z e =MLP(F 12 )

[0121]

[0122] In the formula, T represents the input task text description, CLIP represents the text encoder, F represents the feature vector output by the network, FFN represents the feedforward layer, SelfAttenuation represents the self-attention layer, MLP represents the multilayer perceptron, and z eThe feature vector is aligned with the latent action vector, where C represents the codebook, and c j Let ||·||2 represent the latent action vector in the codebook learned in step S3, ||·||2 represent the Euclidean norm, and z represent the latent action vector obtained from the output of the latent policy model.

[0123] The latent policy model obtains latent action vectors, which are then used together with the current observed image as input to the forward model to obtain the predicted future observed image.

[0124] F′0=ViT(I t )

[0125] F′ i+1 =FFN(SelfAtten(CrossAtten(F′ i ,z e +sg[zz e ])+F′ i )

[0126]

[0127] In the formula, I t The input represents the currently observed image, ViT represents the image encoder, F′ represents the feature vector output by the network, FFN represents the feedforward layer, SelfAtten represents the self-attention layer, CrossAtten represents the cross-attention layer, sg[·] represents the gradient cutoff operation, and z e The feature vector is aligned with the latent action vector, z represents the latent action vector output by the latent policy model, and DownSampled represents the downsampling layer. This represents the future observation image predicted by the forward model.

[0128] The calculation process of the loss function is as follows:

[0129]

[0130] L total =L recon +L commit

[0131] In the formula, L recon Indicates the image reconstruction loss, MSE represents the mean squared error, and I represents the mean squared error. t+1 Represents a true future observation image. L represents the future observation image predicted by the forward model. commit This indicates that the output should be closer to the codebook to focus on error. total It is the final loss function.

[0132] S5. Based on the latent policy model, train a specific action model; the specific action model is used to take the latent action output by the latent policy model as the sub-task target, and combine it with the current observation to output the specific robotic arm action.

[0133] See Figure 4 The specific action model in this embodiment is as follows: Figure 4 As shown in the dashed box above, the core structure of the specific action model includes 12 attention layers. Each attention layer consists of a self-attention layer and a feedforward layer. It uses the latent action output by the latent policy model as the sub-task objective, and combines it with the current observation to output the specific robotic arm action. In the first attention layer, the input is a concatenation of the feature vector obtained by encoding the current observed image through an image encoder, the vector obtained by encoding the task description through a text encoder, and the latent action vector output by the latent policy model. In this embodiment, a pre-trained Vision Transformer (ViT) is used as the image encoder, and a pre-trained CLIP model is used as the text encoder, with their parameters frozen during training. The input of each subsequent attention layer is the output of the previous attention layer. The predicted specific action is then obtained through a multilayer perceptron. Subsequently, the loss function and gradient are calculated based on the predicted and actual specific actions, and the Adam backpropagation algorithm is used for training to train the network parameters of the specific action model.

[0134] In summary, the specific calculation process of the action model is as follows:

[0135]

[0136] F i+1 =FFN(SelfAtten(F i )+F i )

[0137]

[0138] In the formula, I t This represents the currently observed input image, where ViT stands for image encoder. The concatenation operation is represented by T, the input task text description is represented by CLIP, the text encoder is represented by F, the feature vector output by the network is represented by FFN, the self-attention layer is represented by SelftAttenuation, and MLP is represented by a multilayer perceptron. This represents the specific motion (x, y, z, w) of the robotic arm predicted by the specific motion model. x ,w y ,w z ,g), where x,y,z represent the displacements of the robotic arm's end effector along the x,y,z axes, and w x ,w y,w z The x, y, and z axes represent the rotation angles of the robotic arm's end effector, while g represents the opening and closing of the gripper.

[0139] The calculation process of the loss function is as follows:

[0140]

[0141] In the formula, L recon Let represent the action prediction loss, MSE represent the mean squared error, and 'a' represent the actual action. This indicates the specific action predicted by the specific action model.

[0142] S6. Given a task description, obtain the current observation image of the robotic arm, input the trained latent policy model to obtain the latent action vector; input the task description, the current observation image and the latent action vector into the trained specific action model to obtain the specific action to be executed by the robotic arm.

[0143] For example, given a task description, the current observation image of the robotic arm is acquired, and a trained latent policy model is input to obtain a latent action vector. Then, these three vectors are input into a trained specific action model to obtain a specific action, which is then executed by the robotic arm. The next observation image is then acquired and used as the new current observation image, which is input into the latent policy model. This process is repeated until the task is completed or the maximum number of steps is reached.

[0144] In summary, the method of the present invention has at least the following advantages over the prior art:

[0145] 1) Reduce data dependence and increase diversity: By combining large-scale network video data for training, this invention reduces the dependence on high-quality expert demonstration data, reduces the cost and difficulty of data acquisition, and at the same time broadens the diversity of data sources.

[0146] 2) Introducing latent strategies to enhance adaptability: By introducing a latent strategy model, this invention utilizes the commonalities among potential motion capture tasks to enable the robotic arm to quickly adapt to new tasks and dynamic environments, thereby improving the efficiency and success rate of task execution.

[0147] Example 2

[0148] This invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to achieve the following: Figure 1 This illustrates a robotic arm manipulation method based on potential motion.

[0149] It is understood that the memory may include random access memory (RAM) or read-only memory. Optionally, the memory may include non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the stored data area may store data created according to the use of the server, etc.

[0150] A processor may include one or more processing cores. The processor connects to various parts of the server via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various server functions and process data. Optionally, the processor may be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor may integrate one or more of the following: Central Processing Unit (CPU) and Modem. The CPU primarily handles the operating system and applications; the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0151] Since this electronic device is an electronic device corresponding to a robotic arm control method based on potential actions in an embodiment of the present invention, and the principle of solving the problem by this electronic device is similar to that of the method, the implementation of this electronic device can refer to the implementation process of the above method embodiment, and repeated parts will not be described again.

[0152] Example 3

[0153] This invention also provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to achieve the following: Figure 1 This illustrates a robotic arm manipulation method based on potential motion.

[0154] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0155] Since the storage medium is the storage medium corresponding to a robotic arm control method based on potential actions in an embodiment of the present invention, and the principle of the storage medium in solving the problem is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0156] Example 4

[0157] In some possible implementations, various aspects of the methods of the embodiments of the present invention can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of a robotic arm manipulation method based on potential actions according to various exemplary embodiments of the present application described above. The executable computer program code or "code" for performing the various embodiments can be written in high-level programming languages ​​such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0158] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0159] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0160] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A robotic arm manipulation method based on latent motion, characterized in that, Includes the following steps: Select video clips of humans performing tasks from a first-person perspective and combine them with videos of robotic arm operations for data preprocessing; A latent action extraction module and a forward model are constructed. The latent action extraction module is used to extract the latent actions from the current observation image to the future observation image based on the current observation image and the future observation image. The forward model is used to generate the future observation image based on the current observation image and the extracted latent actions. The latent action extraction module and the forward model are trained using preprocessed video data; A latent policy model is constructed and trained by combining it with a trained forward model; the latent policy model is used to predict potential actions from the currently observed image and the specified task. Based on the latent policy model, a specific action model is trained; the specific action model is used to take the latent action output by the latent policy model as the sub-task target, and combine it with the current observation to output the specific robotic arm action. Given a task description, obtain the current observation image of the robotic arm, input it into the trained latent policy model to obtain the latent action vector; input the task description, the current observation image and the latent action vector into the trained specific action model to obtain the specific action to be executed by the robotic arm.

2. The robotic arm control method based on latent motion according to claim 1, characterized in that, The latent action extraction module includes multiple attention layers, each consisting of a self-attention layer and a feedforward layer, used to extract high-level feature representations from the input data; Following the attention layer, a codebook containing multiple learnable vectors was designed, each vector representing a unique latent action, to map the extracted features to a specific action space. In the first attention layer, the input is the concatenation of the feature vectors obtained by encoding the current and future observation images through the image encoder. The input of each subsequent attention layer is the output of the previous attention layer. Through layer-by-layer propagation and optimization, more abstract and task-related potential action vectors are gradually extracted. Then, the dimensions of the feature vectors and potential action vectors are aligned by a multilayer perceptron. Finally, the potential action vector that is closest to the feature vector in the codebook is selected as the potential action.

3. The robotic arm control method based on latent motion according to claim 2, characterized in that, The image encoder is implemented using a pre-trained Vision Transformer; The calculation process of the potential action extraction module is as follows: In the formula, and These represent the current observation image and the future observation image, respectively. Indicates an image encoder. This indicates a splicing operation. This represents the feature vector output by the network. Indicates feedforward layer, Indicates the self-attention layer. This represents a multilayer perceptron. This represents the feature vector aligned with the potential action vector. Represents the codebook. This represents the learnable potential action vectors in the codebook. Denotes the Euclidean norm. This represents the latent action vector extracted by the latent action extraction module; This represents the features of layer 12 of the latent action extraction module. This represents a function that searches for the minimum value.

4. The robotic arm control method based on latent motion according to claim 1, characterized in that, The forward model includes multiple attention layers and a downsampling layer. Each attention layer consists of a cross attention layer, a self attention layer and a feedforward layer, which are used to predict future observation images from the current observation image and potential actions. In the first cross-attention layer, the input consists of the feature vector obtained from the current observed image after passing through the image encoder and the potential action vector extracted from the potential action extraction module. After passing through the cross-attention layer, a fused feature vector is generated and then input into the self-attention layer. After passing through the feedforward layer, the feature vector and the potential action vector are input into the next cross-attention layer. After passing through multiple attention layers, the obtained feature vector is finally downsampled and recombined to obtain the predicted future observed image.

5. A robotic arm manipulation method based on potential actions according to claim 1 or 4, characterized in that, The computation process of the forward model is as follows: In the formula, This represents the currently observed image as input. Indicates an image encoder. This represents the feature vector output by the network. Indicates feedforward layer, Indicates the self-attention layer. Indicates a cross-attention layer. This indicates the gradient cutoff operation. Indicates the downsampling layer. This represents the future observation image predicted by the forward model; This represents the latent action vector extracted by the latent action extraction module. This represents the feature vector aligned with the potential action vector. This represents the features of layer 24 of the latent action extraction module.

6. The robotic arm control method based on latent motion according to claim 1, characterized in that, The step of training the latent action extraction module and the forward model using preprocessed video data includes: Extract the current observation image and future observation image from the preprocessed video data and input them as training data into the potential action extraction module; The latent action extraction module extracts latent action vectors from the current observation image and the future observation image. These vectors contain semantic information about the action that transitions from the current observation to the future observation. The extracted latent action vectors are used together with the current observed image as input to the forward model. The forward model combines the detailed features of the current observed image with the high-level semantic information of the latent actions to predict future observed images. The loss function is calculated based on the predicted future observation images and the actual future observation images, and the parameters of the potential action extraction module and the forward model are optimized. The formula for calculating the loss function is as follows: In the formula, Indicates image reconstruction loss. Indicates mean square error. Represents a true future observation image. This represents the future observation image predicted by the forward model. This represents the encoding loss used for training the codebook. Its corresponding weight coefficient, This indicates a focus on error correction to make the output closer to the codebook. It is the final loss function; This indicates the gradient truncation operation; This represents the latent action vector extracted by the latent action extraction module. The feature vector is aligned with the potential action vector.

7. The robotic arm control method based on latent motion according to claim 1, characterized in that, The latent policy model includes multiple attention layers, each consisting of a cross attention layer, a self attention layer, and a feedforward layer, used to predict potential actions from the currently observed image and the specified task. In the first cross-attention layer, the input consists of a vector obtained by the task description through a text encoder and a feature vector obtained by the current observed image through an image encoder. After passing through the cross-attention layer, a fused feature vector is generated and then input into the self-attention layer. After passing through the feedforward layer, the feature vector obtained and the latent action vector are input into the next cross-attention layer. After passing through multiple attention layers, the latent action is output. The latent actions output by the latent policy model and the current observed image are used as inputs to the trained forward model. The forward model outputs the predicted future observed image. The loss function is calculated based on the predicted future observed image and the actual future observed image, and the network parameters of the latent policy model are trained.

8. The robotic arm control method based on latent motion according to claim 7, characterized in that, The text is encoded as a CLIP model, and its parameters are frozen during training; The computation process of the potential strategy model is as follows: In the formula, This represents the text description of the input task. Indicates a text encoder. This represents the feature vector output by the network. Indicates feedforward layer, Indicates the self-attention layer. This represents a multilayer perceptron. This represents the feature vector aligned with the potential action vector. Represents the codebook. This represents the latent action vector in a well-learned codebook. Denotes the Euclidean norm. This represents the latent action vector obtained from the output of the latent policy model; This represents the features of layer 12 of the latent action extraction module. This represents a function that seeks the minimum value. The latent policy model obtains latent action vectors, which are then used together with the current observed image as input to the forward model to obtain the predicted future observed image. In the formula, This represents the currently observed image as input. Indicates an image encoder. This represents the feature vector output by the network. Indicates feedforward layer, Indicates the self-attention layer. Indicates a cross-attention layer. This indicates the gradient cutoff operation. This represents the feature vector aligned with the potential action vector. This represents the latent action vector obtained from the output of the latent policy model. Indicates the downsampling layer. This represents the future observation image predicted by the forward model; This represents the features of the 24th layer of the forward model; The calculation process of the loss function is as follows: In the formula, Indicates image reconstruction loss. Indicates mean square error. Represents a true future observation image. This represents the future observation image predicted by the forward model. This indicates a focus on error correction to make the output closer to the codebook. It is the final loss function.

9. The robotic arm manipulation method based on latent motion according to claim 1, characterized in that, The specific action model includes multiple attention layers, each of which consists of a self-attention layer and a feedforward layer; In the first attention layer, the input is a concatenation of the feature vector obtained by encoding the current observed image through the image encoder, the vector obtained by encoding the task description through the text encoder, and the potential action vector output by the potential policy model; the input of each subsequent attention layer is the output of the previous attention layer; finally, the specific action is predicted by the multilayer perceptron. The loss function is calculated based on the predicted specific action and the actual specific action, and the network parameters of the specific action model are trained.

10. A robotic arm control method based on latent motion according to claim 9, characterized in that, The image encoder is implemented using a pre-trained Vision Transformer, and the text encoder is implemented using a pre-trained CLIP model, with the parameters of both frozen during training. The calculation process of the specific action model is as follows: In the formula, This represents the currently observed image as input. Indicates an image encoder. This indicates a splicing operation. This represents the text description of the input task. Indicates a text encoder. This represents the feature vector output by the network. Indicates feedforward layer, Indicates the self-attention layer. This represents a multilayer perceptron. This represents the specific movements of the robotic arm predicted by the specific motion model. ,in Indicates the end effector of the robotic arm at Displacement on three axes Indicates the end effector of the robotic arm at Rotation angles on three axes Indicates the opening and closing of the grippers; This represents the features of the 12th layer of the latent action extraction module; The calculation process of the loss function is as follows: In the formula, Indicates the loss from action prediction. Indicates mean square error. Indicates real, specific actions. This represents the specific action predicted by the specific action model.

Images