A method of controlling a robot and a method of training a vision-action model

Abstract

The specification provides a method for controlling a robot and a method for training a vision-action model, in the method for controlling a robot, a vision-action model is executed, the vision-action model comprises a vision prediction module and an action prediction module. First, the historical video data representing the video shot by the robot, the historical action data representing the action performed by the robot, and the first task data identifying the task to be performed by the robot are obtained. Then, according to the historical video data, the historical action data and the first task data, the first video data is predicted by using the vision prediction module; on the basis of obtaining the first video data, the first action data is predicted by using the action prediction module according to the first video data and the historical video data; the robot is controlled to perform the first action according to the first action data.

Description

Technical Field

[0001] This specification relates to the field of robot control technology, and more particularly to a method for controlling a robot and a method for training a vision-motion model. Background Technology

[0002] With the advancement of computer technology, robots have been developed more extensively. For example, robotic arms are used to complete industrial production, and mobile robots are used in warehousing and logistics to complete tasks such as shelf handling and package sorting.

[0003] In existing technologies, robots are generally controlled based on programmed instructions. Maintenance personnel can pre-write corresponding programming instructions based on the tasks to be performed, enabling the robot to complete those tasks. However, this method makes it difficult for robots to complete tasks that are not pre-defined.

[0004] With the development of neural network technology, existing technologies attempt to control robots using neural network models, enabling them to complete undefined tasks. However, existing methods suffer from limitations in achieving satisfactory robot control. Summary of the Invention

[0005] In view of the above, one or more embodiments of this specification provide a method for controlling a robot and a method for training a vision-motion model.

[0006] According to a first aspect of one or more embodiments of this specification, a method for controlling a robot is proposed, which is executed using a vision-motion model, the vision-motion model including a vision prediction module and an action prediction module, the method comprising:

[0007] Acquire historical video data, historical action data of the robot, and data of the first task to be executed;

[0008] Based on the historical video data, historical action data, and first task data, the visual prediction module is used to predict the first video data.

[0009] Based on the first video data and historical video data, the motion prediction module is used to predict the first motion data;

[0010] The robot is controlled to perform the first action based on the first action data.

[0011] According to a second aspect of one or more embodiments of this specification, a method for training a vision-action model is provided, the vision-action model including a vision prediction module and an action prediction module, the method comprising:

[0012] Acquire training data, which includes at least the real first-label video and the executed first-label action in the first time period, the second task data to be executed in the second time period, and the real second-label video and the executed second-label action in the second time period.

[0013] Based on the first tagged video, the second task data, and the first tagged action, the second predicted video is obtained using the visual prediction module;

[0014] Based on the first tagged video and the second predicted video, the action prediction module is used to obtain the second predicted action;

[0015] Based on the second predicted action and the second labeled action, obtain the action prediction loss; based on the second predicted video and the second labeled video, obtain the visual prediction loss.

[0016] The parameters of the vision-action model are updated using the action prediction loss and the vision prediction loss.

[0017] According to a third aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the method described in the first or second aspect of the embodiments of this specification.

[0018] According to a fourth aspect of the embodiments of this specification, a computer device is provided, the computer device comprising:

[0019] processor;

[0020] Memory used to store processor-executable instructions;

[0021] The processor executes the executable instructions to implement the method as described in the first or second aspect of the embodiments of this specification.

[0022] According to a fifth aspect of the embodiments of this specification, a computer program product is provided that, when executed by a processor, implements the method described in the first or second aspect of the embodiments of this specification.

[0023] Based on this, this specification provides a method for controlling a robot, which utilizes a vision-motion model, including a vision prediction module and a motion prediction module. First, historical video data representing videos captured by the robot, historical motion data representing actions performed by the robot, and first task data identifying the task to be performed by the robot are acquired. Then, based on the historical video data, historical motion data, and first task data, the vision prediction module predicts the first video data; based on the first video data and historical video data, the motion prediction module predicts the first motion data; and the robot is controlled to perform a first motion based on the first motion data.

[0024] The method described above predicts robot motion data in two steps. First, visual assistance is introduced to predict the robot's posture after completing the task, using data such as the task to be performed. This posture is represented by video data. Then, based on the robot's current posture and the posture after completing the task (both postures represented by video), explicit dynamic modeling is performed to obtain the motion data to be performed. Compared to end-to-end models in related technologies, the method in this specification uses explicit dynamic modeling, which enhances the model's understanding of the object's state, making the generated control actions more consistent with task requirements. Furthermore, this allows the model to maintain motion continuity during long-term robot tasks.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0027] Figure 1 This is an architectural diagram of a vision-action model illustrated in this specification according to an exemplary embodiment.

[0028] Figure 2 This is a structural diagram of a characterizer illustrated in this specification according to an exemplary embodiment.

[0029] Figure 3 This is a flowchart illustrating a method for training a video-action model according to an exemplary embodiment.

[0030] Figure 4 This is a flowchart illustrating a method for controlling a robot according to an exemplary embodiment.

[0031] Figure 5This is a hardware structure diagram of a computer device illustrated in this specification according to an exemplary embodiment. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0033] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0034] In one implementation, the related technology uses a Vision-Language-Action (VLA) model to control the robot. In the VLA model, the robot's current state (such as the robot's joint angles) is taken as input, and the robot's action commands are taken as output, such as the rotation speed and rotation angle of the robot's motors.

[0035] In this approach, because video frames are directly mapped to robot movements, there is a lack of explicit dynamic modeling. This results in the model's insufficient understanding of changes in the object's state, leading to problems such as disjointed movements during long-duration tasks.

[0036] Based on this, this specification provides a method for controlling a robot, which utilizes a vision-motion model, including a vision prediction module and a motion prediction module. First, historical video data representing videos captured by the robot, historical motion data representing actions performed by the robot, and first task data identifying the task to be performed by the robot are acquired. Then, based on the historical video data, historical motion data, and first task data, the vision prediction module predicts the first video data; based on the first video data and historical video data, the motion prediction module predicts the first motion data; and the robot is controlled to perform a first motion based on the first motion data.

[0037] The method described above predicts robot motion data in two steps. First, visual assistance is introduced to predict the robot's posture after completing the task, using data such as the task to be performed. This posture is represented by video data. Then, based on the robot's current posture and the posture after completing the task (both postures represented by video), explicit dynamic modeling is performed to obtain the motion data to be performed. Compared to end-to-end models in related technologies, the method in this specification uses explicit dynamic modeling, which enhances the model's understanding of the object's state, making the generated control actions more consistent with task requirements. Furthermore, this allows the model to maintain motion continuity during long-term robot tasks.

[0038] The following section will describe a method for training a vision-motion model and a method for controlling a robot, as provided in this specification.

[0039] To facilitate understanding, the architecture of the vision-action model in this manual will first be explained. For example... Figure 1 As shown, the vision-action model illustrated in this specification may include: a visual tokenizer, a planner, an action prediction module, and a visual prediction module.

[0040] The system comprises several modules: a visual representation module, a visual predictor module, and an action prediction module. The visual predictor module encodes the acquired video (video frames / blocks) to obtain video data, such as latent variables. The planner decomposes the task step-by-step, executing the task repeatedly and outputting task data at each execution. The visual prediction module predicts the first video data based on the input historical video data, historical action data, and the task data to be executed. The action prediction module predicts which action data can be used to convert the historical video data into the first video data, based on the future first video data and historical video data. The functions and implementations of each module will be described in detail later.

[0041] First, a method for training a representation device illustrated in this specification will be described. In related technologies, pre-trained general-purpose visual representation devices are generally used to encode video frames. In this approach, the encoded data is not natively designed for robot control, thus affecting subsequent robot control.

[0042] To address this issue, this specification describes the joint training of a visual representation for encoding videos and an action representation for encoding actions, so that the output visual latent variables and action latent variables reside in the same semantic representation space.

[0043] The characterizer is as follows Figure 2 As shown, it includes visual representative and action representative. Figure 2Solid squares represent modules of the model, while dashed squares represent data. The visual representation unit includes an encoder and a decoder. The encoder encodes the original video block o (hereinafter referred to as the training video block) into a visual latent variable z, and the decoder decodes the visual latent variable z into a reconstructed video block. Latent variables refer to information representations in data that are not directly and explicitly observed, but can internally characterize their key structures or generative factors. In other words, latent variables are hidden, essential feature expressions behind the data; they are not the original input itself, but rather internal representations learned by the model to understand, compress, reconstruct, or generate the data. In one alternative implementation, a visual latent variable z can be encoded from a video block o, where a video block o includes at least one video frame.

[0044] In one alternative implementation, the visual representation can employ a Vision Transformer (ViT) architecture. In this architecture, the encoder first divides each video frame into multiple patches, calculating spatial attention within each frame. For multiple frames, causal attention is applied in the temporal dimension; that is, when encoding a latent variable, attention is calculated only based on the video block corresponding to that latent variable and the preceding video frames. Since the method described in this specification needs to be applied to real-time robot control, future video frames cannot be obtained during real-time video encoding. The causal attention mechanism conforms to the unidirectional temporal flow constraint of robot control.

[0045] The motion representation model includes a dynamic model, a forward prediction model, and a backward prediction model. The dynamic model is used to reverse the motion based on the video; specifically, it takes two adjacent visual latent variables as input, such as... and The latent variables of the actions performed between two visual latent variables This is the output. It includes two adjacent visual latent variables. and This refers to two visual latent variables whose corresponding video blocks were captured at adjacent times. Since the video sampling frequency is generally lower than the robot control signal frequency, a motion latent variable... It can correspond to multiple control signals between two video blocks. Specifically, the control signal can be the posture that the robot should reach when a control cycle arrives, or the movement (such as the rotation angle / speed) that the robot's drive device (such as a motor) should make when a control cycle arrives.

[0046] Backward and forward prediction models are used to output another visual latent variable based on visual and action latent variables. Specifically, the forward prediction model uses the previous visual latent variable... and action latent variables As input, a visual latent variable For output. The backward prediction model corresponds to the forward prediction model, with the subsequent visual latent variable... and action latent variables For input, a previous visual latent variable This is the output.

[0047] This specification does not limit the architecture of the action representation. In an alternative implementation, a multilayer perceptron (MLP) can be used to reduce the training and inference burden.

[0048] The process of training the representative will be explained next.

[0049] First, acquire real-shot training video blocks, which can be videos of people and / or machines performing actions. Then, input the training video blocks into the encoder of the visual representation device to obtain sample visual latent variables. Then, a decoder can be used to analyze the visual latent variables of the samples. Decoding yields reconstructed video blocks Based on the training video block o and the reconstructed video block , thus incurring the first loss.

[0050] Secondly, regarding the action representative, the visual latent variables obtained by the visual representative for the first sample from adjacent first and second video blocks can be used. Second sample visual latent variables Input the dynamic model of the action representative to obtain the latent variables of the sample actions. .

[0051] After obtaining the latent action variables, forward prediction models and / or backward prediction models are used to provide self-supervised signals. Specifically, this can be done based on the sample latent action variables. By restoring the visual latent variables of the first sample and / or the visual latent variables of the second sample, we obtain the restored visual latent variables. and / or The loss is calculated based on the difference between the sample visual latent variable and the restored visual latent variable.

[0052] Specifically, the visual latent variables of the first sample can be... and the latent variables of the sample actions Inputting the forward prediction model yields the second reconstructed visual latent variable. ; and / or, the second sample visual latent variables and the latent variables of the sample actions Inputting the inverse prediction model yields the first restored visual latent variable. .

[0053] Then, the second loss can be determined based on the first sample visual latent variable and / or the second sample visual latent variable, as well as the restored visual latent variable. In other words, it can be determined based on... and The difference between them, and / or, according to and The difference between them determines the second loss.

[0054] After obtaining the first and second losses, a comprehensive loss can be calculated based on both, for example, by weighting them together. Then, the visual and action representations are updated based on this comprehensive loss, which includes both the first and second losses. This completes the joint training process.

[0055] In the training process described above, since the output of the encoder of the visual representation is used as the input of the dynamic model of the action representation, and the parameters of the two representations are updated by combining the first loss and the second loss, the joint training process links the space of visual latent variables with the space of action latent variables. The constructed latent space is no longer a simple visual latent space, but a representation space that integrates semantic understanding and physical evolution laws, with vision and action located in a unified semantic space. This improves the alignment between visual state and action, making it easier for the model to learn the laws of action-induced state changes.

[0056] Furthermore, since this representation space is a native design for robot control, the visual-action model is further trained on the basis of this semantic representation space. This solves the problem of the visual latent variables generated by general visual representations being disconnected from the control target in related technologies. This enables the subsequent video-action model to learn and reason directly in a feature space with physical meaning, thereby improving the robot control effect of the trained video-action model.

[0057] Furthermore, in an alternative implementation, a teacher model can be introduced for the visual representation, which can be a visual representation with frozen parameters that has already been trained. The training video block o can be input into the teacher model to obtain the teacher's visual latent variables, and a third loss can be obtained based on the teacher's visual latent variables and the sample visual latent variables. If a third loss exists, the aforementioned comprehensive loss may include this third loss.

[0058] Since the feature space of the teacher model contains semantics related to objects, scenes, and tasks, the visual representation can learn the semantics related to objects, scenes, and tasks in the teacher feature space by projecting the sample visual latent variables onto the feature space of the teacher visual latent variables and calculating the loss between the two.

[0059] Furthermore, it should be noted that the action representation in the diagram is not used in the actual training of the vision-action model, or it only utilizes the dynamics model to generate labels for action latent variables. The decoder of the visual representation is also not used later. The reason for training modules that will not be used later is twofold: first, to provide supervision signals, such as reconstructing video blocks through the decoder of the visual representation to calculate the first loss, or reconstructing visual latent variables through the forward prediction / backward prediction model to calculate the second loss; second, to align the visual latent space and the action latent space, so that they can have unified semantic representations.

[0060] After the visual and action representations are trained, a video-action model can be pre-trained in a semantic representation space aligned with the visual and action latent spaces mentioned earlier. This video-action model includes a visual prediction module and an action prediction module. The method for training the video-action model will be described below, as follows: Figure 3 As shown, it includes the following steps.

[0061] Step 301: Obtain training data.

[0062] The training data includes at least the real first-label video and the executed first-label action in the first time period, the second task data to be executed in the second time period, and the real second-label video and the executed second-label action in the second time period.

[0063] The training data includes videos taken at multiple time points and the specific actions performed. Figure 3 The examples shown use the first and second time periods as examples for illustration. The processing methods for videos and actions in other time periods are similar and will not be repeated here.

[0064] In one optional implementation, the labeled video can be a latent variable encoded by the visual representation method described above, and the labeled action can be obtained by processing the visual latent variable (i.e., the labeled video) using the action representation method described above. In this way, since the labeled action can be obtained from the action representation method, the sources of data used for training can be expanded, the training samples enriched, and the model training not limited to the scarcer robot control data, thus improving training efficiency. Of course, the sources of the labeled video and labeled action described above do not imply limitation on this specification.

[0065] Furthermore, in one alternative implementation, the task data can be actual task text. In another alternative implementation, the task data can be obtained by the planner mentioned above. The planner can run at a certain frequency, generating one piece of task data each time it runs. Its running frequency is lower than that of the video-action model, ensuring that the video-action model can obtain the task data generated by the planner during each inference. Moreover, the planner generates one piece of task data per run, rather than generating multiple pieces of task data at once, making the task data more aligned with the task objectives.

[0066] For the planner, it can take the target task input by the user, historical video data, and historical task data generated based on the target task as input, and then process them to obtain task data, such as the second task data to be executed in the second time period mentioned above.

[0067] The target task can be a text command input by the user, the historical video data can be the visual latent variables obtained after processing the historical video as mentioned above, and the historical task data can be the task data that the planner has previously generated for the target task.

[0068] In one optional implementation, the task data can be formatted task data, such as including fields like "done," "instruction," "generation_instruction," and "local_scene_description." Here, "done" indicates whether the most recently generated task data has been completed; "instruction" indicates the specific instruction to be executed, such as picking up the red bottle; "generation_instruction" indicates the expected action details corresponding to the instruction, providing more specific behavioral expectations for the vision-action model; and "local_scene_description" represents a static scene description, including the current spatial layout and object arrangement, providing a grounding context for the vision-action model. The grounding context refers to the background information that constrains the model's understanding, generation, or inference process and maps it to specific inputs, environment, or real semantic basis.

[0069] In one alternative implementation, the historical video data input to the planner can be generated from videos with lower frame rates, such as keyframes. The historical video data can also be relatively short, for example, including three keyframes. Keyframes refer to frames that can be independently encoded and decoded in video encoding without relying on other frames for reconstruction, while non-keyframes are frames that require reference to keyframes or other encoded frames and cannot be fully reconstructed independently. Since the planner can complete planning even with sparser video, this reduces the planner's inference burden while ensuring the speed and accuracy of the task data.

[0070] In one alternative implementation of the planner architecture, the planner can be a model built on a pre-trained Vision-Language Model (VLM). The planner operates at a different frequency than the Vision-Action Model and can be trained separately with fine-tuning techniques, such as using Low-Rank Adaptation (LoRA).

[0071] Step 303: Based on the first tagged video, the second task data, and the first tagged action, the second predicted video is obtained using the visual prediction module.

[0072] Specifically, the visual prediction module takes task data, already acquired tagged videos, and executed tagged actions as input, and outputs predicted video data. In an optional implementation, the predicted video data can be in the form of visual latent variables.

[0073] For the visual prediction module, it can be implemented based on the Diffusion Transformer (DiT). DiT is a generative model architecture that uses a Transformer as a denoising network for a diffusion model to predict noise and gradually recover the target data during the diffusion process. It is most commonly used for image generation but can also be extended to other modalities. Various types of input data can be used as conditions for inference.

[0074] In one alternative implementation, since it is difficult to perform good reasoning based solely on the task text of the target task, and some actions may be difficult to describe with just words, the user also inputs a demonstration video for the task to help the robot complete the task better. For example, the user demonstrates how to complete the task in a video. In this case, the task text can be input into the planner to generate task data, and the demonstration video can be encoded by a visual representation device and then input into the visual prediction module to assist the visual prediction module in reasoning.

[0075] In other words, step 303 may include using the demonstration video (which may be in the form of a latent variable), the first-label video, the second-task data, and the first-label action as diffusion conditions to perform denoising processing and obtain the second-predicted video.

[0076] In an optional implementation, a multi-chunk prediction (MCP) mechanism can also be introduced. Ordinary training processes typically only generate the next visual latent variable, which can easily lead to short-sighted learning. For example, the generation module might directly copy the visual latent variable corresponding to the most recently acquired video block as its output, which affects the generation of motion data. Therefore, in this specification, the visual prediction module may include multiple prediction blocks. The first prediction block, used to predict the second predicted video, is the main architecture, followed by multiple prediction blocks in series. These multiple prediction blocks are lightweight MCP modules used to predict video frames following the second predicted video.

[0077] Specifically, the process of obtaining the second predicted video may include: generating the second predicted video using the first prediction block in the video prediction module, and outputting a second intermediate representation. The second intermediate representation is data generated by the first prediction block when predicting video frames; it represents not only the second predicted video but also information about the data input to the first prediction block. The first prediction block is also the backbone network in the video prediction module.

[0078] Then, using the second prediction block in the video prediction module, and taking the second intermediate representation as a condition, the third predicted video is obtained. It should be noted that there can be multiple prediction blocks. The second prediction block can also output a third intermediate representation, which can also be input into the third prediction block to predict the visual latent variables corresponding to the next video block. The processing method of the remaining prediction blocks in subsequent concatenation is similar to the prediction process of the first and second prediction blocks described above, and will not be repeated here.

[0079] It should also be noted that when calculating the loss, it can be based on the second predicted video and its corresponding label, as well as the third predicted video and its corresponding label. In other words, the visual prediction loss can include a first visual prediction loss and a second visual prediction loss. The first visual prediction loss can be obtained based on the second predicted video and the second labeled video, and the second visual prediction loss can be obtained based on the third predicted video and the third labeled video. The third labeled video is the labeled video taken in the third time period included in the training data. Then, a weighted sum of the first and second visual prediction losses can be calculated to obtain the final visual prediction loss, which can be used to update the parameters of the vision-action model.

[0080] With multiple MCPs participating in joint training, the backbone network (i.e., the first prediction block) is forced to encode multiple future dynamics in the current time-instance representation, rather than simply copying the appearance of the nearest neighbor block. This improves the long-term prediction ability of the world model and indirectly improves the quality of action decoding.

[0081] It should be noted that multiple MCP blocks do not participate in inference during actual use; only the first prediction block participates in the inference process, and the remaining prediction blocks are only used to assist the training of the first prediction block during training.

[0082] Step 305: Based on the first tagged video and the second predicted video, the second predicted action is obtained using the action prediction module.

[0083] Specifically, the action prediction module takes at least the tagged video collected in the first time period and the predicted second video in the second time period as input, and outputs the predicted second action to be performed in the second time period. In an optional implementation, the first tagged video, the second predicted video, and the second predicted action can all be in the form of latent variables. Accordingly, the vision-action model may also include a decoder for decoding action data (such as the second predicted action).

[0084] In an optional implementation, the action prediction module can also share a causal self-attention layer with the visual prediction module. Here, causal attention refers to the fact that in the standard Transformer self-attention mechanism, each position can see all other positions in the sequence (including the future), which is called bidirectional attention. However, in robot control, when the model makes a decision at time t, it is physically impossible to know what will happen at time t+1. Therefore, the causal self-attention layer enforces, through a mask matrix, that the current time can only focus on itself and all latent variables before it, and cannot see any future latent variables. Sharing the causal self-attention layer allows the video prediction module and the action prediction module to share the self-attention calculation results for the input data. In other words, the action prediction module also uses the first labeled video, the second task data, and the first labeled action as input to obtain more accurate prediction results.

[0085] The architecture of the action prediction module can also be implemented based on DiT. Accordingly, the first labeled video and the second predicted video are in the form of latent variables. Accordingly, step 305 can be to use the nearest labeled video and the second predicted video among the plurality of first labeled videos as diffusion conditions, and perform denoising processing using the context data of the visual prediction module to obtain the second predicted action.

[0086] In an alternative implementation, for the structure of the visual prediction module, based on its Transformer architecture, its feedforward network can be replaced with an expert network, i.e., a Mixture of Experts (MoE) model.

[0087] In other words, the visual prediction module includes a feature fusion layer, a router, and multiple expert networks. The process of predicting the second predicted video includes: processing the first labeled video, the first labeled action, and the second task data using the feature fusion layer to obtain an intermediate representation; then inputting this intermediate representation into the router, which determines the target expert network from among the multiple expert networks. Finally, based on the intermediate representation, the target expert network predicts the second predicted video. The feature fusion layer can be the causal self-attention layer shared with the action prediction module mentioned earlier.

[0088] For MoE training, in order to avoid overloading some experts, load balancing bias updates without auxiliary loss can be used to make the use of experts more balanced, ensuring that each expert is activated during training, and thus better completing the training of each expert network.

[0089] Furthermore, for the action prediction module, based on its Transformer architecture, in an optional implementation, it can differ from the visual prediction module and employ a lightweight dense feedforward network. Since visual prediction is more complex and bears the majority of the modeling burden, a MoE layer is used to expand capacity. The action branch, which does not bear the main modeling burden, uses a feedforward network to reduce inference overhead. Simultaneously, when using MoE layers, only a small number of experts are activated during each inference, thus reducing inference overhead while maintaining model capacity.

[0090] Step 307: Obtain the action prediction loss based on the second predicted action and the second labeled action, and obtain the visual prediction loss based on the second predicted video and the second labeled video.

[0091] Step 309: Update the parameters of the vision-action model using the action prediction loss and the vision prediction loss.

[0092] Specifically, the visual prediction loss can be calculated based on the second predicted video output by the visual prediction module and its corresponding label (second label video). In an optional implementation, as mentioned above, when multiple MCP blocks exist, the visual prediction loss can also be calculated based on the output of the MCP blocks. For the second predicted action output by the action prediction module, the action prediction loss can be calculated based on its corresponding label (second label action).

[0093] In terms of the form of loss, in one alternative implementation, visual flow-matching loss and action flow-matching loss can be used.

[0094] After obtaining the action prediction loss and the visual prediction loss, the model parameters can be updated based on both.

[0095] For the training data in the above training process, robot video data can be used for training. In another optional implementation, to increase the training samples, joint training with human data can be used. Specifically, the training data includes human video data obtained by filming human movements and machine video data obtained by filming robot movements. In application, for the movements corresponding to the human video data, the 6-DoF root pose of the hand and the finger joint configuration can be converted into a robot-compatible end effector interface using a mapping function, including the root pose of each hand and the scalar gripper opening degree.

[0096] For the processing of both types of data, the same visual prediction module and action prediction module are shared, but different action encoders and decoders are deployed for each type of data. In other words, the visual-action model also includes an encoder and decoder for action data. The action data corresponding to human video data and machine video data are encoded and decoded using different encoders and decoders in the visual-action model. In this way, although the actions corresponding to human video data and machine video data differ significantly, using different action encoders and decoders ensures that human action encoding and decoding does not affect machine action encoding and decoding, resulting in better training performance. Furthermore, the encoder and decoder branch corresponding to human actions is not used in the actual inference process, so adding this encoder and decoder does not increase the model's inference burden.

[0097] In an alternative implementation, instead of using completely separate staged training, multiple tasks can be jointly optimized on the same semantic latent space and the backbone of the vision-action model, and the task sampling ratio can be gradually shifted from image / video tasks to vision-action and robot control tasks as the training progresses.

[0098] In other words, the training task set of the vision-action model includes text-to-image tasks, text-to-video tasks, text-to-image-to-video action tasks, context learning tasks, and human-machine joint training tasks. The sampling ratio of each task during the training process is configured to be dynamically adjusted according to the training progress, and the sampling ratio of the human-machine joint training tasks increases as the training progresses.

[0099] This results in a coarse-to-fine hybrid training architecture. In this architecture, the model first learns the general appearance and dynamic state of the world from the training data, including images and videos, based on the image / video generation task. Then, human-machine joint training tasks (used to generate robot motion data) are gradually added to progressively strengthen motion-related control capabilities and reduce catastrophic forgetting that may occur later when training only with scarce robot data.

[0100] After describing the method for training the vision-action model shown in this specification, the method for using the vision-action model will be described next. Specifically, this specification illustrates a method for controlling a robot, such as... Figure 4 As shown, it includes the following steps:

[0101] Step 401: Obtain historical video data, historical action data of the robot, and data of the first task to be executed.

[0102] Specifically, in one alternative implementation, historical video data can be a sequence of video frames collected by sensors (such as cameras) mounted on the robot body over several control cycles.

[0103] In another optional implementation, the historical video data can be obtained by encoding the aforementioned video frame sequence using the visual representation device described above. In other words, step 401 may include: acquiring multiple video frames collected by the robot, and encoding them using a visual representation device to obtain several historical visual latent variables as the historical video data; the visual representation device is jointly trained with a motion representation device used to encode actions, so that the output visual latent variables and motion latent variables are in the same semantic representation space. At least one video frame can form a video block, and one visual latent variable can correspond to one video block.

[0104] Since the visual representation and the action representation used to encode actions are jointly trained, their output visual latent variables and action latent variables are in the same semantic representation space. This ensures that the subsequent visual prediction module and action prediction module can interact with each other in the same semantic space, avoiding cross-modal semantic misalignment.

[0105] Historical motion data refers to the sequence of control commands executed by the robot within the time period corresponding to the aforementioned historical video frames. This sequence of control commands can be generated by a vision-motion model. In an alternative implementation, if the historical video data is in the form of latent variables, the historical motion data can also be in the form of latent variables to align with the historical video data.

[0106] The first task data is a description of the subtasks that the robot needs to execute at the current moment. In an optional implementation, the first task data can be generated by the planner mentioned above. Specifically, the target task input by the user, historical video data, and historical task data generated based on the target task can be obtained; the first task data is generated using the planner based on the target task, the historical video data, and the historical task data. When the first task data is generated by the planner, the form of the first task data can be referred to the above description, and will not be repeated here.

[0107] The target task can include task text and a human demonstration video for that task. The task text is used as input to the planner to generate structured subtask instructions; the demonstration video is used as input to the visual prediction module as a condition for visual context learning, helping the model understand operational details that are difficult to describe with text alone.

[0108] In an alternative implementation, as described above, the planner can run in the background at a frequency lower than that of the vision-action model (e.g., about 1 Hz), writing the generated structured task data to an asynchronous shared buffer. The vision-action model reads the latest task data as conditional input at each action block boundary. The two are decoupled through the buffer, ensuring that the planner's inference delay does not block the high-frequency control loop.

[0109] Step 403: Based on the historical video data, historical action data, and first task data, use the visual prediction module to predict the first video data.

[0110] In an optional implementation, as described above, the visual prediction module can adopt a DiT architecture. Based on this, step 403 can specifically be that the visual prediction module uses historical visual latent variables, historical action latent variables, and the first task data as diffusion conditions to perform noise reduction processing to obtain the first video data.

[0111] Furthermore, as mentioned above, in an optional implementation where the visual prediction module employs a Transformer architecture, the feedforward network of the Transformer architecture can be replaced with a MoE to increase model capacity. Accordingly, the visual prediction module includes a feature fusion layer, a router, and multiple expert networks. The process of predicting the first video data specifically includes: processing the historical video data, historical action data, and first task data using the feature fusion layer to obtain a first intermediate representation; then, based on the first intermediate representation, determining a target expert network from multiple expert networks using the router; and finally, predicting the first video data using the target expert network based on the first intermediate representation.

[0112] The meaning of the first intermediate representation is detailed above and will not be repeated here. In an optional implementation, the feature fusion layer is a causal self-attention layer shared with the action prediction module. It enforces through a causal mask that the current moment can only focus on itself and previous latent variables, which conforms to the unidirectional time flow constraint of robot control. The router dynamically selects the activated expert network based on the first intermediate representation, enabling the model to call the expert network matching the scenario for different scenarios (such as rigid grasping, flexible manipulation, etc.), thereby expanding the model capacity without increasing the computational load of a single inference.

[0113] In an optional implementation, as described above, the target task also includes a demonstration video for that task. In this case, predicting the first video data includes: using the demonstration video, the historical video data, historical action data, and the first task data as diffusion conditions, performing denoising processing to obtain the first video data. The demonstration video may be processed by a visual representation device before being input into the visual prediction module.

[0114] Demonstration videos can help models better predict first-view video data, leading to more accurate completion of the target task.

[0115] Step 405: Based on the first video data and historical video data, use the motion prediction module to predict the first motion data.

[0116] In an optional implementation, as described above, the motion prediction module may adopt a DiT architecture. Based on this, step 405 may specifically be that the motion prediction module uses at least the most recent visual latent variable in the historical video data and the first video data (i.e., the first visual latent variable) as diffusion conditions, and performs denoising processing using the context data of the visual prediction module to obtain the first motion latent variable as the first motion data.

[0117] The context data for the visual prediction module here refers to the key-value cache (KV Cache) calculated by the causal self-attention layer shared by the visual prediction module and the action prediction module, as mentioned earlier. Since the two modules share the same set of self-attention calculation results, the action prediction module does not need to repeatedly calculate the attention weights of the historical context, which saves computational resources and ensures that both modules have a consistent understanding of what happened in the past.

[0118] Furthermore, regarding the structure of the action prediction module, as mentioned earlier, when it adopts the Transformer architecture, its feedforward network can employ a lightweight, dense structure, excluding routers and expert networks. The reason for this asymmetric design between the action prediction module and the vision prediction module is that action decoding essentially involves inverse dynamics solving given the current state and the expected future state. The required knowledge complexity is far lower than the world simulation in the vision prediction module, and the parameter capacity of the dense structure is sufficient to cover it. Simultaneously, since the action output directly drives the motor, it has high requirements for real-time performance and determinism. The feedforward network has a smaller inference burden, avoiding the additional latency and jitter introduced by a larger inference burden.

[0119] Step 407: Control the robot to perform the first action based on the first action data.

[0120] In one optional implementation, the first motion data can be a specific low-level control command. In another optional implementation, the first motion data can be in the form of a motion latent variable, and correspondingly, it can be decoded into a low-level control command executable by the robot using the motion encoding / decoding decoder mentioned above. The decoded control command is sent to the robot controller to drive the robot to perform the corresponding action.

[0121] Furthermore, this specification proposes a look-ahead inference mechanism to ensure that the robot's response is not delayed. Specifically, video-action models in related technologies often employ a synchronous serial mechanism during inference, meaning they wait for the current action to complete and for new observations to be acquired before initiating the next round of video and action prediction. This approach results in high inference latency, leading to sluggish robot responses and making it difficult to meet real-time requirements.

[0122] Based on this, this specification also proposes a look-ahead reasoning mechanism, in which the next action data is predicted in parallel based on the predicted first video data during the execution of step 407.

[0123] Specifically, the robot is controlled to perform a first action during a first time period. During this first time period, the prediction of the next action data is also performed in parallel. Specifically, based on at least a portion of the historical video data, the first video data, the first action data, the historical action data, and the second task data that follows the first task data and is to be executed, the visual prediction module predicts the second video data; the second video data is added to a cache pool; and based on the first and second video data, the action prediction module predicts the second action data.

[0124] At least a portion of the historical video data can be obtained by removing the data from the historical video data whose acquisition time is furthest from the current time. Similarly, the historical motion data used in this step can also be obtained by removing the historical motion data whose execution time is furthest from the current time. The generation method of the second task data can be the same as that of the first task data, and will not be repeated here. Furthermore, regarding the second task data, if it is generated by the planner, since the planner's execution frequency can be lower than the video-motion model's execution frequency, in an optional implementation, the second task data can be the same as the first task data. Of course, the second task data can also be different from the first task data; this specification does not limit this.

[0125] The cache pool stores information such as historical video data and first video data. Specifically, the cache pool can be in the form of KVCache.

[0126] The first video data is the video data predicted in the aforementioned steps. Since the first action has not been completed in this case, the actual video data corresponding to the first video data cannot be obtained. Therefore, the first video data is used for inference here.

[0127] Correspondingly, if the first video data is not the first predicted video data, the historical video data obtained in step 401 may also include the video data predicted by the visual-action model in the previous cycle.

[0128] In this way, the prediction of the next action data and the execution of the current action data are carried out in parallel, ensuring the real-time performance of action prediction.

[0129] In addition, to prevent the model from continuously predicting actions by imagination, which would affect the accuracy of the predicted action data, this manual allows the predicted video data stored in the cache pool to be replaced with the newly acquired video data after the actual video data is collected.

[0130] Specifically, the system can also collect the first observation video obtained when the first action is performed in the first time period; replace the first video data stored in the cache pool with the video data corresponding to the first observation video and then incorporate it into the historical video data for prediction in subsequent time periods. Here, the first observation video can be a collected video frame, and the video data corresponding to the first observation video can be the video data obtained after encoding the first observation video through a visual representation device.

[0131] Thus, the method in this manual does not rely solely on imagination for prediction, but rather continuously updates the historical video data used in the prediction based on the actual video footage captured, thereby ensuring the accuracy of the prediction results.

[0132] Corresponding to the embodiments of the foregoing methods, this specification also provides embodiments of the apparatus and the computer equipment to which it is applied.

[0133] This specification also provides a method for controlling a robot, which utilizes a vision-motion model, the vision-motion model including a vision prediction module and a motion prediction module, and the device includes:

[0134] The first acquisition module is used to acquire historical video data, historical action data executed by the robot, and data of the first task to be executed.

[0135] The first prediction module is used to predict the first video data based on the historical video data, historical action data, and first task data using the visual prediction module.

[0136] The second prediction module is used to predict the first action data based on the first video data and historical video data using the action prediction module;

[0137] The execution module is used to control the robot to perform the first action based on the first action data.

[0138] This specification also provides an apparatus for training a vision-action model, the vision-action model including a vision prediction module and an action prediction module, the apparatus comprising:

[0139] The second acquisition module is used to acquire training data, which includes at least the real first-label video and the executed first-label action in the first time period, the second task data to be executed in the second time period, and the real second-label video and the executed second-label action in the second time period.

[0140] The third prediction module is used to obtain the second predicted video based on the first labeled video, the second task data, and the first labeled action using the visual prediction module.

[0141] The fourth prediction module is used to obtain a second predicted action based on the first tagged video and the second predicted video, using the action prediction module.

[0142] The loss calculation module is used to obtain the action prediction loss based on the second predicted action and the second labeled action, and to obtain the visual prediction loss based on the second predicted video and the second labeled video.

[0143] The parameter update module is used to update the parameters of the vision-action model using the action prediction loss and the vision prediction loss.

[0144] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0145] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0146] This manual also provides a computer system 1000. Figure 5 An example computer system 1000 is illustrated. In a particular embodiment, one or more computer systems 1000 may perform one or more steps of one or more methods described or illustrated herein. In a particular embodiment, one or more computer systems 1000 may also provide functionality described or illustrated herein, such as the functionality of one or more engines in machine learning system 100 (or other arbitrary functionality).

[0147] This disclosure contemplates any suitable number of computer systems 1000. The computer systems 1000 are contemplated in any suitable physical form. By way of example and not limitation, the computer system 1000 may be an embedded computer system, a system-on-a-chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a notebook or laptop system, an interactive kiosk, a mainframe, a grid of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or any suitable combination thereof. Where appropriate, one or more computer systems 1000 may be single or distributed; may span multiple locations or multiple machines; may span one or more data centers; or reside in the cloud, which may include one or more cloud components and communicate through one or more networks. Where appropriate, one or more computer systems 1000 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations, and may be performed in real-time or batch mode, or at different times and / or different locations.

[0148] In a particular embodiment, computer system 1000 includes: processor 1002, memory 1004, storage 1006, input / output (I / O) interface 1008, communication interface 1010, and interconnect structure 1012. Although this disclosure describes and illustrates an example computer system 1000 having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable number of any suitable components and any suitable arrangement.

[0149] In a particular embodiment, processor 1002 is used to execute instructions. Processor 1002 can retrieve instructions through internal registers, internal caches, memory 1004, or storage 1006, decode and execute them, and write one or more results to the internal registers, internal caches, memory 1004, or storage 1006. Where appropriate, processor 1002 may include one or more internal caches. Processor 1002 may also include one or more internal registers, one or more arithmetic logic units (ALUs), and one or more processor cores. Processor 1002 may be a multi-core processor and may include, but is not limited to: microcontrollers, microprocessors, RISC, ASICs, ASIPs, CPUs, GPUs, PPUs, DSPs, FPGAs, ARMs, and programmable logic devices (PLDs), or other circuits or processors capable of performing one or more functions, or any combination thereof.

[0150] In a particular embodiment, memory 1004 may be used to store instructions to be executed by processor 1002 and / or data to be operated on by processor 1002. Memory 1004 may include main memory and may include random access memory (RAM). Where appropriate, memory 1004 may be volatile memory (e.g., dynamic RAM (DRAM) or static RAM (SRAM)), or other suitable memory types. Processor 1002 may couple memory 1004 to it via one or more memory paths for instruction and data access. Where appropriate, memory 1004 may include one or more memory cells.

[0151] In certain embodiments, storage 1006 may be used to provide non-volatile storage for data and / or instructions. Storage 1006 may include, but is not limited to: hard disk drive (HDD), solid-state drive (SSD), flash memory, ROM, erasable ROM, optical media (e.g., optical disc), magnetic media (e.g., magnetic disk or magnetic tape), USB storage device, or combinations thereof. Where appropriate, storage 1006 may be located inside or outside computer system 1000 and may include one or more storage control units to facilitate communication between processor 1002 and storage 1006.

[0152] In a particular embodiment, I / O interface 1008 may be used to provide an interface between computer system 1000 and one or more I / O devices. I / O interface 1008 may include hardware, software, or a combination of both. I / O devices enable communication between a person and computer system 1000 or between computer system 1000 and external devices. Exemplary I / O devices may include, but are not limited to: keyboard, keypad, microphone, display, mouse, printer, scanner, speaker, camera, touch input device, stylus, trackball, tablet, and other suitable I / O devices; and I / O devices may also include one or more sensors. Where appropriate, I / O interface 1008 may include one or more device or software drivers to enable processor 1002 to control one or more associated I / O devices.

[0153] In a particular embodiment, communication interface 1010 may be used to provide a communication interface between computer system 1000 and one or more other computer systems 1000 and / or one or more networks. Communication interface 1010 may include hardware, software, or a combination of both. Communication interface 1010 may support wired or wireless communication and may include a network interface controller (NIC) or a wireless adapter (e.g., a Wi-Fi adapter). Exemplary networks may include, but are not limited to: ad hoc networks, personal area networks (PANs), local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), one or more portions of the Internet, and so on. Where appropriate, communication interface 1010 may include one or more communication interfaces 1010.

[0154] In a particular embodiment, the interconnect structure 1012 is used to interconnect the components of the computer system 1000. The interconnect structure 1012 may include, but is not limited to: bus, switching structure, crossbar switch, on-chip interconnect network, point-to-point interconnect or other suitable interconnect structure, or any combination thereof.

[0155] This specification also provides a computer-readable non-volatile storage medium that stores a computer program. When executed by a processor, the computer program can be used to perform one or more steps of one or more methods described or shown herein, or to implement one or more functions described or shown herein. The computer-readable non-volatile storage medium may include one or more semiconductor or other integrated circuit (IC) based memories (e.g., FPGA or ASIC built-in memory / configuration memory), HDDs, HHDs, optical discs, optical disc drives (ODDs), magneto-optical discs, magneto-optical drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, SSDs, RAM drives, secure digital cards, or combinations thereof, or other suitable media.

[0156] This specification also provides a computer program product that, when executed by a processor, implements the aforementioned method.

[0157] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0158] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0159] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

Claims

1. A method for controlling a robot, executed using a vision-motion model, the vision-motion model including a vision prediction module and an action prediction module, the method comprising: Acquire historical video data, historical action data of the robot, and data of the first task to be executed; Based on the historical video data, historical action data, and first task data, the visual prediction module is used to predict the first video data. Based on the first video data and historical video data, the motion prediction module is used to predict the first motion data; The robot is controlled to perform the first action based on the first action data.

2. The method according to claim 1, further comprising: Obtain the user-input target task, historical video data, and historical task data generated based on the target task; The first task data is generated using a planner based on the target task, the historical video data, and the historical task data.

3. The method according to claim 2, wherein, The target task includes task text and a demonstration video for the task; the task text is used to input the planner, and the demonstration video is used to input the visual prediction module. The visual prediction module is implemented based on a diffusion transformer (DiT); the predicted first video data includes: The demonstration video, historical video data, historical action data, and first task data are used as diffusion conditions to perform noise reduction processing to obtain the first video data.

4. The method according to claim 1, wherein, The acquisition of historical video data includes: Multiple video frames collected by the robot are acquired and encoded using a visual representation device to obtain several historical visual latent variables as the historical video data; the visual representation device is jointly trained with the action representation device used to encode the action so that the output visual latent variables and action latent variables are in the same semantic representation space.

5. The method according to claim 4, wherein, The motion prediction module is implemented based on DiT. The first video data is the first visual latent variable; The predicted first action data includes: Using the historical visual latent variable corresponding to the most recent video frame among the multiple video frames and the first visual latent variable as diffusion conditions, denoising processing is performed using the context data of the visual prediction module to obtain the first action latent variable as the first action data.

6. The method according to claim 1, wherein, The method further includes executing the first action of the controlled robot in a first time period; the method also includes executing the action in parallel during the first time period. Based on at least a portion of the historical video data, the first video data, the first motion data, the historical motion data, and the second task data that follows the first task data and is to be executed, the visual prediction module is used to predict the second video data. Add the second video data to the cache pool; Based on the first video data and the second video data, the motion prediction module is used to predict the second motion data.

7. The method according to claim 6, further comprising: The first observation video was obtained when the first action was performed in the first time period; The first video data is replaced with the video data corresponding to the first observed video and then incorporated into the historical video data for subsequent time period prediction.

8. The method according to claim 1, wherein, The visual prediction module includes a feature fusion layer, a router, and multiple expert networks; The predicted first video data includes: The historical video data, historical action data, and first task data are processed using the feature fusion layer to obtain a first intermediate representation; Based on the first intermediate representation, the target expert network is determined from multiple expert networks using the router; Based on the first intermediate representation, the target expert network is used to predict the first video data.

9. A method for training a vision-action model, the vision-action model comprising a vision prediction module and an action prediction module, the method comprising: Acquire training data, which includes at least the real first-label video and the executed first-label action in the first time period, the second task data to be executed in the second time period, and the real second-label video and the executed second-label action in the second time period. Based on the first tagged video, the second task data, and the first tagged action, the second predicted video is obtained using the visual prediction module; Based on the first tagged video and the second predicted video, the action prediction module is used to obtain the second predicted action; Based on the second predicted action and the second labeled action, obtain the action prediction loss; based on the second predicted video and the second labeled video, obtain the visual prediction loss. The parameters of the vision-action model are updated using the action prediction loss and the vision prediction loss.

10. The method according to claim 9, wherein, The training data also includes third-label videos taken in the third time period; The process of obtaining the second predicted video includes: The first prediction block in the video prediction module is used to generate a second predicted video, and a second intermediate representation is output. Using the second prediction block in the video prediction module, and taking the second intermediate representation as a condition, a third predicted video is obtained; The visual prediction loss includes a first visual prediction loss and a second visual prediction loss; obtaining the visual prediction loss based on the second predicted video and the second labeled video includes: A first visual prediction loss is obtained based on the second predicted video and the second labeled video, and a second visual prediction loss is obtained based on the third predicted video and the third labeled video.

11. The method according to claim 9, wherein, The first labeled video and the second labeled video are encoded using a visual representation device, which is trained in the following way: The training video block is input into the encoder of the visual representation to obtain sample visual latent variables; the reconstructed video block is obtained by decoding the sample visual latent variables; and a first loss is obtained based on the training video block and the reconstructed video block. The first sample visual latent variable and the second sample visual latent variable obtained by the visual representation device for adjacent first and second video blocks are input into the dynamic model of the action representation device to obtain sample action latent variables; the first sample visual latent variable and / or the second sample visual latent variable are restored according to the sample action latent variable to obtain the restored visual latent variable; The second loss is determined based on the first sample visual latent variable and / or the second sample visual latent variable, as well as the restored visual latent variable; The visual and action representations are updated based on a comprehensive loss, which includes a first loss and a second loss.

12. The method according to claim 11, wherein, Based on the sample action latent variables, the visual latent variables of the first sample and / or the visual latent variables of the second sample are reconstructed to obtain the reconstructed visual latent variables, including: The first sample visual latent variable and the sample action latent variable are input into the forward prediction model to obtain the second restored visual latent variable; and / or, The second sample visual latent variable and the sample action latent variable are input into the inverse prediction model to obtain the first restored visual latent variable.

13. The method according to claim 11, wherein, The training of the visual representation also includes: The training video blocks are input into the teacher model to obtain the teacher's visual latent variables. Based on the teacher's visual latent variables and the sample visual latent variables, the third loss is obtained. The total loss also includes a third loss.

14. The method according to claim 11, wherein, The first tag action and the second tag action are generated by the action representative based on the input tag video.

15. The method according to claim 9, wherein, The training task set of the vision-action model includes text-to-image tasks, text-to-video tasks, text-to-image-to-video action tasks, context learning tasks, and human-machine joint training tasks. The sampling ratio of each task during the training process is configured to be dynamically adjusted according to the training progress, and the sampling ratio of the human-machine joint training tasks increases as the training progresses.

16. The method according to claim 9, wherein, The training data includes human video data obtained by filming human actions and machine video data obtained by filming robot actions; the motion data corresponding to the human video data and machine video data are encoded and decoded using different codecs in the vision-motion model.

17. A computer device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-16 by executing the executable instructions.

Images