Model training method, action generation method, device and electronic equipment

Abstract

This disclosure provides a model training method, action generation method, apparatus, and electronic device. During the model training phase, by introducing a causal-based attention mask in the diffusion transformer, access to future visual latent variables by action sequence latent variables is restricted. However, future visual latent variables can access action sequence latent variables, ensuring that the action sequence prediction branch is independent of future visual information during training. This forces the model to learn the causal relationship between actions and environmental changes based on multimodal data. As a result, future visual latent variables and related computational paths can be directly removed during the inference phase, achieving decoupling between training and inference. This allows the model to stably output high-quality action sequences without relying on predicting future visual images during the inference phase, fundamentally solving the problems of high inference costs and severe error accumulation, significantly reducing inference latency, and improving model inference efficiency and robot motion control accuracy.

Description

Technical Field

[0001] This disclosure relates to artificial intelligence technology, and in particular to a model training method, an action generation method, a device, and an electronic device. Background Technology

[0002] As robot control tasks become increasingly complex, Vision Language Action (VLA) models are gradually becoming the core framework for enabling robots to understand their environment and perform high-level tasks. VLA model training typically relies on imitation learning paradigms, with training signals primarily derived from sparse action labels. However, the robot's observation space (e.g., multi-view RGB video) exhibits significant spatiotemporal continuity. This mismatch between sparse supervision and dense observation makes it difficult for the model to capture realistic physical dynamics, resulting in a lack of continuity constraints in action distribution and a tendency to become dependent on specific contexts, thus affecting generalization ability.

[0003] Related technologies typically use predicting future visual images as an auxiliary task, enabling models to predict future videos or images through inference to obtain additional temporal supervision signals, thereby improving the continuity of action prediction. However, this approach leads to high model inference latency and low inference efficiency, and errors in visual prediction accumulate over time and propagate into action prediction. Summary of the Invention

[0004] This disclosure provides a model training method, action generation method, apparatus, and electronic device that can reduce inference latency, improve model inference efficiency, and enhance the accuracy of robot motion control.

[0005] One aspect of this disclosure provides a model training method, including: Acquire first training data, which includes sample action instructions, and a first sample video and sample action sequence when the executor executes the sample action instructions; The first training data is encoded by the encoder in the visual language action model to obtain the observed visual latent variables and future visual latent variables corresponding to the first sample video, the current state latent variables and action sequence latent variables corresponding to the sample action sequence, and the instruction text features corresponding to the sample action instruction. The observed visual latent variable, the future visual latent variable, the current state latent variable, the action sequence latent variable, and the instruction text feature are input into the diffusion transformer in the visual language action model to obtain the prediction results for the future visual latent variable and the action sequence latent variable. The diffusion transformer is used to restrict the action sequence latent variable from accessing the future visual latent variable during the attention calculation process by using an attention mask. The model parameters of the diffusion converter are updated based on the prediction results.

[0006] Optionally, the step of inputting the observed visual latent variable, the future visual latent variable, the current state latent variable, the action sequence latent variable, and the instruction text features into the diffusion transformer in the visual language action model to obtain prediction results for the future visual latent variable and the action sequence latent variable includes: Noise is added to both the future visual latent variable and the action sequence latent variable; The observed visual latent variables, the current state latent variables, the instruction text features, the noisy future visual latent variables, and the action sequence latent variables are concatenated into an input sequence. The input sequence is input into the diffusion transformer, which generates a feature matrix and performs masking and forward propagation on the feature matrix to obtain a first prediction result for the future visual latent variable and a second prediction result for the action sequence latent variable.

[0007] Optionally, updating the model parameters of the diffusion converter based on the prediction results includes: The visual flow matching loss is determined based on the future visual latent variables and the first prediction result; The action flow matching loss is determined based on the latent variables of the action sequence and the second prediction result; The total flow matching loss is determined based on the visual flow matching loss and the action flow matching loss. Backpropagation is performed based on the total flow matching loss to update the model parameters of the diffusion transformer.

[0008] Optionally, before acquiring the first training data, the method further includes: Acquire second training data, which includes second sample videos, which include videos used to demonstrate the motion of objects; The second training data is encoded by the encoder in the visual language action model to obtain the observed visual latent variables and future visual latent variables corresponding to the second sample video. The observed visual latent variables and future visual latent variables corresponding to the second sample video are input into the diffusion transformer to pre-train the diffusion transformer.

[0009] Optionally, the encoding of the first training data using the encoder in the visual language action model to obtain the observed visual latent variables and future visual latent variables corresponding to the first sample video, the current state latent variables and action sequence latent variables corresponding to the sample action sequence, and the instruction text features corresponding to the sample action command, includes: The first sample video is input into a pre-trained variational autoencoder. The first frame of the first sample video is encoded by the variational autoencoder to obtain the observed visual latent variable, and the remaining video frames of the first sample video are encoded to obtain the future visual latent variable. The sample action sequence is input into a pre-trained state action encoder. The state action encoder encodes the initial state data in the sample action sequence to obtain the current state latent variable, and encodes the action sequence data in the sample action sequence to obtain the action sequence latent variable. The sample action command is input into a text encoder, and the text encoder encodes the sample action command to obtain the command text feature.

[0010] Another aspect of this disclosure provides an action generation method, including: Acquire action commands, as well as the current observation video and current status data of the actuator; The action command, the currently observed video, and the current state data are encoded by the encoder in the visual language action model to obtain command text features, observed visual latent variables, and current state latent variables. The visual language action model is trained using the model training method described in any one of claims 1 to 5 above. The input data is input into the diffusion transformer of the visual language action model to obtain the action sequence latent variable output by the diffusion transformer. The input data includes the instruction text features, the observed visual latent variable, the current state latent variable, and the action sequence latent variable to be predicted. The latent variables of the action sequence are decoded into the action sequence of the actuator.

[0011] Optionally, the input data further includes future visual latent variables to be predicted. The step of inputting the input data into the diffusion transformer of the visual language action model to obtain the action sequence latent variables output by the diffusion transformer includes: The instruction text features, the observed visual latent variables, the current state latent variables, the action sequence latent variables to be predicted, and the future visual latent variables to be predicted are input into the diffusion transformer. The action sequence latent variable to be predicted is diffused through the diffusion transformer, and the future visual latent variable to be predicted is denoised using the key-value cache in the diffusion transformer, so as to obtain the action sequence latent variable and the future visual latent variable output by the diffusion transformer. The method further includes: The future visual latent variables are decoded into predictive videos.

[0012] Another aspect of this disclosure provides a model training apparatus, comprising: The first acquisition module is used to acquire first training data, the first training data including sample action instructions, and a first sample video and sample action sequence when the executor executes the sample action instructions; The first encoding module is used to encode the first training data through the encoder in the visual language action model to obtain the observed visual latent variable and future visual latent variable corresponding to the first sample video, the current state latent variable and action sequence latent variable corresponding to the sample action sequence, and the instruction text feature corresponding to the sample action instruction. The latent variable prediction module is used to input the observed visual latent variable, the future visual latent variable, the current state latent variable, the action sequence latent variable, and the instruction text features into the diffusion transformer in the visual language action model to obtain the prediction results for the future visual latent variable and the action sequence latent variable. The diffusion transformer is used to restrict the action sequence latent variable from accessing the future visual latent variable during the attention calculation process by using an attention mask. The model training module is used to update the model parameters of the diffusion converter based on the prediction results.

[0013] Another aspect of this disclosure provides an action generation apparatus, comprising: The second acquisition module is used to acquire action commands as well as the current observation video and current status data of the executor; The second encoding module is used to encode the action instruction, the currently observed video, and the current state data respectively through the encoder in the visual language action model to obtain instruction text features, observed visual latent variables, and current state latent variables. The visual language action model is trained using the model training method as described in any one of claims 1 to 5 above. The latent variable inference module is used to input input data into the diffusion transformer of the visual language action model to obtain the action sequence latent variable output by the diffusion transformer. The input data includes the instruction text features, the observed visual latent variable, the current state latent variable, and the action sequence latent variable to be predicted. The first decoding module is used to decode the action sequence latent variables into the action sequence of the actuator.

[0014] In another aspect of this disclosure, an electronic device is provided, comprising: Memory, used to store computer programs; A processor is configured to execute a computer program stored in the memory, wherein, when the computer program is executed, it implements the methods described above.

[0015] In another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the methods described above.

[0016] In another aspect of this disclosure, a computer program product is provided, including computer program instructions that, when executed by a processor, implement the method described above.

[0017] Based on the embodiments of this disclosure, a world-action modeling architecture centered on action is provided. During the model training phase, by introducing a causal-based attention mask in the diffusion transformer, access to future visual latent variables by action sequence latent variables is restricted. However, future visual latent variables can access action sequence latent variables. While introducing future visual prediction, this ensures that the action sequence prediction branch is independent of future visual information during training. This forces the model to learn the causal relationship between actions and environmental changes based on multimodal data, thereby constructing a denser and more physically consistent temporal supervision, improving the stability and physical reliability of the model's action prediction. During the inference phase, future visual latent variables and related computational paths can be directly removed, achieving decoupling between training and inference. This allows the model to stably output high-quality action sequences without relying on predicting future visual images during the inference phase. This fundamentally solves the problem of high inference costs and severe error accumulation caused by traditional models still needing to generate future visual images for action inference during the inference phase. It significantly reduces inference latency and improves model inference efficiency and the accuracy of robot action control.

[0018] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0019] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0020] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein: Figure 1 A flowchart of one embodiment of the model training method disclosed herein; Figure 2 A flowchart of another embodiment of the model training method disclosed herein; Figure 3 This is a schematic diagram illustrating a masked feature matrix as an exemplary embodiment of the present disclosure; Figure 4 A flowchart of another embodiment of the model training method disclosed herein; Figure 5This is a schematic diagram illustrating a model training process as an exemplary embodiment of the present disclosure; Figure 6 A flowchart of one embodiment of the action generation method of this disclosure; Figure 7 This is a schematic diagram illustrating a model reasoning process as an exemplary embodiment of the present disclosure; Figure 8 This is a structural block diagram of one embodiment of the model training apparatus disclosed herein; Figure 9 This is a structural block diagram of one embodiment of the action generation apparatus of this disclosure; Figure 10 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0021] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0022] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0023] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0024] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0025] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0026] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0027] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0028] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0029] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0030] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0031] Currently, to address the mismatch between sparse supervision and dense observations in VLA model training, three main types of solutions have been proposed: 1. VLA Model Based on Vision Language Model (VLM) These methods introduce future visual state prediction as an auxiliary task on top of conventional imitation learning, allowing the model to obtain additional temporal supervision signals by predicting future videos or images, thereby improving the continuity of action prediction. However, discriminative VLMs are not suitable for modeling complex generative dynamics, making it difficult to provide high-quality future visual predictions. Furthermore, the auxiliary future state prediction often lacks realistic physical consistency, resulting in insufficient stability of action prediction. Moreover, its supervision still mainly relies on action labels, failing to truly compensate for the problem of sparse temporal supervision and providing weak support for long-term tasks.

[0032] 2. World-Action Model for Joint Video-Motion Modeling This type of method utilizes video generation models as its backbone to simultaneously predict future visual dynamics and action sequences. By modeling visual evolution and action decisions within a unified framework, it achieves more detailed modeling of environmental dynamics, thereby enhancing supervision density and action plausibility. However, its inference phase requires generating a large number of future video tokens and necessitates multi-step diffusion sampling, resulting in extremely high inference latency. Furthermore, video prediction relies on pixel-level modeling, making it easy for errors to accumulate over time and propagate to action prediction, affecting the robot's execution stability. Moreover, the quality of action prediction is highly dependent on the accuracy of video generation, thus limiting its overall robustness.

[0033] 3. Two-stage video-motion generation method These methods employ phased modeling. The first phase generates future visual frames using a video prediction model, and the second phase uses an inverse dynamics model to infer the corresponding actions based on the predicted video. By separating visual prediction and action decoding, they attempt to enhance the controllability of action prediction. However, video prediction errors directly affect the final action generation, causing errors to cascade and amplify. Furthermore, the two-stage inference structure is lengthy, involving multiple modules in series, resulting in low inference efficiency and high deployment complexity. The inference process still requires rolling out the future video portion, making it unavoidable for high computational costs and failing to meet real-time control requirements.

[0034] In summary, while the aforementioned methods improve supervision density to some extent, they still have shortcomings, mainly including two major technical bottlenecks: First, these methods require generating future videos during the model inference stage, leading to high latency and error accumulation, making real-time deployment in real robot systems difficult. Second, there is still a lack of unified methods for effectively utilizing large-scale, heterogeneous video data to construct robot control models that can generalize to multiple tasks and scenarios. To address these issues, this disclosure provides a motion-centric world-motion modeling framework and its training method, enabling the model to utilize future vision as a supervisory constraint during the training stage, while eliminating the need for video generation during the inference stage, thereby achieving efficient inference and broad generalization.

[0035] Figure 1 A flowchart illustrating a model training method provided for an exemplary embodiment of this disclosure. Figure 1 As shown, the method includes the following steps: Step 101: Obtain first training data, which includes sample action instructions, and first sample video and sample action sequence when the executor executes the sample action instructions.

[0036] The first training data can be data related to the sample actuator (e.g., robot, robotic arm) performing task actions, including sample action commands issued by the user, sample action sequences during the execution of the sample action commands by the actuator, and the first sample video collected. The sample action commands can be natural language commands, input by the user through voice input, text input, etc., for example, the natural language command could be "clean the table". The first sample video can be video data collected from at least one perspective by the video acquisition device on the actuator. For example, it could be three videos collected from three angles by the camera on the robot's left arm, the camera on its right arm, and the head camera when the robot performs the sample action command "clean the table". The sample action sequence includes the action data sequence generated by the actuator when executing the sample action commands, and may also include body state data collected by devices such as LiDAR and inertial measurement units in the actuator, such as the coordinates of the robotic arm's key points in the world coordinate system and the included angle of the robotic arm.

[0037] Step 102: Encode the first training data using the encoder in the visual language action model to obtain the observed visual latent variables and future visual latent variables corresponding to the first sample video, the current state latent variables and action sequence latent variables corresponding to the sample action sequence, and the instruction text features corresponding to the sample action instructions.

[0038] In one possible implementation, the model architecture of the visual language action model includes an encoder and a diffusion transformer, wherein the encoder is used to encode the input data to obtain the corresponding token, and the diffusion transformer is used to perform attention calculation and diffusion processing based on the input token sequence to obtain the corresponding prediction result.

[0039] Optionally, the encoder encodes the first sample video to obtain observed visual latent variables and future visual latent variables. The observed visual latent variables characterize the current observed content of the actuator before executing the sample action command; for example, they can be obtained by encoding the first frame of the first sample video. The future visual latent variables characterize the latent image of the future visual scene predicted by the actuator based on the environmental scene when executing the sample action command; for example, they can be obtained by encoding the sequence of other video frames in the first sample video besides the first frame. The encoder encodes the sample action sequence to obtain current state latent variables and action sequence latent variables. The current state latent variables characterize the initial state of the actuator when executing the sample action command; for example, they can be obtained by encoding the state data of the first frame in the sample action sequence. The action sequence latent variables characterize the action sequence predicted by the actuator for executing the sample action command; for example, they can be obtained by encoding the action data of other action data in the sample action sequence besides the state data of the first frame.

[0040] Step 103: Input the observed visual latent variables, future visual latent variables, current state latent variables, action sequence latent variables, and instruction text features into the diffusion transformer in the visual language action model to obtain the prediction results for the future visual latent variables and action sequence latent variables.

[0041] The diffusion transformer is used to restrict the access of action sequence latent variables to future visual latent variables during the attention calculation process by using an attention mask.

[0042] Optionally, noise can be added to the future visual latent variables and action sequence latent variables, allowing the diffusion transformer to progressively predict the noise-added future visual latent variables and action sequence latent variables based on the observed visual latent variables, the current state latent variables, and the command text features. This yields the predicted results for the future visual latent variables and the action sequence latent variables (i.e., the velocity fields corresponding to the noise-added future visual latent variables and the velocity fields corresponding to the noise-added action sequence latent variables). During attention calculation, the model first generates a latent variable matrix based on the sequence of input observed visual latent variables, future visual latent variables, current state latent variables, action sequence latent variables, and command text features. Then, a causal attention mask is used to mask the latent variable matrix, preventing the masked action sequence latent variables from accessing the future visual latent variables, while allowing the future visual latent variables to access the action sequence latent variables. This enables the model to learn and understand the physical laws governing the visual transformations caused by actions.

[0043] In terms of model structure, this invention employs a single Transformer to uniformly process visual tokens, robot state tokens, action tokens, and future visual tokens. To ensure that future vision serves only as a supervisory signal for the physical consistency of actions during the training phase, and does not cause additional computation during the inference phase, this disclosure proposes a causal-based attention masking mechanism: action tokens (i.e., containers carrying latent variables of action sequences) are only allowed to access the current visual token and the current state token during training, and are prohibited from accessing any future visual tokens; future visual latents (i.e., future visual latent variables) can access action tokens to learn the causal relationship between actions and future visual changes. This mechanism ensures that the model's action prediction branch maintains a consistent internal dependency structure during training, regardless of the presence of future visual tokens. Therefore, even if the future visual prediction branch is directly deleted during the inference phase, the model can still function normally without structural damage or performance degradation.

[0044] Step 104: Update the model parameters of the diffusion converter based on the prediction results.

[0045] Specifically, based on the predicted results of the future visual latent variables and the predicted results of the action sequence latent variables, the loss function value is determined. Then, backpropagation is performed based on the function value to adjust the model parameters of the diffusion transformer. Steps 101 to 104 are repeated until at least one of the following conditions is met: the number of training iterations reaches a preset number, the training duration reaches a preset duration, or the loss function value is less than a preset value.

[0046] Based on the embodiments of this disclosure, a world-action modeling architecture centered on action is provided. During the model training phase, by introducing a causal-based attention mask in the diffusion transformer, access to future visual latent variables by action sequence latent variables is restricted. However, future visual latent variables can access action sequence latent variables. While introducing future visual prediction, this ensures that the action sequence prediction branch is independent of future visual information during training. This forces the model to learn the causal relationship between actions and environmental changes based on multimodal data, thereby constructing a denser and more physically consistent temporal supervision, improving the stability and physical reliability of the model's action prediction. During the inference phase, future visual latent variables and related computational paths can be directly removed, achieving decoupling between training and inference. This allows the model to stably output high-quality action sequences without relying on predicting future visual images during the inference phase. This fundamentally solves the problem of high inference costs and severe error accumulation caused by traditional models still needing to generate future visual images for action inference during the inference phase. It significantly reduces inference latency and improves model inference efficiency and the accuracy of robot action control.

[0047] In one possible implementation, such as Figure 2 As shown, step 103 above may specifically include the following steps: Step 201: Noise is added to the future visual latent variables and the action sequence latent variables respectively.

[0048] Step 202: The observed visual latent variables, current state latent variables, instruction text features, and noise-added future visual latent variables and action sequence latent variables are concatenated into an input sequence.

[0049] Optionally, random noise (such as Gaussian noise) can be used to add noise to the future visual latent variables and action sequence latent variables to obtain the noise-added results of the future visual latent variables and action sequence latent variables. Then, the observed visual latent variables, current state latent variables, command text features, and the noise-added future visual latent variables and action sequence latent variables are concatenated into the input sequence according to a preset order.

[0050] Optionally, after adding noise to the future visual latent variables and the action sequence latent variables, the first sample velocity field corresponding to the future visual latent variables and the second sample velocity field corresponding to the action sequence latent variables are calculated based on the noise addition results, so as to calculate the function value of the loss function in the subsequent calculation.

[0051] Step 203: Input the input sequence into the diffusion transformer, generate a feature matrix through the diffusion transformer, and perform masking and forward propagation on the feature matrix to obtain the first prediction result for the future visual latent variable and the second prediction result for the action sequence latent variable.

[0052] Optionally, after inputting the input sequence into the diffusion transformer, a feature matrix is ​​generated based on the observed visual latent variables, the noisy future visual latent variables, the current state latent variables, and the noisy action sequence latent variables in the input sequence. The feature matrix is ​​then masked, preventing latent variables in rows covered by the mask from accessing latent variables in their corresponding columns, while latent variables in rows not covered by the mask can access latent variables in their corresponding columns. Self-attention is calculated based on the masked feature matrix, and cross-attention is performed in conjunction with the instruction text features in the input sequence. Velocity field prediction is then performed progressively on the noisy future visual latent variables and the noisy action sequence latent variables, yielding a first prediction result and a second prediction result. The first prediction result includes the first predicted velocity field obtained by the model predicting the future visual latent variables, and the second prediction result includes the second predicted velocity field obtained by the model predicting the action sequence latent variables.

[0053] Indicative Figure 3 A schematic diagram of a feature matrix after masking is shown. For example... Figure 3 As shown, the latent variables in the matrix are ordered along both the row and column directions as follows: observed visual latent variable To, noisy future visual latent variable Tf, current state latent variable Ts, and noisy action sequence latent variable Ta. Light-colored matrix cells are those covered by a mask, while dark-colored matrix cells are those without a mask. Specifically, the observed visual latent variable To can access itself and the current state latent variable Ts; the noisy future visual latent variable Tf can access all latent variables; the current state latent variable Ts can access itself and the observed visual latent variable To; and the action sequence latent variable Ta can access itself, the observed visual latent variable To, and the current state latent variable Ts.

[0054] Accordingly, step 104 above may specifically include the following steps: The visual flow matching loss is determined based on the future visual latent variables and the first prediction result; the action flow matching loss is determined based on the action sequence latent variables and the second prediction result; the total flow matching loss is determined based on the visual flow matching loss and the action flow matching loss; backpropagation is performed based on the total flow matching loss to update the model parameters of the diffusion transformer.

[0055] Optionally, the visual flow matching loss can be obtained based on the first sample velocity field and the first predicted velocity field. For example, the mean square error between the first sample velocity field and the first predicted velocity field can be determined as the visual flow matching loss. The action flow matching loss can be obtained based on the second sample velocity field and the second predicted velocity field. The total flow matching loss can be the sum of the visual flow matching loss and the action flow matching loss.

[0056] In this embodiment, the model simultaneously predicts latent variables of action sequences and future visual variables during training, and both predictions use flow matching as a unified objective. By learning the velocity field of each token over continuous time, the model learns the temporal evolution law in a consistent manner. The prediction of future visual latent variables forces the model to generate action sequences that can drive reasonable changes in the environment, thereby forming an implicit physical constraint that ensures the strategy trained by the model has temporal consistency and physical rationality.

[0057] In one possible implementation, the two-stage training method of this disclosure first uses large-scale internet videos without action labels to perform general world dynamic pre-training on the model, enabling the model to learn the laws of object motion, dynamic relationships, and temporal characteristics, providing a basic world understanding ability for subsequent robot tasks. Then, using robot-related first sample data, the model gradually learns the spatial relationships from the robot's perspective, the dynamic characteristics of the manipulated objects, and the task structure. Figure 4 As shown, prior to step 101 above, the method provided in this embodiment may further include the following steps: Step 401: Obtain second training data, which includes second sample videos, including videos used to demonstrate the motion of objects.

[0058] In illustrative terms, the second sample video can be a video downloaded from the internet or a video captured using an image acquisition device. The video content of the second sample video can include, for example, the process of an object falling, the process of a human walking, or a human picking up an object, so that the model can learn basic physical laws.

[0059] Step 402: Encode the second training data using the encoder in the visual language action model to obtain the observed visual latent variables and future visual latent variables corresponding to the second sample video.

[0060] Optionally, a variational autoencoder is used to encode the second sample video. The observed visual latent variable is obtained by encoding the first frame of the second sample video, and the future visual latent variable is obtained by encoding the sequence of other video frames besides the first frame. In addition, the second training data may also include corresponding descriptive text, such as "a person is running".

[0061] Step 403: Input the observed visual latent variables and future visual latent variables corresponding to the second sample video into the diffusion transformer to pre-train the diffusion transformer.

[0062] Optionally, the diffusion transformer can also use a causal-based mask to mask the feature matrix during the first stage of pre-training. For example, it can restrict the observed visual latent variables from accessing future visual latent variables, while allowing future visual latent variables to access the observed visual latent variables.

[0063] Based on the embodiments of this disclosure, in order to solve the problem of how to make full use of large-scale video data to build a robot control model with broad generalization capabilities, a two-stage unified pre-training process is highlighted, which enables the model to learn basic physical laws from general videos and then learn task-related interaction patterns from robot videos.

[0064] In one possible implementation, the visual language action model includes different types of encoders, each used to encode different input data. Step 102 above may specifically include the following steps: Step 102a: Input the first sample video into the pre-trained Variational Auto-Encoder. The Variational Auto-Encoder encodes the first frame of the first sample video to obtain the observed visual latent variables, and encodes the remaining video frames of the first sample video to obtain the future visual latent variables.

[0065] The observed visual latent variable is used to characterize the current observed image, while the future visual latent variable corresponds to the future observed image after the actuator executes the action sequence. Therefore, the first frame of the first sample video can be determined as the current observed image. The observed visual latent variable is obtained by encoding the first frame, and the future visual latent variable is obtained by encoding the remaining video frames.

[0066] Step 102b: Input the sample action sequence into the pre-trained State / ActionEncoder. The State / ActionEncoder encodes the initial state data in the sample action sequence to obtain the current state latent variable, and encodes the action sequence data in the sample action sequence to obtain the action sequence latent variable.

[0067] Optionally, the current state latent variable is used to characterize the current state of the executor, while the action sequence latent variable corresponds to the action sequence predicted by the executor to complete the instruction. Therefore, the initial state data of the sample action sequence can be encoded to obtain the current state latent variable, and the action sequence data in the sample action sequence can be encoded to obtain the action sequence latent variable. The action sequence data corresponds to the video frames of the first sample video; for example, each video frame corresponds to four sets of action sequence data.

[0068] Step 102c: Input the sample action command into the text encoder, and encode the sample action command through the text encoder to obtain the command text feature.

[0069] In combination with the above embodiments, Figure 5 A schematic diagram of a model training process is shown. (For example...) Figure 5 As shown, In the pre-training phase, the video frames of the second sample video are encoded using a VAE Encoder to obtain the observed visual latent variable To and the future visual latent variable Tf. The future visual latent variable Tf is then noise-added and input together with the observed visual latent variable To into the Casual DiT Blocks diffusion transformer. Based on the prediction results of the output future visual latent variable, the loss function value Loss is calculated, and the model parameters are updated. In the second training phase, the post-training phase, the video frames of the first sample video are encoded using a VAE Encoder to obtain the observed visual latent variable To and the future visual latent variable Tf. The sample action sequence is encoded using a State / Action Encoder to obtain the current state latent variable Ts and the action sequence latent variable Ta. The future visual latent variable Tf and the action sequence latent variable Ta are then noise-added and input together with other latent variables and the instruction text feature Tt into the Casual DiTBlocks. Based on the prediction results of the output future visual latent variable and the prediction results of the action sequence latent variable, the loss function value Loss is calculated, and the model parameters are updated.

[0070] On the other hand, this disclosure also provides an action generation method applied to an executor, wherein the executor is equipped with a VLA model trained using the model training method corresponding to any of the above embodiments. Figure 6 As shown, the method includes the following steps: Step 601: Obtain the action command and the current observation video and current status data of the executor.

[0071] After the VLA model is trained, it can be deployed in actuators such as robots, or in devices such as servers that communicate with the actuators. Upon receiving a user's action command, the actuator acquires its current observation video and current state data. The current observation video can be video from at least one perspective captured by a video acquisition device on the actuator after receiving the action command, and the current state data includes the actuator's own state data acquired when it received the action command.

[0072] Step 602: The encoder in the visual language action model encodes the action command, the currently observed video, and the current state data respectively to obtain the command text features, the observed visual latent variables, and the current state latent variables.

[0073] The VLA model is trained using the model training methods described in the embodiments above. The encoder includes a variational autoencoder, a state-action encoder, and a text encoder. The variational autoencoder encodes the currently observed video to obtain the observed visual latent variables, the state-action encoder encodes the current state data to obtain the current state latent variables, and the text encoder encodes the action commands to obtain the command text features.

[0074] Step 603: Input the input data into the diffusion transformer of the visual language action model to obtain the action sequence latent variables output by the diffusion transformer.

[0075] The input data includes instruction text features, observed visual latent variables, current state latent variables, and action sequence latent variables to be predicted.

[0076] The latent variables of the action sequence to be predicted can be random noise, such as Gaussian noise. A diffusion transformer infers the latent variables of the action sequence to be predicted based on the command text features, observed visual latent variables, and current state latent variables, gradually predicting the corresponding velocity field, thus obtaining the predicted action sequence latent variables.

[0077] Step 604: Decode the latent variables of the action sequence into the action sequence of the executor.

[0078] After obtaining the latent variables of the action sequence output by the diffusion transformer, the decoder is used to decode the latent variables of the action sequence to obtain the action sequence that the actuator can recognize, such as data such as the direction of movement and distance of movement of each movable part of the actuator at each time point.

[0079] Based on the embodiments of this disclosure, a world-action modeling architecture centered on action is provided. During the model training phase, by introducing a causal-based attention mask in the diffusion transformer, access to future visual latent variables by action sequence latent variables is restricted. However, future visual latent variables can access action sequence latent variables. While introducing future visual prediction, this ensures that the action sequence prediction branch is independent of future visual information during training. This forces the model to learn the causal relationship between actions and environmental changes based on multimodal data, thereby constructing a denser and more physically consistent temporal supervision, improving the stability and physical reliability of the model's action prediction. During the inference phase, future visual latent variables and related computational paths can be directly removed, achieving decoupling between training and inference. This allows the model to stably output high-quality action sequences without relying on predicting future visual images during the inference phase. This fundamentally solves the problem of high inference costs and severe error accumulation caused by traditional models still needing to generate future visual images for action inference during the inference phase. It significantly reduces inference latency and improves model inference efficiency and the accuracy of robot action control.

[0080] In one possible implementation, the model can perform pure action reasoning or reasoning about future visual latent variables. That is, the input data can also include the future visual latent variables to be predicted. Video-action joint generation is then performed to obtain corresponding predicted videos, which are used to verify the rationality of the actions or generate demonstration data, etc. Step 603 above may include the following steps: Step 603a: Input the instruction text features, observed visual latent variables, current state latent variables, action sequence latent variables to be predicted, and future visual latent variables to be predicted into the diffusion transformer.

[0081] Step 603b involves performing a diffusion transformation on the latent variables of the action sequence to be predicted using a diffusion transformer, and using a key-value cache (KV Cache) in the diffusion transformer to denoise the latent variables of the future vision to be predicted, thereby obtaining the latent variables of the action sequence and the latent variables of the future vision output by the diffusion transformer.

[0082] Accordingly, the method provided in this disclosure may further include the following steps: Decode future visual latent variables into predictive videos.

[0083] Indicative Figure 7 A schematic diagram of a model reasoning process is shown. For example... Figure 7 As shown, when the selected input is the future visual latent variable Tf to be predicted, a key-value cache is set in the model to cache the key values ​​of tokens such as instruction text features, observed visual latent variables, current state latent variables, and predicted action sequence latent variables. The generation process of the future visual latent variable can access other latent variables and features, consistent with the setting during training where the future visual latent variable can access the action sequence latent variable. Simultaneously, because the causal mask still exists, the action sequence latent variable still cannot access the future visual latent variable, thus maintaining the independence of action generation.

[0084] Please refer to Figure 8 This illustration shows a structural block diagram of a model training apparatus provided in an exemplary embodiment of the present disclosure. The model training apparatus provided in this embodiment includes: The first acquisition module 801 is used to acquire first training data, which includes sample action instructions, and first sample video and sample action sequence when the executor executes the sample action instructions; The first encoding module 802 is used to encode the first training data through the encoder in the visual language action model to obtain the observed visual latent variables and future visual latent variables corresponding to the first sample video, the current state latent variables and action sequence latent variables corresponding to the sample action sequence, and the instruction text features corresponding to the sample action instructions. The latent variable prediction module 803 is used to input the observed visual latent variables, future visual latent variables, current state latent variables, action sequence latent variables and instruction text features encoded by the first encoding module 802 into the diffusion transformer in the visual language action model to obtain the prediction results for the future visual latent variables and action sequence latent variables. The diffusion transformer is used to restrict the action sequence latent variables from accessing the future visual latent variables during the attention calculation process by using an attention mask. The model training module 804 is used to update the model parameters of the diffusion converter based on the prediction results obtained from the latent variable prediction module 803.

[0085] Optionally, the latent variable prediction module 803 described above can be further used for: Noise was added to both the future visual latent variables and the action sequence latent variables. The observed visual latent variables, current state latent variables, instruction text features, and noise-added future visual latent variables and action sequence latent variables are concatenated into the input sequence. The input sequence is fed into a diffusion transformer, which generates a feature matrix. The feature matrix is ​​then masked and forward-propagated to obtain a first prediction result for the future visual latent variable and a second prediction result for the action sequence latent variable.

[0086] Optionally, the model training module 804 described above can be further used for: The visual flow matching loss is determined based on future visual latent variables and the first prediction result. The action flow matching loss is determined based on the latent variables of the action sequence and the second prediction result. The total flow matching loss is determined based on visual flow matching loss and action flow matching loss; Backpropagation is performed based on the total loss of flow matching to update the model parameters of the diffusion transformer.

[0087] Optionally, the apparatus provided in this disclosure may further include: The third acquisition module is used to acquire the second training data, which includes the second sample video, and the second sample video includes a video used to show the motion process of the object. The third encoding module is used to encode the second training data through the encoder in the visual language action model to obtain the observed visual latent variables and future visual latent variables corresponding to the second sample video. The pre-training module is used to input the observed visual latent variables and future visual latent variables corresponding to the second sample video into the diffusion transformer to pre-train the diffusion transformer.

[0088] Optionally, the first encoding module 802 described above can be further used for: The first sample video is input into a pre-trained variational autoencoder. The first frame of the first sample video is encoded by the variational autoencoder to obtain the observed visual latent variables, and the remaining video frames of the first sample video are encoded to obtain the future visual latent variables. The sample action sequence is input into the pre-trained state-action encoder. The state-action encoder encodes the initial state data in the sample action sequence to obtain the current state latent variable, and encodes the action sequence data in the sample action sequence to obtain the action sequence latent variable. The sample action command is input into the text encoder, and the text encoder encodes the sample action command to obtain the command text feature.

[0089] Please refer to Figure 9 This illustration shows a structural block diagram of an action generation apparatus provided in an exemplary embodiment of the present disclosure. The action generation apparatus provided in this embodiment includes: The second acquisition module 901 is used to acquire action commands as well as the current observation video and current status data of the executor; The second encoding module 902 is used to encode the action instructions, the current observation video and the current state data obtained by the second acquisition module 901 through the encoder in the visual language action model, so as to obtain instruction text features, observation visual latent variables and current state latent variables. The visual language action model is trained using the model training method in any of the above embodiments. The latent variable reasoning module 903 is used to input the input data encoded by the second encoding module 902 into the diffusion transformer of the visual language action model to obtain the action sequence latent variable output by the diffusion transformer. The input data includes instruction text features, observation visual latent variable, current state latent variable and action sequence latent variable to be predicted. The first decoding module 904 is used to decode the latent variable action sequence output by the latent variable reasoning module 903 into the action sequence of the executor.

[0090] Optionally, the input data also includes future visual latent variables to be predicted, and the aforementioned latent variable inference module 903 can be further used for: Input the instruction text features, observed visual latent variables, current state latent variables, action sequence latent variables to be predicted, and future visual latent variables to be predicted into the diffusion transformer; The action sequence latent variable to be predicted is obtained by performing a diffusion transformation on the latent variable to be predicted using a diffusion transformer, and by using the key-value buffer in the diffusion transformer to denoise the future visual latent variable to be predicted. The motion generation device may also include: The second decoding module is used to decode future visual latent variables into predictive videos.

[0091] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar, identical, or corresponding parts between embodiments can be referred to mutually. Since the method, apparatus, and device embodiments are basically corresponding, relevant parts can be referred to the corresponding descriptions. The methods, apparatus, and devices in the embodiments of this disclosure also correspond to each other in specific implementation and beneficial technical effects; related content can be referred to mutually and will not be repeated here.

[0092] In addition, this disclosure also provides an electronic device, including: Memory, used to store computer programs; A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the model training method or action generation method described in any of the above embodiments of the present disclosure.

[0093] Figure 10 This is a schematic diagram illustrating the structure of an application embodiment of the electronic device disclosed herein. Below, reference is made to… Figure 10 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0094] like Figure 10 As shown, the electronic device includes one or more processors and memory.

[0095] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0096] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the model training methods or action generation methods of the various embodiments of this disclosure described above, and / or other desired functions.

[0097] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0098] In addition, the input device may include, for example, a keyboard, a mouse, etc.

[0099] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0100] Of course, for the sake of simplicity, Figure 10 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0101] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the model training method or action generation method according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0102] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0103] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the model training method or action generation method according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0104] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0105] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0106] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0107] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0108] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0109] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0110] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0111] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0112] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A model training method, characterized in that, include: Acquire first training data, which includes sample action instructions, and a first sample video and sample action sequence when the executor executes the sample action instructions; The first training data is encoded by the encoder in the visual language action model to obtain the observed visual latent variables and future visual latent variables corresponding to the first sample video, the current state latent variables and action sequence latent variables corresponding to the sample action sequence, and the instruction text features corresponding to the sample action instruction. The observed visual latent variable, the future visual latent variable, the current state latent variable, the action sequence latent variable, and the instruction text feature are input into the diffusion transformer in the visual language action model to obtain the prediction results for the future visual latent variable and the action sequence latent variable. The diffusion transformer is used to restrict the action sequence latent variable from accessing the future visual latent variable during the attention calculation process by using an attention mask. The model parameters of the diffusion converter are updated based on the prediction results.

2. The method according to claim 1, characterized in that, The step of inputting the observed visual latent variable, the future visual latent variable, the current state latent variable, the action sequence latent variable, and the instruction text features into the diffusion transformer in the visual language action model to obtain prediction results for the future visual latent variable and the action sequence latent variable includes: Noise is added to both the future visual latent variable and the action sequence latent variable; The observed visual latent variables, the current state latent variables, the instruction text features, the noisy future visual latent variables, and the action sequence latent variables are concatenated into an input sequence. The input sequence is input into the diffusion transformer, which generates a feature matrix and performs masking and forward propagation on the feature matrix to obtain a first prediction result for the future visual latent variable and a second prediction result for the action sequence latent variable.

3. The method according to claim 2, characterized in that, Updating the model parameters of the diffusion converter based on the prediction results includes: The visual flow matching loss is determined based on the future visual latent variables and the first prediction result; The action flow matching loss is determined based on the latent variables of the action sequence and the second prediction result; The total flow matching loss is determined based on the visual flow matching loss and the action flow matching loss. Backpropagation is performed based on the total flow matching loss to update the model parameters of the diffusion transformer.

4. The method according to any one of claims 1 to 3, characterized in that, Before acquiring the first training data, the process also includes: Acquire second training data, which includes second sample videos, which include videos used to demonstrate the motion of objects; The second training data is encoded by the encoder in the visual language action model to obtain the observed visual latent variables and future visual latent variables corresponding to the second sample video. The observed visual latent variables and future visual latent variables corresponding to the second sample video are input into the diffusion transformer to pre-train the diffusion transformer.

5. The method according to any one of claims 1 to 3, characterized in that, The first training data is encoded using an encoder in the visual language action model to obtain the observed visual latent variables and future visual latent variables corresponding to the first sample video, the current state latent variables and action sequence latent variables corresponding to the sample action sequence, and the instruction text features corresponding to the sample action command, including: The first sample video is input into a pre-trained variational autoencoder. The first frame of the first sample video is encoded by the variational autoencoder to obtain the observed visual latent variable, and the remaining video frames of the first sample video are encoded to obtain the future visual latent variable. The sample action sequence is input into a pre-trained state action encoder. The state action encoder encodes the initial state data in the sample action sequence to obtain the current state latent variable, and encodes the action sequence data in the sample action sequence to obtain the action sequence latent variable. The sample action command is input into a text encoder, and the text encoder encodes the sample action command to obtain the command text feature.

6. A method for generating actions, characterized in that, include: Acquire action commands, as well as the current observation video and current status data of the actuator; The action command, the currently observed video, and the current state data are encoded by the encoder in the visual language action model to obtain command text features, observed visual latent variables, and current state latent variables. The visual language action model is trained using the model training method described in any one of claims 1 to 5 above. The input data is input into the diffusion transformer of the visual language action model to obtain the action sequence latent variable output by the diffusion transformer. The input data includes the instruction text features, the observed visual latent variable, the current state latent variable, and the action sequence latent variable to be predicted. The latent variables of the action sequence are decoded into the action sequence of the actuator.

7. The method according to claim 6, characterized in that, The input data also includes future visual latent variables to be predicted. The step of inputting the input data into the diffusion transformer of the visual language action model to obtain the action sequence latent variables output by the diffusion transformer includes: The instruction text features, the observed visual latent variables, the current state latent variables, the action sequence latent variables to be predicted, and the future visual latent variables to be predicted are input into the diffusion transformer. The action sequence latent variable to be predicted is diffused through the diffusion transformer, and the future visual latent variable to be predicted is denoised using the key-value cache in the diffusion transformer, so as to obtain the action sequence latent variable and the future visual latent variable output by the diffusion transformer. The method further includes: The future visual latent variables are decoded into predictive videos.

8. A model training device, characterized in that, include: The first acquisition module is used to acquire first training data, the first training data including sample action instructions, and a first sample video and sample action sequence when the executor executes the sample action instructions; The first encoding module is used to encode the first training data through the encoder in the visual language action model to obtain the observed visual latent variable and future visual latent variable corresponding to the first sample video, the current state latent variable and action sequence latent variable corresponding to the sample action sequence, and the instruction text feature corresponding to the sample action instruction. The latent variable prediction module is used to input the observed visual latent variable, the future visual latent variable, the current state latent variable, the action sequence latent variable, and the instruction text features into the diffusion transformer in the visual language action model to obtain the prediction results for the future visual latent variable and the action sequence latent variable. The diffusion transformer is used to restrict the action sequence latent variable from accessing the future visual latent variable during the attention calculation process by using an attention mask. The model training module is used to update the model parameters of the diffusion converter based on the prediction results.

9. An action generation device, characterized in that, include: The second acquisition module is used to acquire action commands as well as the current observation video and current status data of the executor; The second encoding module is used to encode the action instruction, the currently observed video, and the current state data respectively through the encoder in the visual language action model to obtain instruction text features, observed visual latent variables, and current state latent variables. The visual language action model is trained using the model training method as described in any one of claims 1 to 5 above. The latent variable inference module is used to input input data into the diffusion transformer of the visual language action model to obtain the action sequence latent variable output by the diffusion transformer. The input data includes the instruction text features, the observed visual latent variable, the current state latent variable, and the action sequence latent variable to be predicted. The first decoding module is used to decode the latent variables of the action sequence into the action sequence of the actuator.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program stored in the memory, wherein when the computer program is executed, it implements the method of any one of claims 1-5 or the method of any one of claims 6-7.

Images