A video world model training method and device based on object condition attention calibration

Abstract

The application provides a video world model training method and device based on object condition attention calibration, acquires a video sample and a condition sequence sample input model, generates a noisy video latent variable and a speed objective function, extracts a related word element of an operated object in an instruction, calculates an initial cross-attention matrix and corrects the same based on the related word element to generate an object condition attention distribution, generates an interaction graph according to the distribution, calculates a supervision weight based on the interaction graph and weights the speed objective function to obtain a supervision function, and updates parameters in different ranges respectively through multiple back propagations to cover all model parameters. The application reuses an initial cross-attention generated by an original video world model to generate an attention matrix of a guided model learning an operated object, does not need external dense labeling such as optical flow, decouples parameter updating through multiple back propagations to eliminate self-reference dependence, improves the accuracy of a video generated by prediction in estimating an operated object, and does not increase additional calculation burden.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method and apparatus for training a video world model based on object-conditional attention calibration. Background Technology

[0002] In embodied intelligence control scenarios, it is often necessary to predict the state changes of various objects in space at the next moment using video world models to guide the embodied intelligence in making accurate decisions and controls. Video world models are commonly used to generate predictive video information, providing material support for the simulation, planning, policy learning, and policy evaluation of embodied intelligence. Video world models can predict the future visual state of the environment based on initial observations, language tasks, and action control conditions. With the development of large-scale pre-trained video generation models, video world models, represented by video diffusion models, can now generate videos with high visual clarity and a certain degree of temporal coherence. However, because embodied intelligence will engage in local interactions with manipulated objects in space, such as grasping, contacting, pushing, stacking, and inserting, in embodied application scenarios, the videos predicted by video world models not only require a natural overall picture, but also require that the actions of the embodied intelligence and the changing states of the manipulated objects conform to the action conditions and physical relationships.

[0003] Existing techniques for training video diffusion models typically average the prediction errors across all potential spatiotemporal locations. This training method means that a region's contribution to parameter optimization depends primarily on its area, rather than its importance to the task. However, in embodied manipulation videos, non-manipulated objects such as desktops, walls, and stationary objects usually occupy a large portion of the frame, while motion-condition-changing areas, such as the manipulated object, end effector, and its contact area, occupy only a small portion. Therefore, the aforementioned method of calculating prediction errors allows the model to reduce global loss by learning from large areas of non-manipulated object content, while sparse motion-condition-changing areas may still not be sufficiently trained. The allocation of supervision in the spatiotemporal locations during training is mismatched with the importance of the interactive task, leading to problems such as deformed predicted model objects, discontinuous motion, weak action coupling, and inconsistent contact relationships.

[0004] To enhance a model's learning of content within regions of varying motion conditions, existing techniques typically introduce spatiotemporally dense representations during training or generation, such as optical flow, point trajectories, depth maps, surface normals, 3D reconstruction results, hand meshes, or manually segmented annotations, to increase the information content and feature proportion of these regions. However, the generation of these spatiotemporally dense representations usually relies on additional estimation models, geometric processing procedures, or manual verification. As the number and duration of training videos increase, the representations introduced during training lead to a corresponding increase in preprocessing overhead, storage overhead, and error propagation risk, hindering scalability on large-scale heterogeneous training data. Furthermore, the acquisition methods of external representations often correspond to specific object categories, camera configurations, or control methods, making it difficult to reuse spatiotemporally dense representations across different interaction scenarios, such as robotic arms and human hands. Summary of the Invention

[0005] In view of this, this application provides a video world model training method and apparatus based on object conditional attention calibration, which solves the problem of how to guide the video world model to focus on the relevant region of the manipulated object without relying on external dense representation and without changing the inference architecture of the video world model, so as to improve the accuracy of the model's prediction of the video of the embodied agent's related actions.

[0006] Specifically, this application is implemented through the following technical solution:

[0007] The first aspect of this application provides a method for training a video world model based on object conditional attention calibration, the method comprising:

[0008] Video samples and conditional sequence samples are input into the video world model. The conditional sequence samples include at least task text instructions, which at least describe the object being manipulated by the embodied intelligence in space.

[0009] The video world model generates noisy video latent variables based on the video samples, and generates a velocity objective function based on the noisy video latent variables and clean latent variables.

[0010] Extract related words of the operated object based on the task text instructions;

[0011] Calculate an initial cross-attention matrix, modify the initial cross-attention matrix based on the associated lexical units, and generate an object conditional attention distribution. Compared with the initial cross-attention matrix, the attention score of the manipulated object is increased, and the attention scores of objects other than the manipulated object in the space are decreased.

[0012] Generate an interaction graph based on the object's conditional attention distribution;

[0013] The supervision weights at each location in the space are calculated based on the interaction graph, and the velocity objective function is weighted based on the supervision weights to obtain the supervision function;

[0014] Multiple backpropagations are performed based on the supervision function, with each backpropagation updating a different range of parameters. The set of parameters updated by all backpropagations is the set of all parameters of the video world model.

[0015] In a preferred embodiment of this application, the initial cross-attention matrix includes at least multiple attention heads and multiple tile attentions; the calculation of the initial cross-attention matrix, the modification of the initial cross-attention matrix based on the associated lexical units, and the generation of object conditional attention distributions include:

[0016] Using video location as the query and the corresponding word of the task text instruction as the key, calculate the original cross-attention distribution of single head and single block to obtain the initial cross-attention matrix;

[0017] Based on the associated lexical units, the target attention probabilities corresponding to the associated lexical units are selected from the initial cross-attention matrix;

[0018] The target attention probabilities are aggregated to generate the object conditional attention distribution.

[0019] In another preferred embodiment of this application, the step of extracting related terms of the operated object based on the task text instruction includes:

[0020] The large language model is frozen by inputting the task text instructions as parameters.

[0021] The large language model locates the associated text of the manipulated object from the task text instructions;

[0022] Align the associated text with the task text instruction, and determine the associated words based on the aligned index. The associated words contain only text information related to the object being operated on.

[0023] In another preferred embodiment of this application, generating an interaction graph based on the object conditional attention distribution includes:

[0024] The gradient is stopped for the conditional attention distribution of the object, and the log odds are calculated; a smooth perturbation is calculated on a coarse spatiotemporal grid, and candidate responses are generated based on the smooth perturbation and the log odds;

[0025] Multiple candidate regions are generated from the candidate responses based on the coverage ratio;

[0026] The candidate weights are determined by the average of the candidate regions where the gradient error stops within the plurality of candidate regions.

[0027] An interaction graph is calculated based on the candidate weights and multiple candidate regions.

[0028] In another preferred embodiment of this application, the step of calculating the supervision weights at each location in the space based on the interaction graph includes:

[0029] Traverse each spatiotemporal location in the space, calculate the first weight based on the first information in the interaction graph of the current spatiotemporal location, calculate the supervision weight of the current spatiotemporal location based on the product of the first weight and the enhancement factor, the supervision weight of the spatiotemporal location where the manipulated object is located is greater than 1, and the supervision weight of all spatiotemporal locations in the space other than the manipulated object is less than 1.

[0030] In another preferred embodiment of this application, the step of weighting the velocity objective function based on the supervision weights to obtain the supervision function includes:

[0031] The supervisory weights are used to calculate the weighted sum of the spatiotemporal position function values ​​in the velocity objective function;

[0032] The supervision function is obtained by normalizing the weighted sum based on the sum of the supervision weights.

[0033] In another preferred embodiment of this application, the backpropagation includes at least backpropagation for updating attention parameters, and multiple backpropagations based on the supervision function, including:

[0034] Preserve the global target gradient across attention parameters;

[0035] Suppress the parameter gradients of the video world model other than the cross-attention parameters;

[0036] The backpropagation is performed based on the global target gradient to update the cross-attention parameters.

[0037] In another preferred embodiment of this application, the backpropagation includes at least backpropagation for updating model parameters other than the cross-attention parameters, and multiple backpropagations based on the supervision function, including:

[0038] Preserve the interactive weighted gradients of model parameters other than those across attention parameters;

[0039] Suppress the parameter gradients across attention parameters;

[0040] The backpropagation is performed based on the interaction-weighted gradient to update the model parameters in the video world model other than the attention parameters.

[0041] In another preferred embodiment of this application, after the video world model has been trained, the method further includes:

[0042] Input the task instruction text, reference observations, and control signals into the trained video world model;

[0043] The video world model generates full-information encoding of the task instruction text and initial video latent variables;

[0044] Based on the full information encoding and the initial video latent variables, cross-attention is calculated to generate denoised video latent variables;

[0045] Decode the latent variables of the denoised video to generate a predicted video.

[0046] A second aspect of this application provides a video world model training apparatus based on conditional attention calibration, the apparatus comprising:

[0047] The acquisition module is used to acquire video samples and conditional sequence samples and input them into the video world model. The conditional sequence samples include at least task text instructions, which at least describe the object being manipulated by the embodied intelligence in space.

[0048] The generation module is used for the video world model to generate noisy video latent variables based on the video samples, and to generate a velocity objective function based on the noisy video latent variables and the clean latent variables;

[0049] The extraction module is used to extract related words of the operated object based on the task text instructions;

[0050] The calculation module is used to calculate an initial cross-attention matrix, modify the initial cross-attention matrix based on the associated lexical units, and generate an object conditional attention distribution. Compared with the initial cross-attention matrix, the object conditional attention distribution increases the attention score of the manipulated object and decreases the attention score of objects other than the manipulated object in the space.

[0051] The generation module is used to generate an interaction graph based on the object's conditional attention distribution;

[0052] The generation module is used to calculate the supervision weights of each position in the space based on the interaction graph, and to weight the velocity objective function based on the supervision weights to obtain the supervision function;

[0053] The update module is used to perform multiple backpropagations based on the supervision function. The parameter range updated in each backpropagation is different, and the set of parameters updated by all backpropagations is all the parameters of the video world model.

[0054] This application provides a video world model training method and apparatus based on object-conditional attention calibration. During the training process of the video world model, conditional attention focused on the manipulated object is generated by reusing the initial cross-attention matrix generated by the model itself. This conditional attention information is then represented in the form of an interaction graph. Simultaneously, the velocity objective function from the model's own training process is reused to generate supervisory guidance information for video world model training, including both the interaction graph and the supervisory function. This allows the video world model to primarily learn content related to the manipulated object, reducing the learning of non-manipulated objects such as large static areas. Without adjusting the inference structure of the video world model or introducing new complex algorithms to compute any external dense representations, it can still maintain compatibility with existing video world models. Furthermore, it improves the accuracy of the video world model in predicting changes in the manipulated object, enhances its video prediction capabilities, and improves its adaptability to embodied intelligence application scenarios. Moreover, the entire process does not recalculate the attention matrix, fully reusing the initial cross-attention matrix generated by the video world model itself, significantly reducing the time consumption for large-scale training of the video world model. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating a video world model training method based on object conditional attention calibration according to an exemplary embodiment;

[0056] Figure 2 This is a block diagram illustrating a video world model training apparatus based on object conditional attention calibration according to an exemplary embodiment;

[0057] Figure 3 This is a block diagram of an electronic device illustrating a video world model training method based on object-conditional attention calibration according to an exemplary embodiment. Detailed Implementation

[0058] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0059] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0060] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0061] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0062] Example 1:

[0063] Figure 1 For a flowchart of a video world model training method based on object conditional attention calibration provided in this application, please refer to [the flowchart]. Figure 1 The method provided in this embodiment includes steps S101 to S107.

[0064] Step S101: Obtain video samples and conditional sequence samples and input them into the video world model. The conditional sequence samples include at least task text instructions, which at least describe the embodied intelligence as the manipulated object in space.

[0065] Specifically, embodied intelligence refers to an intelligent agent that performs specific operations through automatic control within a space, such as a robot. Video samples refer to real or simulated videos of a preset duration containing the space where the embodied intelligence resides, such as videos of a robot folding clothes with its arms or a human hand grasping an object. The video sample includes multiple objects, which are at least divided into manipulated objects and non-manipulated objects. Manipulated objects are those that interact with the embodied intelligence's actions, such as a robotic arm, folded clothing, or a grasped object. Non-manipulated objects are static objects in the video, such as a wall or a sofa that cannot be sat on. Conditional sequence samples refer to the conditional sequences paired with video samples and input into the video world model. Specifically, they are the basic information guiding the training of the video world model. The conditional sequence samples at least include task text instructions, control signals, and reference observations. The task text instruction is a string of commands used to describe the video generation task of the video world model. Further, the task text instruction at least describes the object being manipulated by the embodied intelligence in space. Optionally, the text may include the object being manipulated, such as "blue bowl" or "robotic arm," and may also include the action, such as "grab," and the target state or position, such as "place on the white table." As an optional embodiment, the video world model at least includes a video diffusion model. The training of the video world model uses a stream matching framework, and denoising training is performed in the latent variable space encoded by a variational autoencoder. The input of the trained video world model is the task instruction text, reference observations, and control signals, and the output is the predicted video. Preferably, the embodied intelligence can be a robotic arm, a humanoid robot, a dexterous hand, or other intelligent agents with manipulative capabilities.

[0066] Understandably, this application provides explicit semantic guidance for the video world model by acquiring conditional sequence samples containing task text instructions, providing basic information for the subsequent training process regulation, enabling the model to learn the physical evolution of environmental objects under specific instruction drive; pairing video samples with conditional sequence samples as input provides a training basis for subsequent interactive region supervision based on object conditional attention calibration.

[0067] As an optional embodiment, the video world model is a video diffusion model, including at least an encoder module and a DiT module. The encoder module is used to encode input variables and includes at least a text encoder, a variational autoencoder (VAE), and a control encoder. The text encoder is used to encode the filtered text related to the operated object in the task text instructions. The variational autoencoder (VAE) is used to generate clean latent variables of video samples and generate noisy video latent variables based on random noise. As an optional embodiment, after obtaining video samples and conditional sequence samples as input to the video world model, the method further includes: generating clean latent variables of the input video samples based on the variational autoencoder. At this time, the parameters of the variational autoencoder are frozen, and the video samples are encoded into clean latent variables with four dimensions: channel, latent time, height, and width by the frozen variational autoencoder. The freezing refers to not updating the weights of that part during training, i.e., not updating the weights of the variational autoencoder. The control encoder is used to encode the control signals of embodied intelligence. The DiT module includes a self-attention module, a cross-attention module, and a feedforward network (FFN). The DiT module predicts the motion velocity field, calculates the supervision function of the global spatiotemporal position error, and updates the parameters through multiple backpropagations.

[0068] Step S102: The video world model generates noisy video latent variables based on the video samples, and generates a velocity objective function based on the noisy video latent variables and the clean latent variables.

[0069] Specifically, the clean latent variables are the latent variables obtained by encoding video samples before adding noise, while the noisy video latent variables are the manifestation of Gaussian noise mixed into the clean latent variables according to noise scheduling. The velocity objective function is the global loss under the flow matching framework, used to measure the difference between the denoising direction predicted by the model and the actual denoising direction.

[0070] The following is a specific embodiment to illustrate the implementation process of this step in detail:

[0071] (1) Constructing noisy video latent variables based on stream matching.

[0072] Specifically, at time step t, Gaussian noise ε is randomly sampled and analyzed according to noise level coefficients. Linear combination of clean latent variables and Gaussian noise in [0,1]:

[0073] ;

[0074] in, For noisy latent variables, For clean latent variables, To control the noise mixing ratio, when When approaching 0, Approximately clean latent variables, when Approaching 1 o'clock Approximately pure noise.

[0075] (2) The video world model obtains velocity field prediction based on noisy video latent variables, time steps and conditional sequence samples.

[0076] Specifically, the DiT module of the video world model uses noisy latent variables. The diffusion time step t and the conditional sequence sample c are used as inputs to output a predicted velocity field. The implementation process of the DiT module is based on the prediction process of existing video diffusion models, and will not be elaborated upon here.

[0077] (3) Calculate the velocity objective function based on velocity field prediction and actual velocity.

[0078] Specifically, the true velocity field variable is calculated by the difference between the noise and the clean latent variable. The position-by-position prediction error is constructed based on the velocity field prediction and the actual velocity:

[0079] ;

[0080] in, Let C be the prediction error at spatiotemporal location p, and C be the channel number normalization factor.

[0081] Based on the prediction errors at all spatiotemporal locations, a velocity objective function is constructed:

[0082] ;

[0083] in, The velocity objective function is essentially a global monitoring objective. At this point, adjustments have not yet been made based on the associated text of the manipulated object. The mean of the prediction error for all spatiotemporal locations is calculated. It is the set of all spatiotemporal locations.

[0084] Understandably, by constructing noisy latent variables and velocity objective functions through the flow matching framework, a basic supervisory signal covering the entire spatiotemporal range is provided for the video world model; by using task text instructions and action control signals as conditional inputs, the model can learn how environmental objects should physically evolve in the global spatiotemporal range under the drive of specific instructions and actions.

[0085] Step S103: Extract the associated words of the operated object based on the task text instructions.

[0086] Specifically, related lexical units refer to the set of lexical units selected from the sequence of task text instructions that specifically describe the manipulated object, such as "blue bowl" or "red bowl". In other words, they are lexical units in the region of action condition change. Specifically, they are the set of lexical units of the object that has changed due to the embodied intelligence operation, and do not include words that are unrelated to the manipulated object, such as background environment, and remain in a static state.

[0087] The following is a specific example to illustrate the process of extracting related terms:

[0088] (1) Freeze the large language model of the task text instruction input parameters.

[0089] Specifically, the large language model can be a pre-trained language model such as Qwen2.5-0.5B, LLaMA, or GPT. Parameter freezing means that the weights of the large language model are not updated during the training of the video world model, and only its existing semantic understanding capabilities are used to extract lexical features.

[0090] (2) The large language model locates the associated text of the operated object from the task text instruction.

[0091] In a specific implementation, for example, in one embodiment, the task text instruction is "Place the blue bowl on the white table." The large language model identifies "blue bowl" as the phrase to be manipulated, "place on the white table" as the target state or target position, and "place" and "on the white table" as action relation words. Specifically, in one embodiment, the video sample can be 7002 simulation sequences of dual-arm robot operations, with each frame accompanied by 14-dimensional joint angle movements; the task text instruction is segmented and then input into a frozen Qwen2.5-0.5B language model to extract semantic features. Optionally, the task text instruction can be "Place the blue bowl on the white table," and the embodied intelligence corresponding to the task text instruction is a dual-arm robot, with the object to be manipulated by the dual-arm robot in space being "blue bowl."

[0092] (3) Align the associated text with the task text instruction, and determine the associated terms based on the aligned index.

[0093] Specifically, a sequence of instruction lexical terms is generated from all the text of the task text instruction. The associated text located by the large language model is aligned with the sequence of instruction lexical terms of the task text instruction. After alignment, the associated lexical terms related to the associated text are located from the sequence of instruction lexical terms based on their position indices. Optionally, the task text instruction can be "The right gripper stacks the blue bowl and the red bowl onto the white tabletop," where the corresponding object phrases are "blue bowl" and "red bowl." "Blue bowl" and "red bowl" are both subwords, and the positions of all the subwords in the instruction lexical term sequence constitute the object lexical set.

[0094] It should be noted that the associated lexical units only contain text information related to the object being operated on, excluding irrelevant words such as robotic arm, camera, and background. This can eliminate the interference of irrelevant text on the subsequent object attention heatmap, making the attention aggregation result more focused on the real operation target.

[0095] Understandably, by using a frozen large language model to accurately extract related lexical units of the manipulated object from the task text instructions, the need to train additional visual detectors or localization networks is avoided, thus achieving pure text-driven object localization. At the same time, by retaining only object-related lexical units rather than all text lexical units, the noise interference of irrelevant words on the attention distribution is effectively eliminated, providing accurate text-side anchors for the subsequent generation of high-quality object conditional attention distributions.

[0096] Step S104: Calculate the initial cross-attention matrix, modify the initial cross-attention matrix based on the associated lexical, and generate an object conditional attention distribution. Compared with the initial cross-attention matrix, the attention score of the manipulated object is increased, and the attention score of objects other than the manipulated object in the space is decreased.

[0097] Specifically, the initial cross-attention matrix refers to the cross-attention matrix calculated within the cross-attention layer of the video diffusion transformer, using the video spatiotemporal location as the query and the text lexical units corresponding to all task text instructions as the keys. The initial cross-attention matrix includes a set of attention values ​​for multiple tiles calculated by multiple attention heads and multiple attention layers. The object-conditional attention distribution refers to an attention heatmap aligned with the spatial resolution of the video latent variables, obtained after multi-level, multi-head filtering and aggregation of the initial cross-attention matrix based on associated lexical units. Each element in the object-conditional attention distribution has a value range of [0,1]. Compared to the initial cross-attention matrix, the object-conditional attention distribution increases the attention score of the manipulated object and decreases the attention scores of objects other than the manipulated object within the space.

[0098] The following is a specific embodiment to illustrate the implementation process of this step in detail:

[0099] (1) Using the video location as the query and the corresponding word of the task text instruction as the key value, calculate the original cross-attention distribution of the single head and single block to obtain the initial cross-attention matrix.

[0100] Specifically, in the video world model, the video latent variables are linearly projected to obtain the query vector, and the associated terms are linearly projected to obtain the key vector. Based on the query vector and the key vector, the initial cross-attention matrix of a single head and a single block is calculated. The initial cross-attention matrix includes multiple attention calculation layers, each attention calculation head, and the cross-attention value corresponding to each video tile.

[0101] (2) Select the target attention probability corresponding to the associated word from the initial cross-attention matrix based on the associated word.

[0102] Specifically, the target attention probability is:

[0103] ;

[0104] Where p is the video spatiotemporal location index, and j is the associated term index. O is the set of lexical units corresponding to the task text instruction, l is the index of the attention calculation layer, h is the index of the attention head, and d is the vector dimension.

[0105] The target attention probability represents the attention probability using only the set of associated lexical units O as the key. This target attention probability only includes the attention scores corresponding to associated lexical units, excluding interference from non-object lexical units such as "background". For each video position p, its target attention probability at the l-th and h-th head is... .

[0106] (3) Aggregate the target attention probabilities to generate the object conditional attention distribution.

[0107] Specifically, to enhance localization robustness, the target attention probabilities of multiple selected layers and multiple attention heads are summed and averaged:

[0108] ;

[0109] Where B is the set of selected transform layers, and H is the set of all attention heads. Let p be the object conditional attention score at video position p. Further, normalize the corresponding conditional attention score and map it to the interval [0,1] to obtain the object conditional attention distribution A.

[0110] In a specific implementation, for example, in one embodiment, 11 attention calculation layers from the 10th to the 20th layer in the video world model are selected, and the results of all 24 attention heads are averaged. Only the attention scores corresponding to the related words "blue" and "bowl" are accumulated. After min-max normalization, the scores are mapped to the [0,1] interval to obtain the object conditional attention distribution A.

[0111] Furthermore, compared to the initial cross-attention matrix that directly uses all text terms, the attention scores of the video regions containing the manipulated objects, such as the "blue bowl," are significantly improved in the object conditional attention distribution after the related terms are corrected, while the attention scores of other irrelevant objects in the space, such as the background wall and clutter on the table, are greatly suppressed.

[0112] Understandably, this application reuses the initial cross-attention matrix already calculated in the forward propagation of the video world model, without adding any new network layers or additional Cross-Attention calculations. It can extract the object conditional attention distribution from the native attention within the model simply by filtering the text channel through the object lexical set. The multi-head, multi-block aggregation strategy effectively smooths out the drastic fluctuations in the heatmap caused by single-layer, single-head attention noise, providing a stable and reliable attention prior for the generation of subsequent interaction graphs.

[0113] Step S105: Generate an interaction graph based on the object conditional attention distribution.

[0114] Specifically, the interaction graph refers to a soft mask tensor with the same resolution as the video latent variable space. The value range of each element in the soft mask tensor is [0,1]. The higher the value, the more likely the location is to be a region where the embodied intelligence and the manipulated object physically interact, such as the moment when a hand touches an object or the region where an object is deformed by force.

[0115] The following is a specific example to illustrate the process of generating an interaction diagram:

[0116] (1) Stop gradient of the conditional attention distribution of the object and calculate log odds.

[0117] Specifically, firstly, the stopping gradient operator sg[·] is applied to the object conditional attention distribution A to block backpropagation, and then temperature adjustment is performed and the data is clipped to a safe range.

[0118] ;

[0119] in, Conditional attention distribution for the clipped object. This is the temperature control coefficient. It is a minimal constant, 0 < <<1, optional It can be set to 0.01 to prevent numerical anomalies in subsequent logarithmic calculations.

[0120] Transform the cropped object conditional attention distribution into a log-odds space:

[0121] ;

[0122] Where Z is the transformed log-odds space.

[0123] (2) Calculate the smooth perturbation in the coarse spatiotemporal grid, and generate candidate responses based on the smooth perturbation and the log probability.

[0124] Specifically, coarse spatiotemporal grid computing refers to sampling random perturbation noise in the space of video latent variables using a sparse raster array with a resolution lower than the original resolution, and then sampling back to the original latent variable resolution in the noise field with a resolution lower than the original resolution through trilinear interpolation to obtain continuous smooth perturbation.

[0125] Gaussian noise is sampled at each location using coarse spatiotemporal grid calculations. Noise is diffused to fine-grained pixels using trilinear interpolation to obtain a smooth perturbation. This smooth perturbation is then added to a log-probability value, adjusted for temperature parameters, and activated using a sigmoid function to obtain candidate responses.

[0126] ;

[0127] in, For noise scale parameters, For temperature parameters, k = 1, 2, 3, ..., k is the candidate index.

[0128] Trilinear interpolation ensures the continuity of perturbations at adjacent spatiotemporal locations, avoiding the formation of fragmented regions by independent noise.

[0129] (3) Generate multiple candidate regions from the candidate responses based on the coverage ratio.

[0130] Specifically, for each candidate response, the spatiotemporal location with the highest response value is selected according to a preset coverage ratio ρ. Optionally, ρ can be 0.1, resulting in multiple candidate regions.

[0131] ;

[0132] in, This is an indicator function that takes the value 1 when the condition is true and 0 otherwise. It is a binary mask. It is a function. Indicates from From the entire set of spatiotemporal locations, locations with higher scores are selected according to a preset ratio ρ. Then, the scores of all locations are sorted from highest to lowest to find the entry point. The minimum score required.

[0133] Candidate Responses Positions with scores greater than or equal to the minimum score are retained, and the indicator function is set to 1 for candidate responses. Positions below the minimum score are discarded, the indicator function is set to 0, and a binary mask is finally formed.

[0134] This invention employs an equal coverage screening strategy for the corresponding regions to ensure that each candidate region contains the same number of spatiotemporal pixels. This eliminates the impact of differences in object size in space on subsequent scoring calculations, allowing the objects of interest indicated in the interactive graph to be determined solely by the relevance and importance of the task, rather than by size or area. It automatically adjusts the objects of interest in different task contents, thereby improving the accuracy of the model's video prediction and the adaptability of the task.

[0135] (4) The candidate weights are determined by the mean of the candidate regions where the gradient error stops within the plurality of candidate regions.

[0136] Specifically, the score for each candidate region is first calculated based on the object conditional attention distribution with applied stopping gradients, a binary mask, and a minimal constant basis:

[0137] ;

[0138] in, Score the candidate regions.

[0139] Then, the candidate region scores are converted into weights:

[0140] ;

[0141] Where β is a temperature coefficient, which is optional, β=0.7, used to control the steepness of the weight distribution.

[0142] (5) Calculate the interaction graph based on the candidate weights and multiple candidate regions.

[0143] Specifically, all candidate regions are weighted, stacked, and then cropped:

[0144] ;

[0145] The resulting interaction graph G is a continuous soft mask with values ​​between [0,1]. The larger the value of an element, the larger the model prediction error and the higher the overlap with the object's conditional attention, which is a hotspot of embodied intelligence interaction.

[0146] Understandably, by applying stopping gradients, log-odds transformations, coarse-grid smoothing perturbations, and equal-coverage filtering to the object's conditional attention distribution, potential interaction regions are detected from multiple perspectives and candidate weights are calculated, ultimately synthesizing the interaction graph. The entire process does not rely on any external dense annotations such as optical flow, segmentation masks, or depth maps; it only utilizes the model's internal attention and local prediction errors to generate the graph online. This ensures the accuracy of the interaction region while avoiding error propagation and storage overhead from external estimation models.

[0147] Step S106: Calculate the supervision weights of each position in the space based on the interaction graph, and weight the velocity objective function based on the supervision weights to obtain the supervision function.

[0148] Specifically, the supervision weights determine the contribution of each spatiotemporal location in the video latent variable space to the final loss function. Compared to other objects in the space, interactive regions related to the manipulated object receive higher supervision weights, while non-interactive regions receive lower supervision weights. For example, maintaining the base weights means that the supervision weights of interactive regions related to the manipulated object are greater than those of non-interactive regions. The supervision function is a loss function obtained by weighting and normalizing the position-by-position error using the supervision weights based on the global velocity objective function.

[0149] The following is a specific embodiment to illustrate the implementation process of this step in detail:

[0150] (1) Calculate the supervision weights based on the interaction graph.

[0151] Specifically, each spatiotemporal location in the space is traversed, and a first weight is calculated based on the first information of the current spatiotemporal location in the interaction graph. The supervision weight of the current spatiotemporal location is then calculated based on the product of the first weight and the enhancement factor. The supervision weight of the spatiotemporal location where the manipulated object is located is greater than 1, and the supervision weights of all spatiotemporal locations in the space other than the manipulated object are less than 1. For each spatiotemporal location p in the space, the first information is the value of the current spatiotemporal location in the interaction graph. The supervision weight is calculated based on the value Gp of the current spatiotemporal location in the interaction graph.

[0152] ;

[0153] in, The enhancement factor is a preset value. The magnitude of the enhancement factor varies depending on the spatiotemporal location and the degree of association with the operable object, and the two are positively correlated. This is optional. .

[0154] when When, that is, in the strong interaction region, When the supervisory weight reaches its maximum; When it is a non-interactive region, Maintain basic oversight. The oversight weight of the spatiotemporal location of the manipulated object is greater than 1, and the oversight weight of all spatiotemporal locations in space other than the manipulated object is not less than 1.

[0155] (2) The velocity objective function is weighted based on the supervision weights to obtain the supervision function.

[0156] Specifically, the weighted sum of the function values ​​at each spatiotemporal location in the velocity objective function is calculated using the supervised weights, and the weighted sum is normalized based on the sum of the supervised weights to obtain the supervised function:

[0157] ;

[0158] The numerator is the sum of weighted errors at all spatiotemporal locations, and the denominator is the sum of supervision weights at all locations. This is used to normalize the total weights, ensuring the overall scale of the loss is stable and avoiding gradient explosion due to excessive supervision weights.

[0159] In a specific implementation, for example, in one embodiment, it is assumed that there are 18,000 spatiotemporal locations for the video latent variables, of which approximately 10% (1,800) belong to interactive regions (with an average weight of 4.0), and 90% (16,200) belong to non-interactive regions (with a weight of 1.0). Then the denominator... The expected value of the weighted loss remains on the same order of magnitude as that of the unweighted loss, indicating a stable training process.

[0160] Step S107: Perform multiple backpropagations based on the supervision function. The parameter range updated in each backpropagation is different. The set of parameters updated by all backpropagations is all the parameters of the video world model.

[0161] Specifically, multiple backpropagation refers to performing independent backpropagation and parameter update operations for different parameter groups in the video world model. Each backpropagation is independent, the parameters updated in each backpropagation are completely different, and each backpropagation only updates parameters within a specific range. Decoupled training is achieved through parameter grouping and gradient routing. The union of the parameter sets updated by all backpropagations equals all trainable parameters of the video world model.

[0162] The following is a specific example to illustrate the implementation process of two backpropagation steps:

[0163] The backpropagation includes at least a backpropagation to update the cross-attention parameters, which serves as the first backpropagation. In this first backpropagation, the global target gradient corresponding to the cross-attention parameters is retained; the parameter gradients of the video world model other than the cross-attention parameters are suppressed; and backpropagation is performed based on the global target gradient to update the cross-attention parameters.

[0164] Specifically, the global objective gradient originates from the global supervised objective in step S102. In this backpropagation, only the parameters across attention layers in the video diffusion transformer are allowed to flow through and generate updates, including at least the projection matrices of Query / Key / Value and the output projection matrix. The gradients of all other parameters, such as those of the self-attention layer, feedforward network (FFN), AdaLN module, time-step embedded MLP, and action-conditional injection MLP, are explicitly set to zero or masked.

[0165] Update cross-attention parameters:

[0166] ;

[0167] in, For cross-attention parameter set.

[0168] The learning across attention parameters is driven only by the global stream matching loss and is not interfered with by the subsequent interaction-weighted loss, ensuring that the attention mechanism can purely learn the alignment relationship between text and video.

[0169] The backpropagation includes at least a backpropagation to update model parameters other than the cross-attention parameters, which serves as the second backpropagation. In this backpropagation, the interaction-weighted gradients of the model parameters other than the cross-attention parameters are retained; the parameter gradients corresponding to the cross-attention parameters are suppressed; and backpropagation is performed based on the interaction-weighted gradients to update the model parameters other than the cross-attention parameters in the video world model.

[0170] Specifically, the interactive weighted gradient originates from the supervision function in step S106. In this backpropagation, only the parameters in the diffusion transformer, except for those across attention layers, are allowed to flow through and generate updates. These parameters include at least the self-attention layer, FFN, AdaLN, time-step embedded MLP, and action-conditional injection MLP. The gradients of the cross-attention parameters are explicitly set to zero or masked.

[0171] Update model parameters except for cross-attention parameters:

[0172] ;

[0173] in, This is the set of all model parameters except for the cross-attention parameters.

[0174] The parameter updates of the backbone network are precisely guided by interactive weighted supervision, which forces the optimization capabilities to focus on learning the physical details of the interactive regions, rather than tampering with the attention distribution to fit the loss function.

[0175] Furthermore, the two backpropagations share the calculation results of the same forward propagation (i.e., velocity field prediction). (and the intermediate activation values ​​of each layer), parameter grouping updates are only achieved during the backpropagation phase through gradient masking. This avoids the overhead of repeated forward computation and maintains training efficiency.

[0176] Understandably, by reusing the same forward propagation result to perform two backpropagations, the cross-attention parameters are decoupled from the updates of other parameters. This enables cross-attention to change its own attention distribution due to the pressure of interaction weighted loss, while the generated parameters can focus on learning the physical details of the interaction under the guidance of stable attention localization. This dual backpropagation mechanism is a key technical support for achieving the goal of not relying on external annotations and relying solely on internal attention calibration.

[0177] It should be noted that after training is completed, the inference phase only runs the original sampling and decoding process of the video world model. That is, the inference phase does not execute steps S103-107, removes the steps of generating associated lexical units and interaction graphs and calculating supervision functions, and does not load any candidate sampling, interaction graph generation or supervision reallocation modules. The inference architecture is completely consistent with that before training and does not add any additional computational overhead.

[0178] It should also be noted that after the video world model has been trained, the specific steps for using the video world model are as follows: First, input the task instruction text, reference observations, and control signals into the trained video world model; second, the video world model generates a full information encoding of the task instruction text and initial video latent variables; then, the video world model calculates cross-attention based on the full information encoding and initial video latent variables to generate denoised video latent variables; finally, the video world model decodes the denoised video latent variables and generates the predicted video.

[0179] In summary, this embodiment provides a video world model training method based on object-conditional attention calibration. Firstly, from the perspective of the entire training process, this application first acquires the target video, task instructions, and control signals, constructing noisy video latent variables and a global supervision target; then, it extracts object lexical units from the task instructions and aggregates cross-attention to obtain the object-conditional attention distribution; next, it generates candidate regions for coverage based on stopping gradient attention and coarse-grid smoothing perturbation, and generates a synthetic interaction graph based on the global prediction error; then, it constructs an interaction-weighted supervision target normalized by total weights; finally, it uses double backpropagation to update the cross-attention parameters of the global target and update the remaining parameters of the interaction-weighted target. Throughout the process, the object response region is determined online by the model's native cross-attention and per-position error, eliminating the need for external dense annotations such as optical flow, depth, 3D reconstruction, or manual segmentation. This avoids the high preprocessing overhead, high storage costs, and error propagation risks associated with existing technologies that rely on additional estimation models. Coarse-grid interpolation generates smooth perturbations, and equal coverage filtering eliminates the influence of region size on scoring, making candidate regions spatiotemporally coherent and comparable. The interaction graph focuses on the more difficult-to-predict object contact and state change locations. Total weight normalization strengthens interaction location while maintaining global supervision scale stability. By reusing the model's internal attention to achieve zero-cost object localization, and by accurately directing optimization pressure to the interaction region through supervision redistribution, the model's ability to learn physical interaction details is significantly improved without changing the inference architecture.

[0180] Secondly, this method employs object-conditional attention-based gradient stopping, temperature adjustment, and conversion to logarithmic probabilities. Gaussian noise is sampled in a coarse spatiotemporal grid and smoothed using trilinear interpolation. Candidate weights are then determined through equal coverage screening and local error calibration. Simultaneously, gradient stopping blocks backpropagation of candidate generation branches. This candidate generation mechanism utilizes coarse grid interpolation to ensure the spatiotemporal continuity of the perturbation, avoiding fragmented regions formed by noise from independent locations. Equal coverage screening eliminates the influence of object size differences on scoring, making scores from different candidate regions fair and comparable. Local error calibration determines weights based on the mean of the stopping gradient error within the candidate region, giving greater weight to candidates with higher errors due to object contact, occlusion changes, or state transitions, and less weight to candidates dominated by static backgrounds. This reduces dependence on external annotations, improves the accuracy and training stability of interactive region localization, and lowers computational costs.

[0181] Thirdly, using parameter grouping and double backpropagation, a gradient routing mechanism is employed to retain the global target gradient across attention parameters and the interactive weighted target gradient of other parameters. This mechanism ensures that the parameters generating the localization basis are not directly updated by their derived region targets, avoiding the attention distribution distortion caused by the self-referential coupling between region estimation and region optimization. Simultaneously, double backpropagation is applied to the training framework of the video world model, achieving decoupled updates between the localization and generation modules without altering the inference interface. After training, all newly added modules are used only during the training phase; the original backbone, sampler, and decoder are reused during inference, adapting to robot motion trajectories, human hand pose videos, and other control signals. This reduces training instability caused by self-referential dependencies, enabling the model to accurately enhance interactive regions while maintaining global supervision, without adding any additional computational overhead or deployment costs during the inference phase.

[0182] Example 2:

[0183] This application provides an apparatus for training a video world model based on object-conditional attention calibration. The apparatus includes:

[0184] The acquisition module 210 is used to acquire video samples and conditional sequence samples and input them into the video world model. The conditional sequence samples include at least task text instructions, and the task text instructions at least describe the object being manipulated by the embodied intelligence in space.

[0185] The generation module 220 is used for the video world model to generate noisy video latent variables based on the video samples, and to generate a velocity objective function based on the noisy video latent variables and clean latent variables.

[0186] The extraction module 230 is used to extract related words of the operated object based on the task text instruction;

[0187] The calculation module 240 is used to calculate an initial cross-attention matrix, modify the initial cross-attention matrix based on the associated lexical, and generate an object conditional attention distribution. Compared with the initial cross-attention matrix, the object conditional attention distribution increases the attention score of the manipulated object and decreases the attention score of objects other than the manipulated object in the space.

[0188] The generation module 220 is used to generate an interaction graph based on the object conditional attention distribution;

[0189] The generation module 220 is used to calculate the supervision weights of each position in the space based on the interaction graph, and to weight the velocity objective function based on the supervision weights to obtain the supervision function;

[0190] The update module 250 is used to perform multiple backpropagations based on the supervision function. The parameter range updated in each backpropagation is different, and the set of parameters updated by all backpropagations is all the parameters of the video world model.

[0191] Example 3:

[0192] Thirdly, embodiments of this application provide an electronic device, Figure 3 This is a block diagram illustrating an electronic device according to an exemplary embodiment. (e.g.) Figure 3 As shown, the electronic device may include a processor 11 and a memory 12 storing computer program instructions.

[0193] Specifically, the processor 11 may include a central processing unit (CPU) or a specific integrated circuit. The memory 12 may include a large-capacity memory for data or instructions. Where appropriate, the memory 12 may be internal or external to the data processing device. The memory 12 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 11.

[0194] The processor 11 reads and executes computer program instructions stored in the memory 12 to implement any of the video world model training methods based on object conditional attention calibration in the above embodiments.

[0195] In one embodiment, the electronic device may further include a communication interface 13 and a bus 10. Wherein, as... Figure 3 As shown, the processor 11, memory 12, and communication interface 13 are connected through bus 10 and communicate with each other.

[0196] Communication interface 13 is used to enable communication between modules, devices, units, and / or equipment in the embodiments of this application. Bus 10 includes hardware, software, or both, coupling components of electronic devices together. Where appropriate, bus 10 may include one or more buses.

[0197] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0198] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0199] For the apparatus embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. Those skilled in the art can understand and implement these embodiments without any inventive effort.

[0200] Fourthly, embodiments of this application provide a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the training method for a video world model based on object conditional attention calibration provided in the first aspect.

[0201] Among them, readable storage media include at least: portable disks, hard disks, random access memory, read-only memory, etc.

[0202] In possible implementations, this application can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform steps implementing the video world model training method based on object conditional attention calibration provided in the first aspect. The program code for executing this application can be written in any combination of one or more programming languages, and can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0203] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for training a video world model based on object-conditional attention calibration, characterized in that, The system includes: Video samples and conditional sequence samples are input into the video world model. The conditional sequence samples include at least task text instructions, which at least describe the object being manipulated by the embodied intelligence in space. The video world model generates noisy video latent variables based on the video samples, and generates a velocity objective function based on the noisy video latent variables and clean latent variables. Extract related words of the operated object based on the task text instructions; Calculate an initial cross-attention matrix, modify the initial cross-attention matrix based on the associated lexical units, and generate an object conditional attention distribution. Compared with the initial cross-attention matrix, the attention score of the manipulated object is increased, and the attention scores of objects other than the manipulated object in the space are decreased. Generate an interaction graph based on the object's conditional attention distribution; The supervision weights at each location in the space are calculated based on the interaction graph, and the velocity objective function is weighted based on the supervision weights to obtain the supervision function; Multiple backpropagations are performed based on the supervision function, with each backpropagation updating a different range of parameters. The set of parameters updated by all backpropagations is the set of all parameters of the video world model.

2. The method according to claim 1, characterized in that, The initial cross-attention matrix includes at least multiple attention heads and multiple tile attentions; The calculation of the initial cross-attention matrix, the modification of the initial cross-attention matrix based on the associated lexical units, and the generation of the object conditional attention distribution include: Using video location as the query and the corresponding word of the task text instruction as the key, calculate the original cross-attention distribution of single head and single block to obtain the initial cross-attention matrix; Based on the associated lexical units, the target attention probabilities corresponding to the associated lexical units are selected from the initial cross-attention matrix; The target attention probabilities are aggregated to generate the object conditional attention distribution.

3. The method according to claim 1, characterized in that, The step of extracting related terms of the operated object based on the task text instruction includes: The large language model is frozen by inputting the task text instructions as parameters. The large language model locates the associated text of the manipulated object from the task text instructions; Align the associated text with the task text instruction, and determine the associated words based on the aligned index. The associated words contain only text information related to the object being operated on.

4. The method according to claim 1, characterized in that, The step of generating an interaction graph based on the object conditional attention distribution includes: The gradient is stopped for the conditional attention distribution of the object, and the log odds are calculated; a smooth perturbation is calculated on a coarse spatiotemporal grid, and candidate responses are generated based on the smooth perturbation and the log odds; Multiple candidate regions are generated from the candidate responses based on the coverage ratio; The candidate weights are determined by the average of the candidate regions where the gradient error stops within the plurality of candidate regions. An interaction graph is calculated based on the candidate weights and multiple candidate regions.

5. The method according to claim 1, characterized in that, The calculation of the supervision weights at each location in the space based on the interaction graph includes: Traverse each spatiotemporal location in the space, calculate the first weight based on the first information in the interaction graph of the current spatiotemporal location, calculate the supervision weight of the current spatiotemporal location based on the product of the first weight and the enhancement factor, the supervision weight of the spatiotemporal location where the manipulated object is located is greater than 1, and the supervision weight of all spatiotemporal locations in the space other than the manipulated object is less than 1.

6. The method according to claim 1, characterized in that, The step of weighting the velocity objective function based on the supervision weights to obtain the supervision function includes: The supervisory weights are used to calculate the weighted sum of the spatiotemporal position function values ​​in the velocity objective function; The supervision function is obtained by normalizing the weighted sum based on the sum of the supervision weights.

7. The method according to claim 1, characterized in that, The backpropagation includes at least backpropagation updating the attention parameters, and multiple backpropagations based on the supervision function, including: Preserve the global target gradient across attention parameters; Suppress the parameter gradients of the video world model other than the cross-attention parameters; The backpropagation is performed based on the global target gradient to update the cross-attention parameters.

8. The method according to claim 1, characterized in that, The backpropagation includes at least backpropagation to update model parameters other than the cross-attention parameters, and multiple backpropagations based on the supervision function, including: Preserve the interactive weighted gradients of model parameters other than those across attention parameters; Suppress the parameter gradients across attention parameters; The backpropagation is performed based on the interaction-weighted gradient to update the model parameters in the video world model other than the attention parameters.

9. The method according to claim 1, characterized in that, After the video world model has been trained, the method further includes: Input the task instruction text, reference observations, and control signals into the trained video world model; The video world model generates full-information encoding of the task instruction text and initial video latent variables; Based on the full information encoding and the initial video latent variables, cross-attention is calculated to generate denoised video latent variables; Decode the latent variables of the denoised video to generate a predicted video.

10. A video world model training device based on object-conditional attention calibration, characterized in that, The device includes: The acquisition module is used to acquire video samples and conditional sequence samples and input them into the video world model. The conditional sequence samples include at least task text instructions, which at least describe the object being manipulated by the embodied intelligence in space. The generation module is used for the video world model to generate noisy video latent variables based on the video samples, and to generate a velocity objective function based on the noisy video latent variables and the clean latent variables; The extraction module is used to extract related words of the operated object based on the task text instructions; The calculation module is used to calculate an initial cross-attention matrix, modify the initial cross-attention matrix based on the associated lexical units, and generate an object conditional attention distribution. Compared with the initial cross-attention matrix, the object conditional attention distribution increases the attention score of the manipulated object and decreases the attention score of objects other than the manipulated object in the space. The generation module is used to generate an interaction graph based on the object's conditional attention distribution; The generation module is used to calculate the supervision weights of each position in the space based on the interaction graph, and to weight the velocity objective function based on the supervision weights to obtain the supervision function; The update module is used to perform multiple backpropagations based on the supervision function. The parameter range updated in each backpropagation is different, and the set of parameters updated by all backpropagations is all the parameters of the video world model.

Images