A gradient isolation-based embodied world model control mode continuous expansion method and related equipment

Abstract

The application is suitable for the field of embodied intelligence and video generation technology, and provides a gradient isolation-based embodied world model control mode continuous expansion method and related equipment. The method comprises the following steps: dividing model parameters in a first embodied world model into cross-mode shared parameters and mode-specific parameters, respectively performing control following error evaluation on all existing control modes to obtain a first control following error; connecting a newly added special expert and a newly added control path to the first embodied world model and generating a second embodied world model based on the first control following error; updating the cross-mode shared parameters to obtain a shared parameter update result of the current training step; and based on the shared parameter update result, calculating the relative degradation rate of each control mode and generating a sampling ratio and a loss weight of the next evaluation period. The application can maintain a controlled balance between new mode learning and existing mode maintenance without manual intervention in the entire expansion process.

Description

Technical Field

[0001] This invention relates to the fields of embodied intelligence and video generation technology, and in particular to a method and related equipment for the continuous expansion of embodied world model control modes based on gradient isolation. Background Technology

[0002] Existing embodied world models use a diffusion transformer as the backbone, replacing the feedforward network with a sparse hybrid expert layer for control awareness. This layer consists of a shared expert, several specialized experts corresponding to different control modes, and a sparse router. When a new control mode needs to be added, the existing technology adds a new specialized expert, mixes the new mode data with the existing mode data at a fixed ratio, and applies a small learning rate to the shared part for training.

[0003] However, existing sparse routing structures inherently guarantee gradient isolation between modality-specific experts at the structural level. Since the parameters of existing modality-specific experts do not receive gradients from other modalities during the expansion process, freezing them or applying a small overall learning rate is meaningless. The real path to performance degradation of existing modalities lies in the following: shared parameters across modalities, such as shared experts, the diffusion transformer backbone, and the router front trunk, shift under the unidirectional drive of the new modality's gradient, leading to a decrease in the following ability of existing modalities that rely on these shared parameters. Existing technologies proportionally reduce the overall learning rate, suppressing the shift of shared parameters while simultaneously suppressing the necessary adaptation of the new modality in shared representations, creating an irreconcilable dilemma between stability and plasticity. Fixed-ratio mixing cannot adapt to the differences in data scale and convergence speed among modalities; the gradient norm of the new modality naturally dominates in the early stages of training, and the synthesized gradient direction is still dominated by the new modality. Furthermore, the entire process lacks an online observation mechanism for whether existing modalities have degraded; discovering problems requires retraining, which is costly. Summary of the Invention

[0004] This invention provides a method and related equipment for the continuous expansion of embodied world model control modes based on gradient isolation. This invention solves the technical problems of stability and plasticity caused by the meaningless operation of freezing or low learning rate applied to dedicated experts in the prior art. It enables the entire expansion process to maintain a controlled equilibrium between learning new modes and maintaining existing modes without human intervention.

[0005] In a first aspect, embodiments of this application provide a method for controlling modal persistence expansion based on gradient isolation embodied world model, including: The model parameters in the first embodied world model are divided into cross-modal shared parameters and modal-specific parameters. Control follow-up error evaluation is performed on all existing control modes to obtain the first control follow-up error of each existing control mode. The newly added dedicated experts and the newly added control paths are connected to the first embodied world model, and a second embodied world model is generated based on the first control following error. Based on the second embodied world model, projection resolution is performed on the gradient vector of the new control mode and the gradient vectors of each existing control mode, and the cross-modal shared parameters are updated to obtain the shared parameter update result of the current training step; Based on the updated results of the shared parameters, the relative degradation rate of each control mode is calculated and the sampling ratio and loss weight for the next evaluation cycle are generated.

[0006] Optionally, in a first implementation of the first aspect of the present invention, the step of dividing the model parameters in the first embodied world model into cross-modal shared parameters and modality-specific parameters, and performing control following error evaluation on all existing control modes to obtain the first control following error of each existing control mode, includes: The model parameters in the first embodied world model are divided into cross-modal shared parameters and modality-specific parameters. Based on the cross-modal shared parameters and the modality-specific parameters, a fixed probe set is constructed, and deterministic inference is performed on all existing control modes in the fixed probe set using exponential moving average weights to obtain the first generated video of each existing control mode. The first generated video of each existing control mode is input into the corresponding control follow-up error calculation process to obtain the first control follow-up error of each existing control mode.

[0007] Optionally, in a second implementation of the first aspect of the present invention, the step of connecting the newly added dedicated expert and the newly added control path to the first embodied world model based on the first control following error to obtain the second embodied world model includes: Based on the weight matrix of the shared expert two-layer fully connected network in the extended baseline snapshot, the input projection weight matrix and output projection weight matrix of the newly added special expert are calculated. All elements of the output projection weight matrix are set to zero, and scalar gating parameters with initial values ​​of zero are connected in series in the residual branch of the newly added special expert to enable the newly added special expert to complete the hot start initialization. The weights and biases of the last linear layer of the newly added control path are initialized to zero. The newly added dedicated expert and the newly added control path are connected to the first embodied world model. Deterministic inference is performed on all existing control modes and the second embodied world model is constructed by combining the first control following error.

[0008] Optionally, in a third implementation of the first aspect of the present invention, the step of performing projection resolution on the gradient vector of the new control mode and each existing control mode gradient vector based on the second embodied world model and updating the cross-modal shared parameters to obtain the shared parameter update result of the current training step includes: Based on the current sampling ratio, hierarchical sampling is performed on all control modes within each computing node and input into the MoE layer of the second embodied world model to obtain the grouped forward computation results of the MoE layer. The original gradients of the new control mode gradient vector and each existing control mode gradient vector applied to the cross-modal shared parameters are obtained respectively. The L2 moving average of each control mode gradient vector is calculated, and the median of the L2 moving average of all control mode gradients is used as a unified reference scale to scale each control mode gradient vector to obtain the norm-equilibrium new control mode gradient vector and the norm-equilibrium gradient vectors of each existing control mode. Based on the grouped forward computation results of the MoE layer, a projection resolution operation is performed on the norm-equilibrium new control mode gradient vector to obtain the conflict-resolved new control mode gradient vector. Based on the new control mode gradient vector after conflict resolution and the existing control mode gradient vectors after norm equalization, gradient pruning and optimizer step update are performed on the cross-modal shared parameters to obtain the shared parameter update result of the current training step.

[0009] Optionally, in a fourth implementation of the first aspect of the present invention, the step of performing hierarchical sampling of all control modes within each computing node based on the current sampling ratio and inputting it into the MoE layer of the second embodied world model to obtain the grouped forward computation results of the MoE layer includes: Based on the current sampling ratio, hierarchical sampling is performed on the independent data loaders of all control modes within each computing node to obtain a global batch carrying control mode identifiers; The global batch is input into the diffusion transformer backbone of the second embodied world model to obtain the token hidden state grouping of each control mode; The hidden state groups of the tokens for each control mode are input into the router. The router performs mean pooling on the hidden state vector of each token and then performs front trunk mapping to obtain the forward calculation result of the MoE layer.

[0010] Optionally, in a fifth implementation of the first aspect of the present invention, the step of calculating the relative degradation rate of each control mode and generating the sampling ratio and loss weight for the next evaluation period based on the shared parameter update result includes: Based on the shared parameter update results, deterministic reasoning is performed on all existing control modes and new control modes to obtain the third generated video of each control mode. The third generated video of each control mode is then input into the corresponding control following error calculation process to obtain the second control following error of each control mode. The second control following error of each existing control mode is compared with the first control following error solidified before expansion. The first control following error of the new control mode is solidified in the first evaluation cycle after its generated video meets the preset effective generation conditions, and this solidified result is used as the comparison baseline in each subsequent evaluation cycle. Calculate the relative degradation rate of the second control following error and the first control following error for each control mode, and determine the degradation trend of each control mode based on the relative degradation rate; Based on the degradation trend and relative degradation rate of each control mode, the sampling ratio and loss weight for the next evaluation period are calculated.

[0011] Secondly, embodiments of this application provide a gradient-isolated embodied world model control modality persistence extension device, comprising: The error evaluation module is used to divide the model parameters in the first embodied world model into cross-modal shared parameters and modal-specific parameters, and to perform control follow-up error evaluation on all existing control modes to obtain the first control follow-up error of each existing control mode. The access module is used to connect the newly added dedicated experts and the newly added control paths to the first embodied world model and generate a second embodied world model based on the first control following error. The parameter update module is used to perform projection resolution on the gradient vector of the new control mode and the gradient vector of each existing control mode based on the second embodied world model, and update the cross-modal shared parameters to obtain the shared parameter update result of the current training step. The calculation module is used to calculate the relative degradation rate of each control mode based on the update results of the shared parameters and generate the sampling ratio and loss weight for the next evaluation period.

[0012] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-described gradient-isolated embodied world model control mode continuous expansion method.

[0013] Fourthly, embodiments of this application provide a readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described gradient-isolated embodied world model control mode continuous expansion method.

[0014] Fifthly, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the above-described gradient-isolated embodied world model control modality continuous expansion method.

[0015] In one scheme provided by the aforementioned gradient-isolation-based embodied world model control modality continuous expansion method and related equipment, all model parameters are precisely divided into cross-modal shared parameters and modality-specific parameters. This addresses the degradation of existing modal control performance stemming from the displacement of cross-modal shared parameters under the unidirectional drive of the new modal gradient, rather than the rewriting of the specific expert parameters themselves. This precise positioning allows all intervention methods of this invention to focus on the gradient synthesis process of shared parameters, solving the stability and plasticity problems caused by the meaningless operation of freezing or applying small learning rates to specific experts in existing technologies. By projecting the output of newly added specific experts, adding channel weights to the end layer of the new control path, adding channel weights to the shared patch-embed layer, and adding rows to the router output header to perform systematic zero initialization, combined with a linear warmup mechanism of residual branch scalar gating parameters, it ensures that the inference output of all existing control modal samples at the start of expansion is consistent with the bit-level output before expansion, eliminating the functional perturbation of existing modalities at the moment of structural access. Meanwhile, the softmax normalization of the router qualification set is only performed on the qualification set of the corresponding mode. The output logit of the newly added dedicated expert does not enter the softmax denominator of the existing control mode sample. Mathematically, this strictly guarantees that the routing distribution of the existing control mode remains unchanged before and after the expansion. This eliminates the structural hidden danger in the existing technology where the router trunk is implicitly rewritten by the gradient of the new mode, which indirectly disturbs the routing distribution of the existing mode. By performing joint sampling and grouped forward computation on the real data of all control modes in each training step, and performing L2-norm moving average statistics and modal norm equalization on the gradient vectors of each control mode sub-batch on the cross-modal shared parameters, the directional dominance problem caused by the natural dominance of the gradient norm in the early stage of new mode training is eliminated. The components of the new control mode gradient vector with negative cosine values ​​relative to the gradient vectors of each existing control mode are sequentially projected and eliminated, so that the negative cosine components of the new control mode gradient vector relative to the gradient directions of each existing control mode are cleared or significantly reduced after elimination one by one. This reduces the probability of the synthetic update direction of shared parameters conflicting with the gradient directions of existing modes, and ensures that each update of shared parameters simultaneously carries the real gradient constraints of all existing modes. This is different from existing continuous learning methods such as elastic weight solidification, knowledge distillation, and generative replay that use approximations to replace real gradients.By replacing the diffusion reconstruction loss, which is extremely insensitive to control capability degradation, with control-following error probes specific to each modality (absolute trajectory error and relative pose error of the camera trajectory modality, component-normalized L2 distance of the robot operation modality, and normalized two-dimensional joint position error and joint prediction hit rate of the human hand motion graph modality), as the feedback signal, and performing least-squares linear fitting on the relative degradation rate sequence within the sliding window to obtain the slope to suppress single-point noise interference, the determination of degradation trend is moved from evaluation after training to periodic online observation during the training process. This invention uses two independent adjustment knobs, sampling ratio and loss weight, to intervene in the frequency of generating true gradients on shared parameters for each modality and the effective update steps of each modality's dedicated expert, forming a closed-loop correction mechanism for the degradation of existing modalities. The static prior ratio is replaced with dynamic adaptive adjustment based on control fidelity, so that the entire expansion process can maintain a controlled balance between new modality learning and existing modality maintenance without manual intervention. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of a control modal continuous expansion system based on gradient isolation of an embodied world model according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a method for continuous expansion of control modes based on gradient isolation in an embodiment of the present invention. Figure 3 yes Figure 2 A schematic diagram of the implementation process of step S10; Figure 4 yes Figure 2 A schematic diagram of the implementation process of step S20; Figure 5 yes Figure 2 A schematic diagram of the implementation process of step S30; Figure 6 yes Figure 2 A schematic diagram of the implementation process of step S40; Figure 7 This is a schematic diagram of the network structure of the embodied world model in one embodiment of the present invention; Figure 8 This is a schematic diagram comparing the structure before and after the addition of a dedicated expert and the addition of a control path in one embodiment of the present invention; Figure 9This is a schematic diagram of a closed-loop regulation mechanism in one embodiment of the present invention; Figure 10 This is a schematic diagram of a device for continuous expansion of control modes based on gradient isolation of an embodied world model according to an embodiment of the present invention; Figure 11 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. It should also be understood that, as used in this specification and the appended claims, the term "and / or" refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0020] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0021] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0022] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0023] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0024] To address the problems mentioned above, this application proposes a method and related equipment for the continuous expansion of embodied world model control modes based on gradient isolation. The gradient isolation-based method for the continuous expansion of embodied world model control modes provided by this invention can be applied to applications such as... Figure 1 The gradient-isolation-based embodied world model control modality continuous expansion system shown includes an embodied world model training device and a control performance evaluation and scheduling device. The embodied world model training device includes a parameter partitioning module, a gradient isolation update module, and a training state output module. The control performance evaluation and scheduling device includes a deterministic inference and error evaluation module, a relative degradation rate calculation module, and a sampling ratio and loss weight module. The two devices are bidirectionally connected and work together to complete the continuous expansion and training scheduling of the model.

[0025] In one embodiment, such as Figure 2 As shown, a gradient isolation-based method for controlling modal persistence expansion of embodied world models is provided, which is then applied to... Figure 1 The following steps are used as an example to illustrate the control modal continuous expansion system based on gradient isolation embodied world model: S10: Divide the model parameters in the first embodied world model into cross-modal shared parameters and modal-specific parameters, and perform control following error evaluation on all existing control modes respectively to obtain the first control following error of each existing control mode; S20: Connect the newly added dedicated expert and the newly added control path to the first embodied world model and generate the second embodied world model based on the first control following error; S30: Based on the second embodied world model, perform projection resolution on the gradient vector of the new control mode and the gradient vector of each existing control mode, and update the cross-modal shared parameters to obtain the shared parameter update result of the current training step; S40: Based on the shared parameter update results, calculate the relative degradation rate of each control mode and generate the sampling ratio and loss weight for the next evaluation cycle.

[0026] In this embodiment, by performing joint sampling and grouped forward computation on the real data of all control modes in each training step, and performing L2-norm moving average statistics and modal norm equalization on the gradient vectors of each control mode sub-batch on the cross-modal shared parameters, the direction dominance problem caused by the natural dominance of the gradient norm in the early stage of training of the new mode is eliminated. The components of the gradient vector of the new control mode that have negative cosine values ​​relative to the gradient vectors of each existing control mode are sequentially projected and eliminated, so that the negative cosine components of the gradient vector of the new control mode relative to the gradient directions of each existing control mode are cleared or significantly reduced after elimination one by one. This reduces the probability of the synthesis update direction of the shared parameters conflicting with the gradient directions of the existing modes, so that each step update of the shared parameters simultaneously carries the real gradient constraints of all existing modes. This is different from existing continuous learning methods that use approximations to replace the real gradients, such as elastic weight solidification, knowledge distillation and generative replay. By replacing the diffusion reconstruction loss, which is extremely insensitive to control capability degradation, with control-following error probes specific to each modality (absolute trajectory error and relative pose error of the camera trajectory modality, component-normalized L2 distance of the robot operation modality, and normalized two-dimensional joint position error and joint prediction hit rate of the human hand motion graph modality), as the feedback signal, and performing least-squares linear fitting on the relative degradation rate sequence within the sliding window to obtain the slope to suppress single-point noise interference, the determination of degradation trend is moved from evaluation after training to periodic online observation during the training process. This invention uses two independent adjustment knobs, sampling ratio and loss weight, to intervene in the frequency of generating true gradients on shared parameters for each modality and the effective update steps of each modality's dedicated expert, forming a closed-loop correction mechanism for the degradation of existing modalities. The static prior ratio is replaced with dynamic adaptive adjustment based on control fidelity, so that the entire expansion process can maintain a controlled balance between new modality learning and existing modality maintenance without manual intervention.

[0027] In one embodiment, such as Figure 3 As shown, step S10 specifically includes the following steps: S11: Divide the model parameters in the first embodied world model into cross-modal shared parameters and modality-specific parameters; S12: Based on cross-modal shared parameters and modal-specific parameters, a fixed probe set is constructed, and deterministic inference is performed on all existing control modes in the fixed probe set using exponential moving average weights to obtain the first generated video of each existing control mode. S13: Input the first generated video of each existing control mode into the corresponding control follow-up error calculation process to obtain the first control follow-up error of each existing control mode.

[0028] In this embodiment, the network structure tree, parameter names, tensor shapes, and forward computation dependencies of the first embodied world model are read. A parameter partitioning index table is then established based on whether parameters are jointly invoked by different control modes. Parameters from the self-attention layer, cross-attention layer, AdaLN modulation projection layer, patch-embed layer, and time-step embedding layer in the diffusion transformer backbone are included in the backbone shared parameters. The shared expert parameters from the two fully connected layers with an intermediate dimension four times the model's hidden layer dimension are included in the expert shared parameters. Simultaneously, the parameters from the pre-mapping layer in the router, which maps from the model's hidden layer dimension to 256-dimensional features, are included in the routing shared parameters. This forms a cross-modal shared parameter set, i.e. For the camera trajectory mode, robot operation mode, and human hand motion graph mode, corresponding modality-specific parameter sets are established. Each modality-specific parameter set includes dedicated expert parameters with an intermediate dimension twice the hidden layer dimension of the model, as well as control path parameters corresponding to the mode. For example, the dedicated parameters for the camera trajectory mode include a one-dimensional convolutional layer and two fully connected adapter layers. The dedicated parameters for the robot operation mode include two fully connected networks used for motion tensor encoding. The dedicated parameters for the human hand motion graph mode include a linear projection layer used for feature dimension alignment after the frozen variational autoencoder. The router output header registers experts line by line and associates the output lines corresponding to each dedicated expert with the corresponding modality-specific parameters.

[0029] In this embodiment, before the first embodied world model is connected to the new control modality, all model weights, optimizer states, and exponential moving average weights are saved to form a baseline snapshot before expansion. For example, 32 representative probe samples are fixedly selected from the validation data of each existing control modality. These probe samples should cover different control amplitudes, motion directions, action speeds, and scene complexities. Simultaneously, a sample identifier, input control conditions, video generation prompts, and a reference control sequence are fixed for each probe sample to form a fixed probe set. Then, the attenuation coefficient of the exponential moving average weights is set to, for example, 0.9999. During inference, the first embodied world model is switched to the evaluation state, and random deactivation, random depth, random data augmentation, and other random perturbation operations are disabled. Noise seeds are fixed for each of the 32 probe samples. From 0 to 31, the diffusion sampler type and sampling step number are fixed, for example, the sampling step number is set to 50 steps, and the video frame number, resolution and condition input order are kept completely consistent. This ensures that the generation difference of the same probe sample at different evaluation times mainly comes from the change of model parameters, rather than from the initial diffusion noise or sampling process fluctuations. According to the control mode identifier, each probe sample is input into the corresponding dedicated control path, and the router performs normalization and sparse activation only within the qualification set composed of shared experts and dedicated experts of the current control mode. This ensures that each sample only calls the shared experts and their corresponding dedicated experts. After a fixed deterministic diffusion inference process, the first generated video of each existing control mode is obtained. At the same time, the generated video is bound and stored with the corresponding probe sample identifier, control conditions, noise seed and baseline snapshot version.

[0030] In this embodiment, inverse kinematics of control and error calculation are performed according to different control modes. For the camera trajectory mode, the first generated video is input into a fixed version of the DROID-SLAM or ParticleSfM pose estimator to obtain the camera pose sequence corresponding to the generated video. The camera pose sequence is then aligned with the conditional trajectory in the probe sample using Umeyama similarity transformation to eliminate scale ambiguity and global rigid body transformation differences caused by monocular reconstruction. Subsequently, the root mean square value of the absolute trajectory error and the relative pose error are calculated. The absolute trajectory error is in meters, and the relative pose error is recorded by recording the translation and rotation components to avoid directly adding indices with different dimensions. For the robot operation mode, adjacent video frames are input into an inverse dynamics model that has been pre-trained on real data and frozen during the expansion phase. The inverse dynamics model regresses the position of the end effector. The robot estimates pose, joint movements, and gripper opening / closing states, and matches the motion estimation results with the conditional motion sequences in the probe samples time-by-time. After normalization according to the standard deviation of the training set, the L2 distance of the motion is calculated. The end-effector pose and gripper opening / closing terms are weighted with weights of 0.8 and 0.2, respectively, to obtain the first control following error of the robot operation mode. For the human hand motion map mode, the first generated video is input into a fixed version of the HaMeR keypoint reconstruction model. The two-dimensional coordinates of 21 hand joints in each frame are extracted and compared with the expected joint coordinates obtained by soft-argmax of the conditional heatmap. The scale is normalized by the diagonal length of the hand bounding box, and the normalized two-dimensional joint position error and PCK@0.05 are calculated. PCK@0.05 represents the proportion of joints whose joint prediction error does not exceed 5% of the diagonal of the hand bounding box. The error indices obtained from each existing control mode are saved as the first control following error record according to the mode. The camera trajectory mode retains the absolute trajectory error and relative pose error. The robot operation mode saves the normalized motion distance. The human hand motion graph mode saves the normalized two-dimensional joint position error and joint prediction hit rate. These records are then bound to the fixed probe set version and the extended baseline snapshot.

[0031] In one embodiment, such as Figure 4 As shown, step S20 specifically includes the following steps: S21: Based on the weight matrix of the shared expert two-layer fully connected network in the extended baseline snapshot, calculate the input projection weight matrix and output projection weight matrix of the newly added special expert, set all elements of the output projection weight matrix to zero, and connect scalar gating parameters with initial values ​​of zero in the residual branch of the newly added special expert so that the newly added special expert can complete the hot start initialization. S22: Perform zero initialization on the weights and biases of the last linear layer of the newly added control path, connect the newly added dedicated expert and the newly added control path to the first embodied world model, perform deterministic inference on all existing control modes and construct the second embodied world model in combination with the first control following error.

[0032] In this embodiment, the input projection weight matrix and output projection weight matrix of the shared expert's two-layer fully connected network are read from the pre-expansion baseline snapshot. The intermediate dimension of the shared expert is four times the hidden layer dimension of the model, while the intermediate dimension of the newly added specialized expert is twice the hidden layer dimension of the model. Therefore, the two sets of weight matrices of the shared expert cannot be directly copied to the newly added specialized expert. Instead, forward computation is performed on fixed existing control modality calibration samples using the pre-expansion exponential moving average weights to record the activation vectors of each intermediate neuron of the shared expert. The input projection weight matrix and output projection weight matrix of the shared expert's two-layer fully connected network are read from the pre-expansion baseline snapshot… Instead, forward computation is performed on fixed existing control modality calibration samples using the pre-expansion exponential moving average weights to record the activation vectors of each intermediate neuron of the shared expert on all calibration samples. The mean activation amplitude of each intermediate neuron on all calibration samples is calculated, and the importance score is calculated by combining this mean activation amplitude with the weight norms of its corresponding input projection row and output projection column. The calculation relationship used is as follows: ,in, Indicates shared expert number The importance score of each interneuron This represents the mean activation amplitude of the k-th interneuron across all calibration samples. This represents the input projection weight matrix of the shared expert. The output projection weight matrix of the shared expert is represented. The intermediate neurons of the shared expert are sorted from high to low according to their importance scores, and the top 2d neurons are selected, where d is the hidden layer dimension of the model. The corresponding rows are extracted from the input projection weight matrix of the shared expert to form the input projection weight matrix of the new specialized expert. At the same time, the columns corresponding to the same neurons are extracted from the output projection weight matrix of the shared expert to form the dimension-matched output projection weight matrix of the new specialized expert. This allows the input side of the new specialized expert to inherit the general world representations such as lighting, geometric structure, material changes and rigid body motion that the shared expert has already learned, instead of starting training from random parameters.

[0033] In this embodiment, all elements of the output projection weight matrix of the newly added dedicated expert in the actual access model are set to zero, while the warm-start parameters in the input projection weight matrix are retained, and a trainable scalar gating parameter is connected in series on the residual branch of the newly added dedicated expert. ,Will The initial value is set to 0, so that when the new dedicated expert is added in step 0 of the expansion, even if it receives the input hidden state, it will not output a non-zero residual to the diffusion transformer backbone. After training starts, it will proceed according to... To enable linear opening of the gating parameters, for example, the opening period can be adjusted. Set the training steps to 2000 and set the gating limit. Setting it to 1 allows the contribution of newly added specialized experts to the backbone to gradually increase from zero, preventing excessively large output amplitudes and shared parameter gradients from emerging experts in the early stages of training. A corresponding lightweight control path is established based on the data format of the new control modality. For example, when the new control modality is a humanoid robot's full-body motion, the joint positions, joint velocities, and base motion parameters can be mapped to the hidden layer dimension of the model through an action encoding network. When the new control modality is a UAV's six-DOF motion, position, attitude, and velocity sequences can be encoded through a one-dimensional convolutional layer and a fully connected adapter. The weight matrix and bias vector of the last linear layer of the new control path are all set to zero, ensuring that the conditional features injected into the diffusion transformer backbone at the initial access moment are zero vectors. If the new control path uses a channel concatenation method of conditional latent variables and video latent variables, the input channels of the shared patch-embed layer need to be expanded, and the convolutional weights corresponding to the new input channels are all set to zero. Simultaneously, when inputting existing control modality samples, all-zero tensors with the same size as the new conditional latent variables are used to fill the new channels, preventing structural perturbations to the existing control modality caused by changes in input dimensions. Add an output line corresponding to the newly added dedicated expert to the router output header, set the weight and bias of the newly added output line to zero, and keep the parameters in the extended baseline snapshot unchanged in the 256-dimensional shared mapping layer in front of the router. Also, stipulate that the qualification set of the existing control mode sample still only includes the shared expert and its own dedicated expert, so that the routing value of the newly added dedicated expert does not enter the normalized denominator of the existing control mode sample. After completing the access of the newly added dedicated expert, the newly added control path, the newly added patch-embed channel, and the router output line, the fixed probe set, fixed initial noise seed for diffusion, fixed sampler, fixed sampling steps, and exponential moving average weight before expansion used in step S12 are loaded. Deterministic inference is re-executed for all existing control modes to obtain the second generated video of each existing control mode. The second generated video is then compared with the first generated video tensor by tensor. When the second generated video of all existing control modes is consistent with the first generated video at the bit level, it indicates that the function output of the existing control modes has not been changed during the access of the new structure. If any generated video is inconsistent, the output projection of the newly added dedicated expert, the gating parameters, the last layer of the newly added control path, the newly added patch-embed channel, and the newly added router output line are checked in sequence to see if there are any non-zero elements. After successful verification, the extended network structure, zero-disturbance initialization parameters, existing control mode qualification set, fixed probe set configuration, and the first control following error of each existing control mode are written into the model extension state record. The first control following error is used as the baseline for calculating the relative degradation rate and is not directly modified in the model parameters during this access process, thus obtaining the second embodied world model.It should be noted that the output projection weight matrix and the gating parameter α are set to zero at the start of the expansion. This only ensures that the contribution of the newly added dedicated expert to the forward residual generated by the diffusion converter backbone in step 0 of the expansion is zero, and does not mean that the dedicated expert cannot obtain effective learning signals in subsequent training. Since the input projection weight matrix of the newly added dedicated expert uses the non-zero parameters obtained from the hot start based on the importance ranking of shared experts, the new control mode samples can still obtain non-zero intermediate activations after input projection. The gradient of the loss function with respect to each element of the output projection weight matrix depends on the product of the intermediate activation and the gradient with respect to the output residual, and is independent of whether the output projection weight matrix itself is currently zero. Therefore, the output projection weight matrix can obtain non-zero gradients from the first step of training and is gradually updated to non-zero parameters by the optimizer. Similarly, the gradient of the gating parameter α depends on the product of the non-zero output (before being zeroed) of the newly added dedicated expert and the gradient with respect to the residual contribution. This is also independent of the current value of α. Therefore, α can immediately obtain a valid non-zero gradient after training begins and gradually increase according to the aforementioned linear activation mechanism α(t) = min(t / T_heat, 1)α_. Thus, the zero initialization of the output projection weight matrix and the gating parameter α is only used to ensure that the function output of the existing control mode is not disturbed during the expansion access, and does not block the newly added dedicated expert from receiving the training gradient of the new control mode, ensuring that the new module can be trained normally.

[0034] Those skilled in the art should understand that once this round of expansion is completed and the training of the dedicated experts and control pathways for the new control mode converges, the second embodied world model can serve as the first embodied world model in the next expansion of the control mode, and the above steps S10 to S40 can be re-executed to access more new control modes, thereby enabling this method to support continuous and multiple expansions of the embodied world model control modes.

[0035] In one embodiment, such as Figure 5 As shown, step S30 specifically includes the following steps: S31: Based on the current sampling ratio, perform hierarchical sampling of all control modes within each computing node and input them into the MoE layer of the second embodied world model to obtain the grouped forward computation results of the MoE layer; S32: Obtain the original gradients of the new control mode gradient vector and the existing control mode gradient vectors applied to the cross-modal shared parameters, respectively. Calculate the L2 moving average of the gradient vectors of each control mode, and use the median of the L2 moving average of all control mode gradients as a unified reference scale to scale each control mode gradient vector to obtain the norm-equilibrium new control mode gradient vector and the norm-equilibrium gradient vectors of each existing control mode. Based on the grouped forward computation results of the MoE layer, perform a projection resolution operation on the norm-equilibrium new control mode gradient vector to obtain the conflict-resolved new control mode gradient vector. S33: Based on the new control mode gradient vector after conflict resolution and the existing control mode gradient vectors after norm equalization, perform gradient clipping and optimizer step update on the cross-modal shared parameters to obtain the shared parameter update result of the current training step.

[0036] In this embodiment, after the second embodied world model completes the zero-disturbance access of the newly added dedicated experts and the newly added control paths and passes the consistency verification of the outputs of the existing control modes, it enters the multimodal joint training process of the current training step. Independent data loaders are configured for all existing control modes and the new control mode, and the current sampling ratio generated in the previous evaluation cycle is read. Before the first dynamic adjustment is completed, the initial sampling ratio of each control mode can be set to a uniform ratio. That is, when there are M existing control modes and one new control mode, the initial sampling ratio of each control mode is set to 1 / (M+1). Subsequently, within each computing node, samples are extracted from all data loaders according to the same sampling ratio. To obtain a sufficient number of real samples, for example, the batch size of each computing node can be set to 64, and the number of samples in each sub-batch can be determined according to the sampling ratio of each control mode. At the same time, the difference in the number of samples caused by rounding is supplemented according to the sampling ratio from high to low to ensure that the sum of the number of samples in all sub-batch is kept at 64. Then, a control mode identifier is attached to each sample and concatenated in the batch dimension to form a global batch, instead of distributing different control modes to different computing nodes. This ensures that each computing node contains the gradient components of all control modes before performing gradient aggregation, and avoids the situation where a certain expert sample appears only in some computing nodes, causing the effective learning rate of that expert to be scaled unequally by the distributed averaging operation. The global batch is input into the diffusion transformer backbone of the second embodied world model to obtain the hidden state carrying the control mode identifier. In the MoE layer, the tokens corresponding to the hidden state are grouped according to the control mode identifier. Tokens of the same control mode are organized into continuous data segments and then input into the shared expert and the corresponding control mode dedicated expert to perform matrix multiplication calculation. The router only calculates the routing weight within the qualification set composed of the shared expert and the current control mode dedicated expert. The output of each group is weighted according to the routing weight and then backfilled into the global position according to the original token index, so as to obtain the forward calculation result of the group covering all control modes. Each token only activates two experts: the shared expert and the dedicated expert of this control mode. When the video memory cannot accommodate the conditional paths of all control modes at the same time, each control mode sub-batch can be divided into micro-batches to perform forward and backward propagation. However, after all micro-batches have completed gradient accumulation, an optimizer step update is performed uniformly. Cross-modal shared parameters should not be updated separately at the end of each micro-batch.To prevent control modes with low sampling ratios from being rounded to zero during batch size rounding, a minimum sample size limit is pre-set for each control mode, for example, set to 1. After calculating the number of samples allocated to each control mode according to the current sampling ratio and batch size of 64 and performing rounding, for control modes whose rounding result is less than the minimum sample size limit, they are first padded to the number of samples required to reach the minimum sample size limit. A corresponding number is then deducted proportionally from control modes with a larger number of rounded samples that exceeds their own minimum sample size limit, ensuring that the sum of the sample sizes in all sub-batches remains 64. This ensures that, within the allowable range of batch size and sampling ratio, each control mode obtains at least one true sample at each training step in each computation node, preventing control modes with low sampling ratios from failing to generate true gradients on cross-modal shared parameters for extended periods due to rounding issues.

[0037] In this embodiment, the modal loss is calculated based on the diffusion denoising objective corresponding to each control mode, and backpropagation is performed on each modal loss individually to read its gradient vector acting on the cross-modal shared parameters. The gradient vector of the first modal loss is then used to... The shared parameter gradient of each control mode is denoted as... , Indicates the first Loss of each control mode, This represents the set of cross-modal shared parameters, including the diffusion converter backbone, shared experts, and router front-end mapping layer. To prevent new control modes from generating gradient norms significantly higher than existing control modes due to large losses in the early training stages, the L2 norm moving average of the gradient vectors of each control mode can be calculated separately. For example, the actual L2 norm of the gradient of each control mode is remeasured every 100 training steps, and the moving average decay factor is set to 0.9. In the remaining training steps, the scaling factor obtained most recently is used. The median of the moving average of the L2 norm of the gradients of all control modes is determined as a unified reference scale, and the gradient vectors are scaled according to the moving average of the L2 norm of the gradients of each control mode, with the scaling factor limited to between 0.25 and 4.0. When the moving average of the L2 norm of the gradient of any control mode is lower than 1 × 10⁻⁶, the scaling factor is considered to be lower. -8 At this time, no amplification processing is performed on its gradient vector, thus obtaining the gradient vectors of each existing control mode after norm equalization and the gradient vector of the new control mode after norm equalization. The direction cosine value is calculated for each of the new control mode gradient vector and each existing control mode gradient vector. When the direction cosine value is greater than or equal to 0, it indicates that there is no conflict between the two gradient directions, and the current new control mode gradient vector is retained unchanged. When the direction cosine value is less than 0, it is processed according to... Remove the gradient of the new control mode that is related to the first There are several projection components that conflict with the gradient directions of existing control modes, among which... This represents the gradient vector of the new control mode to be processed. Indicates the first The norm-equilibrium gradient vector of each existing control mode is obtained. When there are multiple existing control modes, the processing order of the existing control modes can be randomly shuffled in each training step and then the projection resolution can be completed sequentially to avoid the fixed projection order biasing a certain existing control mode for a long time. After the sequential projection of all existing control modes, the gradient vector of the new control mode after conflict resolution is obtained.

[0038] In this embodiment, the gradient vector of the new control mode after conflict resolution is weighted and synthesized with the norm equilibrium gradient vectors of each existing control mode according to the current loss weight and sampling ratio, so that the gradient received by the cross-modal shared parameters remains unchanged. ,in, Indicates the first The current sampling ratio of each control mode. This represents the current loss weight. Since the sampling ratio of each existing control mode is always greater than 0, the synthesized gradient at each training step includes the true gradient constraints generated by the real data of the existing control modes, rather than approximating the existing control capability solely using parameter anchoring terms or distillation results. After gradient synthesis, a global L2 pruning is performed on the synthesized gradients of cross-modal shared parameters. For example, the pruning threshold can be set to 1.0. When the L2 norm of the synthesized gradient exceeds 1.0, the gradients of all shared parameters are proportionally reduced without changing the gradient direction. Then, the AdamW optimizer is used to perform a unified parameter update. For example, the learning rate of the cross-modal shared parameters can be set to 1×10. -5 The weight decay coefficient is set to 0.01, and the first-order moment estimate, second-order moment estimate, and exponential moving average weights corresponding to the shared parameters are updated synchronously after the optimizer step. At the same time, each control mode expert only receives the gradient generated by the sample of its own control mode and completes the update according to its own parameter group. It does not receive the reverse gradient generated by other control modes. In this way, while maintaining the structural isolation of the dedicated parameters, the update direction of the cross-modal shared parameters takes into account both the learning needs of the new control mode and the maintenance needs of all existing control modes. Finally, the cross-modal shared parameters, optimizer state, and exponential moving average parameters updated after the optimizer step are recorded together as the shared parameter update result of the current training step.

[0039] In one embodiment, step S31 specifically includes the following steps: Based on the current sampling ratio, hierarchical sampling is performed on the independent data loaders of all control modes within each computing node to obtain a global batch carrying control mode identifiers; Input the global batch into the diffusion transformer backbone of the second embodied world model to obtain the token hidden state grouping of each control mode; The hidden state packets of each control mode are grouped and input into the router. The router performs mean pooling on the hidden state vector of each token and then performs front trunk mapping to obtain the grouped forward computation results of the MoE layer.

[0040] In this embodiment, an independent data loader is set up for each existing control mode and each new control mode. At the beginning of each training step, the current sampling ratio is read, and the number of sub-batch samples that each data loader should extract in the current computing node is determined according to the sampling ratio corresponding to each control mode. For example, the batch size of a single computing node is set to 64. When the model includes three existing control modes and one new control mode, 64 samples can be allocated to four data loaders according to the current sampling ratio. The difference in the number of samples after rounding is supplemented according to the sampling ratio from high to low, so that the sum of the number of sub-batch samples of each control mode is always equal to 64. When extracting samples, each data loader also performs intra-layer random sampling according to video length, control sequence length, or scene complexity, thereby avoiding the concentration of samples of a single length or a single difficulty in the same sub-batch. Each sample is attached with a control mode identifier that can uniquely represent the camera trajectory mode, robot operation mode, human hand motion map mode, or new control mode. Then, the sub-batch of all control modes is concatenated along the batch dimension to form a global batch. For data-parallel training, each computing node independently executes the same hierarchical sampling process, instead of assigning different control modes to different computing nodes. This ensures that each computing node contains samples of all control modes before performing gradient aggregation, avoiding inconsistencies in the effective learning rate after distributed averaging caused by a dedicated expert only generating gradients on some computing nodes.

[0041] In this embodiment, the video latent variables, diffusion time steps, text conditions, and corresponding control conditions in the global batch are uniformly input into the diffusion transformer backbone of the second embodied world model. The data are then processed sequentially through the patch-embed layer, self-attention layer, cross-attention layer, and AdaLN modulation to obtain the token hidden state corresponding to each sample. At the same time, the control mode identifier and original position index of the sample to which each token belongs are retained. Based on the control mode identifier, the token hidden state is reorganized into multiple consecutive hidden state groups using sorting or index_select operations, so that tokens of the same control mode are located in the same data segment. The hidden states of tokens for each control modality are grouped and sent to the router. For the hidden states of tokens within the same sample, mean pooling is performed along the token dimension to obtain the mean vector of hidden states that can represent the overall scene content and control condition response state of the sample. The mean vector of hidden states is then input and mapped from the hidden layer dimension of the model to the 256-dimensional front trunk. After linear mapping and activation processing, routing features are obtained. Combined with the control modality identifier, the expert qualification set of the current sample is generated. The qualification set of existing control modality samples only includes shared experts and dedicated experts corresponding to their control modality. The qualification set of new control modality samples only includes shared experts and newly added dedicated experts. The router normalizes expert scores only within the corresponding qualification set to obtain shared expert routing weights and mode-specific expert routing weights, and broadcasts the routing weights of the same sample to its token hidden state group. Continuous token data segments from each control mode are input to the shared expert and the corresponding mode-specific expert for matrix multiplication. The outputs of the two experts are weighted and summed according to their routing weights, and then backfilled into the global token sequence using a scatter operation based on the previously saved original location indexes, thus restoring the MoE layer output consistent with the input order of the diffusion transformer backbone. Through this grouping and merging method, a single global batch can cover all control modes. Moreover, each token only activates the shared expert and its own control mode-specific expert in a single MoE layer forward computation; other control mode-specific experts neither participate in the forward computation of that token nor enter the corresponding routing normalization range. The final result is a MoE layer grouped forward computation that simultaneously retains mode grouping information, routing weights, expert outputs, and the original token location mapping relationship.

[0042] In one embodiment, such as Figure 6 As shown, step S40 specifically includes the following steps: S41: Based on the shared parameter update result, deterministic reasoning is performed on all existing control modes and new control modes respectively to obtain the third generated video of each control mode. The third generated video of each control mode is then input into the corresponding control following error calculation process to obtain the second control following error of each control mode. The second control following error of each existing control mode is compared with the first control following error solidified before expansion. The first control following error of the new control mode is solidified in the first evaluation cycle after its generated video meets the preset effective generation conditions, and the solidified result is used as the comparison baseline in each subsequent evaluation cycle. S42: Calculate the relative degradation rate of the second control following error and the first control following error for each control mode, and determine the degradation trend of each control mode based on the relative degradation rate; S43: Based on the degradation trend and relative degradation rate of each control mode, calculate the sampling ratio and loss weight for the next evaluation period.

[0043] In this embodiment, the control fidelity evaluation cycle is set to be triggered once every 2000 training steps. After the evaluation is triggered, the optimizer update of cross-modal shared parameters and modality-specific parameters is paused. The shared parameter update result obtained in the current training step is synchronized to the exponential moving average weight copy. Then, the same fixed probe set used in step S12 is called to perform deterministic inference on all existing control modes and new control modes. The fixed probe samples, control conditions, video cue information, initial noise seed for diffusion, sampler type, number of sampling steps, number of video frames, and output resolution remain unchanged. The second embodied world model is switched to the evaluation state, and random deactivation, random depth, and random data augmentation are turned off, so that the generation differences between different evaluation cycles are mainly caused by model parameter updates. For existing control modes, the first control following error solidified before expansion is directly used as the baseline. For new control modes, the first control following error of the new control mode can be solidified according to the same probe protocol in the first evaluation after the addition of a dedicated expert and the addition of a control path that can generate effective conditional videos. Subsequently, the same baseline is used for comparison in each evaluation cycle. After completing deterministic reasoning, the third generated videos generated by each control mode are input into the corresponding control following error calculation process. The camera trajectory mode estimates the camera pose sequence using DROID-SLAM or ParticleSfM, aligns the estimated pose sequence with the conditional trajectory using Umeyama similarity transformation, and then calculates the root mean square value of the absolute trajectory error and the relative pose error. The robot operation mode regresses the motion estimation results from adjacent frames of the third generated video using an inverse dynamics model that remains frozen during the expansion period. The end-effector pose term and gripper opening / closing term are normalized according to the standard deviation of the training set, and the motion L2 distance is calculated with weights of 0.8 and 0.2, respectively. The hand motion graph mode extracts the two-dimensional coordinates of 21 hand joints using HaMeR, compares the extraction results with the soft-argmax expected coordinates of the conditional heatmap, calculates the two-dimensional joint position error normalized to the diagonal of the hand bounding box and PCK@0.05, thereby obtaining the second control following error of each control mode in the current evaluation cycle.

[0044] In this embodiment, the smaller the error value, the better the control and following performance. Calculate the first The relative degradation rate of each control mode, where, Indicates the current number The second control following error obtained from the second evaluation. This represents the first control following error of the corresponding control mode. A constant is set to 1×10 to prevent division by zero or numerical amplification caused by baseline error approaching zero. -6For indicators like PCK@0.05, where a larger value indicates better control performance, the positions of the current and baseline values ​​in the numerator are swapped to ensure that all relative degradation rates satisfy the condition that positive values ​​represent degradation and negative values ​​represent improvement. The control tracking error is calculated and averaged for multiple probe samples of the same control mode. For the camera trajectory mode, absolute trajectory error is used as the primary indicator for closed-loop adjustment, while relative pose translation error and relative pose rotation error are saved separately as auxiliary evaluation indicators. For the robot operation mode, the weighted motion L2 distance formed by the end effector pose, joint motion, and gripper opening / closing states is used as the closed-loop adjustment indicator. For the human hand motion graph mode, normalized two-dimensional joint position error is used as the primary indicator for closed-loop adjustment, while PCK@0.05 is saved separately as an auxiliary evaluation indicator. The current relative degradation rate of each control mode is calculated based on its primary indicator. To avoid misjudgments due to errors in video generation or pose inverse kinematics, a sliding window of length 5 is maintained for each control mode. The relative degradation rate of the last 5 evaluations is continuously stored. A least-squares linear fit is performed with the consecutive evaluation period number as the x-axis and the relative degradation rate of the corresponding evaluation period as the y-axis. The slope of the fitted line is taken as the degradation trend value, which represents the average change in the relative degradation rate within each evaluation period. When the degradation trend value is greater than 0.002 per evaluation period and the current relative degradation rate is greater than 0.05, the corresponding control mode is determined to exhibit a continuous degradation trend. When the absolute value of the relative degradation rate is greater than 0.05 and the degradation trend value is greater than 0.002 per evaluation period, the corresponding control mode is determined to exhibit a continuous degradation trend, and subsequent steps are performed. The obtained target sampling ratio and loss weight are updated. When the absolute value of the relative degradation rate is greater than 0.05 but the degradation trend value is not greater than 0.002 per evaluation period, the corresponding control mode is determined to be in a stable degradation state where the degradation level is already high but has not continued to expand. At this time, it is not placed in the adjustment dead zone, but the target sampling ratio and loss weight are calculated and updated according to its actual relative degradation rate in the subsequent steps to avoid the control mode with severe degradation but stable trend being misjudged as not needing adjustment. Only when the absolute value of the relative degradation rate does not exceed 0.05, regardless of the size of the degradation trend value, is the corresponding control mode placed in the adjustment dead zone to maintain its adjustment state in the previous evaluation period, thereby preventing the sampling ratio and loss weight from oscillating frequently due to normal error fluctuations.

[0045] In this embodiment, the average relative degradation rate of each control mode within the current evaluation period is nonnegated, that is, the average value of the relative degradation rate is made nonnegative. ,in Indicates the first The average relative degradation rate of each control mode within the current evaluation period is used to prevent control modes that have improved performance or have not degraded from being negatively reduced. During the period before the new control mode establishes its first control following error, a base adjustment is set for the new control mode to ensure it can continuously obtain the necessary training samples and gradient updates. After the new control mode establishes its first control following error, the fixed base adjustment is discontinued, and a corresponding adjustment is determined based on the actual relative degradation rate of the new control mode. Calculate the target sampling ratio for each control mode and the temperature coefficient. Set it to 0.5, then follow... The sampling ratio from the previous evaluation period is smoothed to the first order, where the smoothing coefficient is... The value is set to 0.3. Then, while ensuring the sum of the sampling ratios of all control modes equals 1, an iterative allocation with upper and lower bound constraints is performed on the sampling ratio of each control mode. The upper limit of the sampling ratio is set to 0.5, and the lower limit is the smaller of 0.05 and 0.5 divided by the total number of control modes. First, the control modes that reach the lower or upper limit of the sampling ratio are fixed. Then, the remaining sampling ratios are redistributed among the remaining control modes according to the target sampling ratio until the sampling ratios of all control modes simultaneously satisfy both the sum constraint and the upper and lower bound constraints. This prevents any control mode from completely losing the gradient of the true samples and prevents a single control mode from occupying too many training batches. Simultaneously, according to... Calculate the initial value of the loss weight, and then apply the amplification factor. Set to 2.0; perform mean normalization on the initial values ​​of the loss weights for each control mode, and then limit the normalized loss weights to between 0.5 and 3.0; if the upper and lower limit processing causes the mean of all loss weights to deviate from 1, then iteratively allocate the difference only among the loss weights that have not yet reached the 0.5 or 3.0 boundary, until the final loss weights are all between 0.5 and 3.0 and the mean of all loss weights is 1, so as to avoid the total loss scale implicitly changing the effective learning rate of the optimizer as the degradation state changes. The normalized sampling ratio is written into the hierarchical sampling configuration of the independent data loader for each control mode, and the normalized loss weight is written into the loss calculation configuration of the corresponding mode. The fixed probe set version, the second control following error, the relative degradation rate, the trend value, the sampling ratio, and the loss weight are saved together as the adjustment state for the next evaluation cycle. The sampling ratio is used to change the frequency of generating real shared gradients in each control mode and the effective update steps of each dedicated expert, while the loss weight is used to change the influence intensity of the individual sample gradient on the update direction of the shared parameters, thereby forming an adjustment process with control following performance as the feedback signal.

[0046] like Figure 7As shown, in one embodiment of the present invention, a network structure diagram of an embodied world model is presented. Multiple conditional sequences, such as text, visual frames, hand skeleton diagrams, camera trajectories, and teleoperation control, are mapped by their respective encoders and then input together with time codes into the backbone of a unified diffusion transformer. This backbone is composed of layer normalization, multi-head self-attention, conditional cross-attention, and a control-aware hybrid expert feedforward network connected in series. The hybrid expert feedforward network includes a control modality router and a trainable sparse gated router, and schedules shared experts as well as dedicated experts for each modality, such as mobile experts, hand control experts, and teleoperation experts. The outputs of each expert are weighted and summed before being sent to the video decoder to obtain the predicted video frames.

[0047] like Figure 8 As shown in the diagram, this invention provides a structural comparison of the addition of a dedicated expert and a new control path before and after accessing the embodied world model in one embodiment. Before the expansion, the existing control path is connected to the diffusion converter backbone and the router in sequence. With the support of cross-modal shared parameters and dedicated parameters for each modality, the router schedules the shared expert and existing expert A and existing expert B. After the expansion, a new control path is added on the basis of the existing control path, and the router adds experts accordingly. The new control path and the new experts are marked with dashed boxes in the figure to show that the original inference path of the existing control mode is not changed when the new structure is accessed.

[0048] like Figure 9 As shown, this is a schematic diagram of a closed-loop adjustment mechanism of a gradient isolation-based embodied world model control modality continuous expansion method in one embodiment of the present invention. The outputs of existing and newly added modalities in the embodied world model are evaluated by control performance to obtain relative degradation rates and degradation trends, and the sampling ratio and loss weight for the next evaluation cycle are generated accordingly to guide training and updates. The training and update results are fed back to the embodied world model to form a closed loop.

[0049] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0050] In one embodiment, a gradient-isolation-based embodied world model control modal persistence extension device is provided, which corresponds one-to-one with the gradient-isolation-based embodied world model control modal persistence extension method described in the above embodiments. For example... Figure 10 As shown, the gradient-isolated embodied world model control modality continuous expansion device includes: Error evaluation module 701 is used to divide the model parameters in the first embodied world model into cross-modal shared parameters and modal-specific parameters, and to perform control follow-up error evaluation on all existing control modes respectively to obtain the first control follow-up error of each existing control mode; The access module 702 is used to connect the newly added dedicated expert and the newly added control path to the first embodied world model and generate the second embodied world model based on the first control following error; The parameter update module 703 is used to perform projection resolution on the gradient vector of the new control mode and the gradient vector of each existing control mode based on the second embodied world model, and update the cross-modal shared parameters to obtain the shared parameter update result of the current training step. The calculation module 704 is used to calculate the relative degradation rate of each control mode based on the update results of shared parameters and generate the sampling ratio and loss weight for the next evaluation period.

[0051] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0052] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0053] This application also provides a computer device, such as... Figure 11 As shown, the computer device includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. When the processor executes the computer program, it implements the steps in any of the above method embodiments, or when the processor executes the computer program, it implements the functions of each module / unit in the above device embodiments.

[0054] For example, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device.

[0055] Those skilled in the art will understand that Figure 11 The computer device described is merely an example and does not constitute a limitation on the computer device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0056] The aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0057] The memory can be an internal storage unit of the computer device, such as a hard drive or RAM. The memory can also be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units of the computer device.

[0058] This application also provides a readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.

[0059] This application provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps described in the various method embodiments above.

[0060] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a computer device or embodied world model training device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0061] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0062] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0063] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0064] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0065] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for controlling modal persistence expansion based on gradient isolation embodied world model, characterized in that, include: The model parameters in the first embodied world model are divided into cross-modal shared parameters and modal-specific parameters. Control follow-up error evaluation is performed on all existing control modes to obtain the first control follow-up error of each existing control mode. The newly added dedicated experts and the newly added control channels are connected to the first embodied world model, and a second embodied world model is generated based on the first control following error. Based on the second embodied world model, projection resolution is performed on the gradient vector of the new control mode and the gradient vectors of each existing control mode, and the cross-modal shared parameters are updated to obtain the shared parameter update result of the current training step; Based on the updated results of the shared parameters, the relative degradation rate of each control mode is calculated and the sampling ratio and loss weight for the next evaluation cycle are generated.

2. The method for controlling modal persistence expansion based on gradient isolation of embodied world models as described in claim 1, characterized in that, The process involves dividing the model parameters in the first embodied world model into cross-modal shared parameters and modality-specific parameters, and then performing control following error evaluation on all existing control modes to obtain the first control following error for each existing control mode, including: The model parameters in the first embodied world model are divided into cross-modal shared parameters and modality-specific parameters. Based on the cross-modal shared parameters and the modality-specific parameters, a fixed probe set is constructed, and deterministic inference is performed on all existing control modes in the fixed probe set using exponential moving average weights to obtain the first generated video of each existing control mode. The first generated video of each existing control mode is input into the corresponding control follow-up error calculation process to obtain the first control follow-up error of each existing control mode.

3. The method for controlling modal persistence expansion based on gradient isolation of embodied world models as described in claim 2, characterized in that, The process of connecting the newly added dedicated expert and the newly added control path to the first embodied world model based on the first control following error to obtain the second embodied world model includes: Based on the weight matrix of the shared expert two-layer fully connected network in the extended baseline snapshot, the input projection weight matrix and output projection weight matrix of the newly added special expert are calculated. All elements of the output projection weight matrix are set to zero, and scalar gating parameters with initial values ​​of zero are connected in series in the residual branch of the newly added special expert to enable the newly added special expert to complete the hot start initialization. The weights and biases of the last linear layer of the newly added control path are initialized to zero. The newly added dedicated expert and the newly added control path are connected to the first embodied world model. Deterministic inference is performed on all existing control modes and the second embodied world model is constructed by combining the first control following error.

4. The method for controlling modal persistence expansion based on gradient isolation of embodied world models as described in claim 1, characterized in that, Based on the second embodied world model, projection resolution is performed on the gradient vector of the new control mode and the gradient vectors of each existing control mode, and the cross-modal shared parameters are updated to obtain the shared parameter update results for the current training step, including: Based on the current sampling ratio, hierarchical sampling is performed on all control modes within each computing node and input into the MoE layer of the second embodied world model to obtain the grouped forward computation results of the MoE layer. The original gradients of the new control mode gradient vector and each existing control mode gradient vector applied to the cross-modal shared parameters are obtained respectively. The L2 moving average of each control mode gradient vector is calculated, and the median of the L2 moving average of all control mode gradients is used as a unified reference scale to scale each control mode gradient vector to obtain the norm-equilibrium new control mode gradient vector and the norm-equilibrium gradient vectors of each existing control mode. Based on the grouped forward computation results of the MoE layer, a projection resolution operation is performed on the norm-equilibrium new control mode gradient vector to obtain the conflict-resolved new control mode gradient vector. Based on the new control mode gradient vector after conflict resolution and the existing control mode gradient vectors after norm equalization, gradient pruning and optimizer step update are performed on the cross-modal shared parameters to obtain the shared parameter update result of the current training step.

5. The method for controlling modal persistence expansion based on gradient isolation of embodied world models as described in claim 4, characterized in that, Based on the current sampling ratio, hierarchical sampling of all control modes is performed within each computing node and input into the MoE layer of the second embodied world model to obtain the grouped forward computation results of the MoE layer, including: Based on the current sampling ratio, hierarchical sampling is performed on the independent data loaders of all control modes within each computing node to obtain a global batch carrying control mode identifiers; The global batch is input into the diffusion transformer backbone of the second embodied world model to obtain the token hidden state grouping of each control mode; The hidden state groups of the tokens for each control mode are input into the router. The router performs mean pooling on the hidden state vector of each token and then performs front trunk mapping to obtain the forward calculation result of the MoE layer.

6. The method for controlling modal persistence expansion based on gradient isolation of embodied world models as described in claim 1, characterized in that, The step of calculating the relative degradation rate of each control mode and generating the sampling ratio and loss weight for the next evaluation period based on the updated results of the shared parameters includes: Based on the shared parameter update results, deterministic reasoning is performed on all existing control modes and new control modes to obtain the third generated video of each control mode. The third generated video of each control mode is then input into the corresponding control following error calculation process to obtain the second control following error of each control mode. The second control following error of each existing control mode is compared with the first control following error solidified before expansion. The first control following error of the new control mode is solidified in the first evaluation cycle after its generated video meets the preset effective generation conditions, and this solidified result is used as the comparison baseline in each subsequent evaluation cycle. Calculate the relative degradation rate of the second control following error and the first control following error for each control mode, and determine the degradation trend of each control mode based on the relative degradation rate; Based on the degradation trend and relative degradation rate of each control mode, the sampling ratio and loss weight for the next evaluation period are calculated.

7. A device for controlling modal continuous expansion based on gradient isolation embodied world model, characterized in that, The steps for implementing the gradient isolation-based embodied world model control modal persistence expansion method as described in any one of claims 1 to 6 include: The error evaluation module is used to divide the model parameters in the first embodied world model into cross-modal shared parameters and modal-specific parameters, and to perform control follow-up error evaluation on all existing control modes to obtain the first control follow-up error of each existing control mode. The access module is used to connect the newly added dedicated experts and the newly added control paths to the first embodied world model and generate a second embodied world model based on the first control following error. The parameter update module is used to perform projection resolution on the gradient vector of the new control mode and the gradient vector of each existing control mode based on the second embodied world model, and update the cross-modal shared parameters to obtain the shared parameter update result of the current training step. The calculation module is used to calculate the relative degradation rate of each control mode based on the update results of the shared parameters and generate the sampling ratio and loss weight for the next evaluation period.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the gradient isolation-based embodied world model control modality continuous expansion method as described in any one of claims 1 to 6.

9. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the gradient isolation-based embodied world model control modality continuous expansion method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the gradient isolation-based embodied world model control modality continuous expansion method as described in any one of claims 1 to 6.

Images