Robot action generation method integrating multi-layer feature bridging and world knowledge prediction

Abstract

The invention discloses a robot action generation method integrating multi-layer feature bridging and world knowledge prediction. The method comprises the following steps: extracting multi-layer middle layer visual features and action query hidden variables by utilizing a pre-trained visual language model; an initialization strategy based on robot body sensing state guidance is adopted, real-time pose priori is injected for action query, and traditional all-zero initialization is replaced; a spatial perception vector gating mechanism is introduced into the bridging attention module, and fine-grained selection of specific image region features is achieved; meanwhile, a world knowledge prediction task is integrated, and physical common knowledge is enhanced through explicit modeling environment depth, semantics and a dynamic region; and finally, replacing a traditional L1 regression action head with a diffusion model architecture, and generating an optimal action sequence under multi-modal distribution through an iterative denoising process. According to the method, the operation precision of the robot in a long-range complex task is improved, and the problem of control failure caused by an action'averaging effect 'is solved while the light weight of the model is kept.

Description

Technical Field

[0001] This invention belongs to the field of embodied intelligence and artificial intelligence, specifically relating to a method for generating robot actions that integrates multi-layer feature bridging and world knowledge prediction. Background Technology

[0002] Humans accomplish complex manipulation tasks through a closed-loop coordination of visual perception, physical intuition prediction, and fine motor feedback. For example, when grasping an object, the visual system identifies the target, the brain's internal physical engine simultaneously predicts the object's spatial depth and its motion tendency after being subjected to force, while the motor cortex adjusts muscle contraction based on the hand's real-time position (proprioceptive state). This "perception-prediction-execution" closed-loop mechanism enables humans to complete tasks such as "aligning with the microwave oven opening" or "precisely placing an object" with extremely high accuracy. However, existing robot vision-language-action (VLA) models still face significant challenges in simulating this mechanism.

[0003] Generally, current robot motion generation methods can be mainly divided into three categories: 1) Models based on large-scale pre-trained models: such as RT-2 and OpenVLA, which use visual language models (VLMs) with 7B or more parameters fine-tuned on large-scale internet and robot data. While these models possess strong language instruction understanding capabilities, the huge number of parameters leads to extremely high training costs and large inference latency, making high-frequency real-time deployment on consumer-grade graphics cards difficult. 2) Methods based on lightweight adapters: such as VLA-Adapter, VLA-OS, and Seer. These solutions reduce overhead by freezing the backbone network and introducing small adapter layers, but when faced with long-horizontal fine-grained manipulation tasks, the lack of explicit reconstruction of underlying features often limits motion accuracy. 3) Methods based on generative policies: such as Diffusion Policy, FlowVLA, and... These methods capture the multimodal distribution of actions through diffusion transformers or flow-matching, demonstrating better probability distribution modeling capabilities compared to traditional regression models.

[0004] While the aforementioned methods, based on large-scale pre-trained models, demonstrate significant performance, their feature extraction process often relies excessively on the last-layer hidden state of the VLM. Existing research (such as SpatialVLA and 4D-VLA) indicates that deep features of VLMs tend to favor high-dimensional semantics, while mid-layer features retain richer geometric structures, object topology, and spatial textures. Due to the lack of in-depth mining and bridging of multi-layer features, existing models often suffer from spatial misalignment when performing millimeter-level alignment tasks due to the loss of low-level visual information.

[0005] Furthermore, in the field of motion modeling, traditional L1 or MSE regression losses suffer from a severe "averaging effect." When multiple legitimate obstacle avoidance paths exist in the training data, the regression model tends to learn the statistical average of the trajectories, leading to the trained model tending to generate "intermediate paths" that collide with obstacles, or producing trajectory jitter during fine alignment. Recent works such as FlowVLA and FAST attempts to alleviate this problem by introducing flow matching or discrete diffusion, but how to efficiently conditionally couple it with the hierarchical features of VLM remains a challenge.

[0006] Meanwhile, regarding the understanding of physical laws, although works such as DreamVLA have attempted to introduce WorldKnowledge Prediction to simulate physical common sense, their diffusion conditional feature injection method cannot adaptively select the affordance region in the image. In scenes with densely packed objects and complex backgrounds, the model's attention is easily interfered with by irrelevant backgrounds, making it unable to accurately focus on key operational points. In addition, existing action query mechanisms mostly use blind all-zero initialization, completely ignoring the robot's real-time proprioceptive state prior feedback, causing the model to be in a "cold start" state during each frame of inference, and the action sequence lacks physical coherence in terms of temporal sequence.

[0007] Therefore, there is an urgent need for a robot motion generation method that can enable models to have stronger spatial perception and physical common sense understanding in long-range, high-precision tasks with fewer parameter increments, thereby improving the success rate and robustness of manipulation. Summary of the Invention

[0008] Purpose of the invention: This invention proposes a robot motion generation method that integrates multi-layer feature bridging and world knowledge prediction to solve the problem of motion trivialization caused by inaccurate positioning and L1 loss, thereby improving the accuracy and robustness of motion generation.

[0009] Technical Solution: The robot motion generation method integrating multi-layer feature bridging and world knowledge prediction described in this invention is implemented as follows:

[0010] (1) Multimodal data preparation and feature extraction: Obtain the robot's visual image, language commands and robot proprioceptive state at the current moment; use the pre-trained visual language model to extract multi-layer visual raw features and action query latent variables;

[0011] (2) Initialization of state-enhanced bridging query vector: The initial bridging query vector is generated by mapping the proprioceptive state through a multilayer perceptron and combining learnable time-step embeddings and random Gaussian noise.

[0012] (3) Spatial-aware bridging attention multi-layer feature fusion: Inject multiple hidden layer features of the visual language model into the bridging attention module, introduce a spatial-aware gating mechanism, control the feature injection intensity of different attention heads and different image positions through vectorized parameters, and extract multi-modal fusion feature vectors through multi-layer attention mechanism;

[0013] (4) Diffusion model action decoding: The multimodal fusion feature vector extracted in step (3) is used as the denoising condition of the diffusion model action head. Gaussian noise is transformed into a continuous action sequence through an iterative denoising process.

[0014] (5) Joint training of world knowledge: Predict future world knowledge in the latent space and calculate the world knowledge prediction loss to constrain the feature distribution during the training phase.

[0015] Furthermore, the visual image in step (1) includes a third-person perspective and a hand-eye perspective.

[0016] Furthermore, the state-enhanced bridging query vector initialization process described in step (2) is represented as follows:

[0017]

[0018] in, This is the initial bridging query vector. For data batch size, The length of the action sequence. For feature dimensions; Features of the robot's proprioceptive state; Represents a multilayer perceptron neural network; Position encoding is composed of learnable time-step embedding layers; This represents the temporal position index in the action sequence, and its range is... ; It is a random noise vector that follows a standard normal distribution.

[0019] Furthermore, the implementation process of step (3) is as follows:

[0020] Selected data are extracted in parallel from the visual language model. The features of the intermediate hidden layer are mapped to a unified low-dimensional embedding space through a projection layer.

[0021] Constructing the initial bridging query vector for the bridging attention module using the robot's proprioceptive state. Bridge query vector As the core control signal, it interacts in parallel with VLM features and proprioceptive state within the bridging attention module through visual cross-attention, action-state cross-attention, and action self-attention.

[0022] The bridging attention module concatenates and aggregates the calculation results of the three attention paths mentioned above along the feature dimension to generate the final multimodal fusion feature vector. :

[0023]

[0024] in, For visual cross-attention mechanism, This is an action-state cross-attention mechanism. For action self-attention mechanism, , , Refers to nonlinear projection functions. Visual features extracted by visual language models Refers to the latent variables of action queries extracted by the visual language model; The tangent activation function; For spatial perception gating parameters, For the number of heads, The number of image blocks.

[0025] Furthermore, the extraction selection The intermediate hidden layer features include: visual features that serve as environmental geometric constraints. And the hidden variables for querying actions as the intent of high-level tasks. .

[0026] Furthermore, the visual cross-attention implementation process is as follows:

[0027] by For querying, visual features from multiple hidden layers are used. Cross-attention is performed on the keys and values ​​to obtain the initial visual feature attention scores. Introducing spatial perception gating parameters To achieve fine-grained selective focus on different regions of an image; utilizing The activation function adjusts the weights of this output, enabling the visual language model to learn spatial weight masks for different operational tasks. The calculation formula is as follows:

[0028] ;

[0029] The gating mechanism independently assigns heterogeneous gating weights to each image block in the visual feature map, resulting in a processed visual feature attention score. This enables adaptive selection of features in task-related regions of an image and suppression of environmental background noise.

[0030] Furthermore, the action-state cross-attention implementation process is as follows:

[0031] by To query, retrieve the hidden variables from the action output of the VLM. Robot's proprioceptive state After concatenating the feature dimensions, they are used as key and value inputs; a mapping relationship between the command intent and the physical state is established to solve the control failure problem caused by the mismatch between the command and the real-time pose.

[0032] Furthermore, the self-attention process for the action is as follows:

[0033] For bridging query vectors Self-attention calculations are performed to enhance the temporal coherence of action sequences through internal interactions, ensuring that the generated trajectory conforms to physical smoothness constraints.

[0034] Furthermore, the denoising process described in step (4) employs a diffusion model to denoise the loss. Training, the loss function is expressed as:

[0035] .

[0036] Furthermore, the implementation process of step (5) is as follows:

[0037] The world knowledge vectors extracted from the visual language model are input into the prediction head, which consists of a lightweight projection layer.

[0038] The ambient optical flow information is extracted and spatial dilation is performed to construct a dynamic perception mask. This mask is then applied to the mean square error loss function of the future visual prediction task to achieve noise suppression of the static background and feature enhancement of the interactive area.

[0039] Based on scale-invariant logarithmic loss, dual-view depth maps are reconstructed in the latent space. By explicitly modeling 3D geometric constraints, the spatial distance sense missing in 2D features is compensated.

[0040] By introducing a visual basic model and using cosine similarity to achieve latent space semantic alignment, the visual language model's understanding of target attributes and topological boundaries is enhanced, ensuring the accuracy of objects generated by actions.

[0041] Define world knowledge prediction loss The difference loss between the predicted value and the real environment label is calculated and then compared with the denoising loss of the diffusion model. Perform weighted joint training:

[0042]

[0043] in, This is the balance coefficient.

[0044] Beneficial Effects: Compared with existing technologies, the present invention offers the following advantages: By introducing spatial perception vector gating into the bridging attention module, combining it with a bridging query vector initialization strategy guided by proprioceptive state, and utilizing a collaborative mechanism of capturing multimodal distributions of actions using a diffusion transformer, a robot motion generation method integrating a multi-layer feature bridging attention module and world knowledge prediction is formed. Placement accuracy is improved through DiT head and geometric feature injection. The diffusion model can capture complex motion distributions, solving the problems of "not releasing when in position" or "unstable gripping." The lightweight bridging attention module maintains a very small parameter scale, supporting real-time operation on consumer-grade graphics cards. Overall, it improves the robot's operational accuracy in long-range complex tasks, and while maintaining a lightweight model, it solves the control failure problem caused by the "averaging effect" of actions. Attached Figure Description

[0045] Figure 1 This is a flowchart of the present invention;

[0046] Figure 2 This is a visualization of the invention on the Libero dataset. Detailed Implementation

[0047] The present invention will now be described in further detail with reference to the accompanying drawings.

[0048] like Figure 1 As shown, this invention proposes a robot motion generation method that integrates multi-layer feature bridging and world knowledge prediction. The specific implementation process is as follows:

[0049] Step 1: Multimodal data preparation and feature extraction: Obtain the robot's visual image at the current moment. (Including third-person perspective and hand-eye perspective), language commands and the robot's proprioceptive state Extracting multi-layered raw visual features using a pre-trained visual language model (VLM). and action query hidden variables The robot's proprioceptive state has seven dimensions, including 3D translation (representing x / y / z spatial position), 3D rotation (roll / pitch / yaw), and 1D gripper (whether it is open or closed).

[0050] Step 2, Bridging Query Vector Initialization: To enhance the bridging query vector's perception of the current motion state, its initialization process incorporates the robot's proprioceptive state. Learnable location coding and random perturbations. The specific calculation is as follows:

[0051]

[0052] in, This is the initial bridging query vector. For data batch size, The length of the action sequence. For feature dimensions; Represents a multilayer perceptron neural network; Position encoding is composed of learnable time-step embedding layers; This represents the temporal position index in the action sequence, and its range is... ; It is a random noise vector that follows a standard normal distribution. This initialization strategy enables the bridging query vector to incorporate the current motion state before participating in attention calculation, thereby allowing for more accurate and efficient retrieval of relevant information from high-dimensional visual features.

[0053] Step 3: Spatial Awareness Bridged Attention Multi-Layer Feature Fusion: Using a Bridge Attention Module, guided by a bridging query vector infused with proprioceptive state, cross-modal alignment and refinement are performed on the multi-layered features of the Visual Language Model (VLM). This step deeply couples heterogeneous features from the intermediate layers of the VLM, including visual geometric representations and action query latent variables, with the robot's real-time physical state, thereby transforming high-dimensional latent spatial information into a multi-modal fusion feature vector that can accurately guide the diffusion strategy. The specific implementation process is as follows:

[0054] Multi-layer hidden state sampling and projection: Parallel extraction of selected data from VLM The intermediate hidden layer features are extracted. The extracted content includes: visual features serving as environmental geometric constraints. And the hidden variables for querying actions as the intent of high-level tasks. By using projection layers to map features from different levels to a unified low-dimensional embedding space, this process achieves preliminary semantic alignment of features across different levels.

[0055] A three-way parallel bridging attention interaction mechanism: This mechanism uses the initialized bridging query vector... As the core control signal, it interacts in three parallel paths with VLM features and proprioceptive features within the bridging attention module:

[0056] ①Visual Cross-Attention: ... For querying, visual features from multiple hidden layers are used. These are keys and values. To precisely control the level of environmental information injection, spatially aware gating parameters are introduced. ( For the number of heads, This allows for fine-grained selective focus on different regions (patches) of the image (based on the number of image patches). The activation function adjusts the weights of this output, enabling the model to learn spatial weight masks for different tasks. The calculation formula is as follows: For example, when performing a "grabbing" task, the gating mechanism automatically increases the weight of the blocks around the gripper and the area where the target object is located, while decreasing the weight of distant, irrelevant background areas. This per-patch feature refinement mechanism ensures that the conditional vector input to the Diffusion Model (DiT) contains a high signal-to-noise ratio, thereby improving the positioning accuracy under fine-grained operations.

[0057] ② Action-State Cross-Attention: To query, retrieve the hidden variables from the action output of the VLM. Robot proprioceptive characteristics After feature dimension concatenation, these are used as key and value inputs. This approach aims to establish a mapping between command intent and physical state, resolving control failure issues caused by mismatch between commands and real-time pose.

[0058] ③ Action Self-Attention: For bridging query vectors Self-attention calculations are performed to enhance the temporal coherence of action sequences through internal interactions, ensuring that the generated trajectory conforms to physical smoothness constraints.

[0059] Multimodal feature aggregation and output: The bridging attention module concatenates and aggregates the calculation results of the three attention paths to generate the final multimodal fusion feature vector. :

[0060]

[0061] in, , For cross-attention mechanism, For self-attention mechanism, , , Refers to nonlinear projection functions. Visual features extracted by visual language models This refers to the latent variables for action queries extracted by the visual language model. The tangent activation function. ( For the number of heads, (where is the number of image blocks) is the spatial perception gating parameter.

[0062] Multimodal fusion feature vector It encapsulates high-precision spatial geometry, mission intent, and physical priors as global guiding conditions for downstream diffusion Transformer heads, enabling optimal motion denoising under multimodal distributions.

[0063] Step 4, DiT Action Generation: The decoder uses a diffusion Transformer structure. During the training phase, noise is added to the real action sequence, and the model learns to fuse feature vectors using multiple modalities. To restore noise under certain conditions:

[0064]

[0065] in, This is a multimodal fusion feature vector extracted by the bridging attention module. During training, a resampling method is used, allowing DiT to observe multiple different noise levels for the same feature vector without increasing the forward overhead of VLM, thus stabilizing the gradient and accelerating convergence. During the inference phase, a smooth, continuous action sequence can be generated through 4-8 sampling steps.

[0066] Step 5, World Knowledge Prediction and Joint Training: To enhance the model's perception of the physical characteristics of the operational space, this step introduces an explicit world knowledge prediction branch. The true labels of future world knowledge are obtained during the training phase through the physics engine interface of the simulation environment, or generated offline through a discrete visual base model, thereby enhancing the model's physical common sense without increasing inference overhead. The specific implementation process is as follows:

[0067] Latent space feature projection: The world knowledge vector extracted by VLM is input into the prediction head, which consists of a lightweight projection layer.

[0068] Perform multi-dimensional physical property prediction; the prediction task covers the following three dimensions:

[0069] a) Dynamic region prediction: Extract ambient optical flow information and perform spatial dilation processing to construct a dynamic sensing mask. Apply this mask to the mean squared error loss function of the future visual prediction task to achieve noise suppression of the static background and feature enhancement of the interactive region.

[0070] b) Depth geometry reconstruction: Based on scale-invariant log loss (SiLogLoss), a dual-view depth map is reconstructed in the latent space. By explicitly modeling 3D geometric constraints, the spatial distance sense of missing 2D features is compensated, and the positioning accuracy of fine operations is improved.

[0071] c) Semantic Feature Alignment: A Visual Base Model (SAM) is introduced to achieve latent space semantic alignment through cosine similarity, which strengthens the model's understanding of target attributes and topological boundaries and ensures the accuracy of objects generated by actions.

[0072] Constructing a joint loss function: Defining the world knowledge prediction loss The world knowledge prediction loss is obtained by calculating the difference losses between dynamic region prediction, deep geometric reconstruction, and semantic feature prediction values ​​and ground truth (the three types of losses are calculated using MSE, SiLogLoss, and cosine similarity, respectively), and then weighted summing them. The world knowledge prediction loss and the diffusion model denoising loss from step four are then jointly trained using a weighted average:

[0073]

[0074] in, This is the balancing coefficient. This mechanism improves the representation quality of intermediate layer features through feature alignment without increasing inference costs, and enhances the model's robustness in high-precision tasks such as alignment and grasping by utilizing auxiliary supervision signals.

[0075] Figure 2The following are visualizations of the execution results of this method on four typical tasks in Libero simulations. In Task 1, facing a target bowl located in a half-open drawer, the method accurately identifies the object's 3D spatial pose, effectively avoids obstacles at the drawer edge while completing refined path planning, and finally places it stably in the center of the target tray. This result verifies the efficient capture capability of the multi-layer feature bridging mechanism in this method for scene spatial details, and the improvement effect of spatial awareness gating on the positioning accuracy of fine-grained operations. In Task 2, it demonstrates excellent semantic alignment and environmental anti-interference capabilities. In a complex scene with multiple interfering objects such as cans and condiment bottles, this method accurately locks onto the barbecue sauce target specified by the instruction and places it in the basket with the optimal motion trajectory. This result reflects the accurate matching capability of the cross-modal feature fusion mechanism of this method for instruction intent and scene target. In Task 3, it successfully achieves end-to-end mapping from abstract semantic instructions to continuous physical actions. This invention can autonomously infer that the precondition for "placing the bowl in the container" is "opening the drawer," and coherently complete a long-term composite task sequence of "opening the drawer - finding the bowl - grabbing and placing it." This demonstrates that the world knowledge prediction assistance method can inject prior common sense conforming to physical rules into the model's decision-making process, effectively improving the reasoning and execution capabilities of long-term tasks. In Task 4, "placing two coffee pots on the stove in sequence," the method can stably maintain task focus for a long time. After completing the first placement action, it can quickly reset the perception anchor point and accurately locate the second target. The action transitions are natural and smooth, without obvious logical pauses or trajectory jitter. This result verifies that the method, through its action self-attention mechanism, can effectively enhance the temporal correlation of action sequences and ensure that the generated trajectory conforms to physical smoothness constraints.

[0076] The precise control of robot pose and smooth generation of motion trajectories in various tasks fully validated the effectiveness of the bridging query vector initialization strategy proposed in this method. This strategy significantly improves the accuracy of cross-modal feature retrieval by pre-integrating the robot's motion state and temporal position information, laying a solid foundation for refined motion generation.

[0077] The quantization effect of this invention on Libero simulation is demonstrated and compared with typical methods in embodied intelligent operation tasks, as shown in Table 1:

[0078] Table 1. Quantization performance of this invention on the Libero dataset.

[0079]

[0080] Table 1 shows four task types in the Libero simulation: Spatial, Object, Goal, and Long. Experimental results based on Libero simulations demonstrate that this invention possesses excellent environmental perception accuracy and semantic understanding capabilities, effectively adapting to complex spatial layouts and abstract semantic instructions. This makes it particularly outstanding in the Spatial task, where spatial perception is a challenge. In the Object task, characterized by various fine-grained operations, and the Long task, characterized by complex and long time-series operations, the model generates coherent actions that conform to physical constraints, achieving optimal performance in both tasks. This fully validates the core application value of this method in embodied intelligent operation tasks.

[0081] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. A method for generating robot actions that integrates multi-layer feature bridging and world knowledge prediction, characterized in that, Includes the following steps: (1) Multimodal data preparation and feature extraction: Obtain the robot's visual image, language commands and robot proprioceptive state at the current moment; use the pre-trained visual language model to extract multi-layer visual raw features and action query latent variables; (2) Initialization of state-enhanced bridging query vector: The initial bridging query vector is generated by mapping the proprioceptive state through a multilayer perceptron and combining learnable time-step embeddings and random Gaussian noise. (3) Spatial-aware bridging attention multi-layer feature fusion: Inject multiple hidden layer features of the visual language model into the bridging attention module, introduce a spatial-aware gating mechanism, control the feature injection intensity of different attention heads and different image positions through vectorized parameters, and extract multi-modal fusion feature vectors through multi-layer attention mechanism; (4) Diffusion model action decoding: The multimodal fusion feature vector extracted in step (3) is used as the denoising condition of the diffusion model action head. Gaussian noise is transformed into a continuous action sequence through an iterative denoising process. (5) Joint training of world knowledge: Predict future world knowledge in the latent space and calculate the world knowledge prediction loss to constrain the feature distribution during the training phase.

2. The robot motion generation method integrating multi-layer feature bridging and world knowledge prediction according to claim 1, characterized in that, The visual images in step (1) include a third-person perspective and a hand-eye perspective.

3. The robot motion generation method integrating multi-layer feature bridging and world knowledge prediction according to claim 1, characterized in that, The state-enhanced bridging query vector initialization process described in step (2) is represented as follows: in, This is the initial bridging query vector. For data batch size, The length of the action sequence. For feature dimensions; Features of the robot's proprioceptive state; Represents a multilayer perceptron neural network; Position encoding is composed of learnable time-step embedding layers; This represents the temporal position index in the action sequence, and its range is... ; It is a random noise vector that follows a standard normal distribution.

4. The robot motion generation method integrating multi-layer feature bridging and world knowledge prediction according to claim 1, characterized in that, The implementation process of step (3) is as follows: Selected data are extracted in parallel from the visual language model. Features of the intermediate hidden layer; And through the projection layer, features from different levels are mapped to a unified low-dimensional embedding space; Constructing the initial bridging query vector for the bridging attention module using the robot's proprioceptive state. Bridge query vector As the core control signal, it interacts in parallel with VLM features and proprioceptive state within the bridging attention module through visual cross-attention, action-state cross-attention, and action self-attention. The bridging attention module concatenates and aggregates the calculation results of the three attention paths to generate the final multimodal fusion feature vector. : in, For visual cross-attention mechanism, This is an action-state cross-attention mechanism. For action self-attention mechanism, , , Refers to nonlinear projection functions. Visual features extracted by visual language models Refers to the latent variables of action queries extracted by the visual language model; The tangent activation function; For spatial perception gating parameters, For the number of heads, The number of image blocks.

5. The robot motion generation method integrating multi-layer feature bridging and world knowledge prediction according to claim 4, characterized in that, The extraction selection The intermediate hidden layer features include: visual features that serve as environmental geometric constraints. And the hidden variables for querying actions as the intent of high-level tasks. .

6. The robot motion generation method integrating multi-layer feature bridging and world knowledge prediction according to claim 4, characterized in that, The process of achieving visual cross-attention is as follows: by For querying, visual features from multiple hidden layers are used. Cross-attention is performed on the keys and values ​​to obtain the initial visual feature attention scores. Introducing spatial perception gating parameters To achieve fine-grained selective focus on different regions of an image; utilizing The activation function adjusts the weights of this output, enabling the visual language model to learn spatial weight masks for different operational tasks. The calculation formula is as follows: ； The gating mechanism independently assigns heterogeneous gating weights to each image block in the visual feature map, resulting in a processed visual feature attention score. This enables adaptive selection of features in task-related regions of an image and suppression of environmental background noise.

7. The robot motion generation method integrating multi-layer feature bridging and world knowledge prediction according to claim 4, characterized in that, The action-state cross-attention implementation process is as follows: by To query, retrieve the hidden variables from the action output of the VLM. Robot's proprioceptive state After concatenating the feature dimensions, they are used as key and value inputs; a mapping relationship between the command intent and the physical state is established to solve the control failure problem caused by the mismatch between the command and the real-time pose.

8. The robot motion generation method integrating multi-layer feature bridging and world knowledge prediction according to claim 4, characterized in that, The self-attention process for the action is as follows: For bridging query vectors Self-attention calculations are performed to enhance the temporal coherence of action sequences through internal interactions, ensuring that the generated trajectory conforms to physical smoothness constraints.

9. The robot motion generation method integrating multi-layer feature bridging and world knowledge prediction according to claim 1, characterized in that, The denoising process described in step (4) uses a diffusion model to denoise the loss. Training, the loss function is expressed as: 。 10. The robot motion generation method integrating multi-layer feature bridging and world knowledge prediction according to claim 1, characterized in that, The implementation process of step (5) is as follows: The world knowledge vectors extracted from the visual language model are input into the prediction head, which consists of a lightweight projection layer. The ambient optical flow information is extracted and spatial dilation is performed to construct a dynamic perception mask. This mask is then applied to the mean square error loss function of the future visual prediction task to achieve noise suppression of the static background and feature enhancement of the interactive area. Based on scale-invariant logarithmic loss, dual-view depth maps are reconstructed in the latent space. By explicitly modeling 3D geometric constraints, the spatial distance sense missing in 2D features is compensated. By introducing a visual basic model and using cosine similarity to achieve latent space semantic alignment, the visual language model's understanding of target attributes and topological boundaries is enhanced, ensuring the accuracy of objects generated by actions. Define world knowledge prediction loss The difference loss between the predicted value and the real environment label is calculated and then compared with the denoising loss of the diffusion model. Perform weighted joint training: in, This is the balance coefficient.

Images