A robot control method, device and system based on a guided mechanism VLA model
By using a guided VLA model to decompose the action decoder into specialized attention heads and combining multiple supervision methods and residual fusion strategies, the problems of attention randomness and lack of 3D geometric perception in existing VLA models are solved, thereby improving the robot's task success rate and stability in complex operations.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FUDAN UNIVERSITY
- Filing Date
- 2026-05-08
- Publication Date
- 2026-07-21
AI Technical Summary
Existing VLA models suffer from problems such as attentional randomness and instability, lack of 3D geometric perception, and weak understanding of long-term skill logic, resulting in insufficient motion accuracy and poor task execution continuity of robots in complex multi-stage operations.
We employ a guided VLA model, which splits the action decoder into three specialized attention heads: object localization, skill recognition, and depth geometry. Through object localization mask supervision, temporal skill soft label supervision, and 3D depth feature structure constraints, combined with a ControlNet-style zero-initialization residual adapter, we achieve explicit guidance and fusion of multimodal features.
It significantly improves the model's robustness to distribution shifts and scene perturbations, increases the robot's task success rate in multi-task scenarios, and enhances the learning effect of task factors while maintaining pre-training capabilities.
Smart Images
Figure CN122425691A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot intelligent control technology, and in particular to a robot control method based on a guided vision-language-action model. Background Technology
[0002] With the development of deep learning and large model technology, vision-language-action (VLA) models that directly generate robot actions based on visual and linguistic input have become an important research direction in the field of embodied intelligence. Existing technologies mainly include the following categories: End-to-end supervised learning approach: Represented by RT-1, RT-2, π0, and OpenVLA, this type of method typically uses a large-scale vision-language model as an encoder to map images and language commands to a unified feature space, and directly generates low-level control actions through a policy network or diffusion model. This method relies on implicit learning; the action decoder automatically extracts task-related features from the demonstration data. Action generation is highly dependent on implicit representations, lacks intermediate constraints, and is prone to action deviation or failure.
[0003] Intermediate representation enhancement approach: Represented by DreamVLA and AdaMoE, this approach enhances VLA models by introducing intermediate representations such as dynamic region prediction, depth maps, and semantic knowledge. DreamVLA allows the VLM to predict dynamic regions and depth information, while AdaMoE uses a hybrid expert mechanism to automatically learn task experts. However, these methods do not achieve factor decoupling, and the representations suffer from entanglement problems.
[0004] 3D geometry enhancement approach: This approach improves spatial awareness through explicit 3D representations such as point clouds and 3D scene tokens, including 3DDiffuser Actor and SpatialVLA. While this method enhances geometric understanding, it does not optimize in conjunction with other key factors such as object localization and temporal logic.
[0005] Most existing robot control models suffer from specialization, either only adapting to a single type of robot hardware or only performing specific, simple tasks. When faced with complex, multi-stage dexterity maneuvers, they are prone to insufficient motion accuracy and poor task execution consistency. While some VLA models attempt to integrate multimodal information for robot control, they often use autoregressive discretization to represent actions, making it difficult to support the demands of high-frequency dexterity maneuvers. Furthermore, they exhibit the following shortcomings: attentional randomness and instability: unguided attention heads focus on task-related regions with high randomness. Lack of 3D geometric perception: standard π0 models are based solely on 2D visual input, lacking explicit 3D spatial structure perception. Weak understanding of long-term skill logic: for multi-stage tasks, existing models struggle to maintain stage awareness, easily experiencing mode collapse or premature termination during sub-skill transitions. Summary of the Invention
[0006] The purpose of this invention is to address the problems of attentional randomness and instability, lack of 3D geometric perception, and weak understanding of long-term skill logic in existing VLA models, and to provide a robot control method based on a guided mechanism VLA model.
[0007] The objective of this invention can be achieved through the following technical solutions: As a first aspect of the present invention, a robot control method based on a VLA (Virtual Auxiliary Leverage) model is provided, the method comprising the following steps: Collect images, language command text, and the current joint state of the robot received by the robot, and input them into the improved VLA model; The improved VLA model encodes images and language command text, and fuses them with the noise action sequence in the VLA model denoising process to extract multimodal features; Multimodal features are input into the action decoder of the improved VLA model. The action decoder includes a standard attention head and an object localization attention head, a skill recognition attention head, and a depth geometry attention head connected in parallel with the standard attention head. The outputs of each attention head are fused using style residuals via ControlNet. The object positioning attention head is used to locate the task target object through mask-guided attention supervision. The skill recognition attention head is used to identify the skill stage and temporal logic of the current operation through soft label classification loss; The depth geometric attention head is used to process three-dimensional spatial geometric information and inject depth features to achieve explicit geometric reasoning; Based on the features obtained after style residual fusion, the robot's future motion sequences are predicted and generated through the motion expert module.
[0008] As a preferred technical solution, the representation fusion is performed to extract multimodal features, specifically as follows: It receives multiple RGB images from different perspectives, natural language command text, and the robot's current joint state as raw input; For each image, an image token is extracted by a visual encoder, and the image token is mapped to a unified space by a multimodal projector; Segment the natural language instruction text and extract the language token; The noise action sequence in the VLA model denoising process is used as the action token; The multi-path image tokens, language tokens, and action token placeholders are concatenated into a unified sequence, input into a visual language model for cross-modal feature fusion, and the fused multimodal features are output.
[0009] As a preferred technical solution, the improved VLA model maps action tokens to action queries in the attention mechanism through a linear projection layer.
[0010] As a preferred technical solution, the object localization attention head uses the action query as the retrieval vector and the key and value extracted by the VLA model as the retrieval objects, and generates a global attention heatmap of the image by scaling dot product attention calculation; The input image and stage-aware object description are used as input. Grounded SAM is used to generate ground truth region masks of stage-aware objects, which serve as the benchmark for supervision constraints. Using a truth mask as a constraint, the attention heatmap is supervised by the mask negative log-likelihood loss, which forces the attention of non-interested regions to be set to 0, and outputs the attention weights that focus on the objects of interest.
[0011] As a preferred technical solution, the skill recognition attention head uses the action query as the retrieval vector and the key and value extracted by the VLA model as the retrieval objects. It obtains a global representation by scaling dot product attention calculation and average pooling, and then performs linear projection mapping on the global representation to obtain the predicted skill probability distribution. The predicted skill probability distribution is used to calculate the KL divergence loss with the skill soft labels based on time window statistics, and the attention weights that encode the temporal semantics of skills are finally output.
[0012] As a preferred technical solution, the depth geometric attention head extracts depth features from the input image using a frozen DepthAnything V3-small depth encoder; after the depth features are downsampled in the Token Merging space, a key-value projector maps the downsampled depth features into keys and values. Using the action query as the retrieval vector and the key and value of the deep feature map as the retrieval objects, the association between the action semantics and 3D geometric features is modeled by scaling dot product attention calculation, and finally the attention weights that inject geometric information are output.
[0013] As a preferred technical solution, ControlNet performs style residual fusion as follows: The calculation results of the standard attention head are used as the main output; the calculation results of the object localization attention head, skill recognition attention head, and geometric perception attention head are used as the guiding output. The various pilot outputs are concatenated and processed through a zero-initialized linear layer, and then residually fused with the trunk output to obtain the final output of the action decoder.
[0014] As a preferred technical solution, for the future motion sequence generated by the motion expert model, the motion is upsampled to the control frequency through linear interpolation to generate a smooth joint trajectory; The robot controller executes the interpolated joint trajectory and feeds back the new image and joint state as input for the joint state of the next time step, forming a closed-loop control.
[0015] As a second aspect of the present invention, a robot control device based on a VLA (Virtual Amplification and Liberation) model is provided, comprising a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the robot control method based on the VLA model as described above.
[0016] As a third aspect of the present invention, a storage medium is provided having a program stored thereon, which, when executed, implements the robot control method based on the VLA model of the guidance mechanism as described above.
[0017] Compared with the prior art, the present invention has the following beneficial effects: 1) This invention designs a guided action decoding framework for vision-language-action (VLA) robot control that balances the preservation of pre-training capabilities with explicit learning of task factors. The multi-head attention of the action decoder is divided into three specialized attention heads: target localization, skill recognition, and depth geometry. Through directional constraints, action tokens actively focus on core task factors, solving the problems of traditional VLA models, such as easy learning of spurious associations during end-to-end training, random attention behavior, and poor generalization.
[0018] 2) This invention integrates target region mask supervision, temporal skill soft label supervision, and 3D deep feature structure constraints, and achieves a balanced fusion of specialized features and main branch features through a ControlNet-style zero-initialization residual adapter. The specialized attention head explicitly captures three core factors: target localization, temporal skill, and spatial geometry, significantly improving the model's robustness to distribution shifts and scene perturbations. The zero-initialization adapter avoids forgetting pre-training capabilities, and the residual fusion strategy ensures that the model's performance does not degrade in the early stages of training. Ultimately, it achieves a significant improvement in task success rate in multi-task scenarios on both simulation and real robot platforms. Attached Figure Description
[0019] Figure 1 This is a flowchart of the robot control method based on the vision-language-action model of the guidance mechanism of the present invention.
[0020] Figure 2 The flowchart shows the calculation process for the object positioning attention head in this invention.
[0021] Figure 3 This is a flowchart illustrating the calculation process of the skill recognition attention head in this invention.
[0022] Figure 4 This is a flowchart illustrating the calculation process of the depth geometric attention head of the present invention. Detailed Implementation
[0023] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.
[0024] Example 1 This invention proposes a robot control method based on a guided vision-language-action model. The action decoder of the traditional VLA model is transformed from a single, holistic learner into a set of components with functional specialization. By assigning explicit task-related factors to different attention heads and applying targeted supervisory constraints, the model is guided to explicitly learn and enhance key task capabilities.
[0025] Specifically, this invention functionally divides the multi-head attention mechanism of the action decoder into three specialized attention heads: an object localization head, a skill recognition head, and a depth geometry head. These specialized attention heads are then used to provide directional supervision of the action token query vector through weighted negative log-likelihood loss, KL divergence loss, and structural constraints, explicitly guiding the capture of task-related factors. Furthermore, a ControlNet-style zero-initialization ZeroConv projection layer is used to residually fuse the output features of the specialized attention heads with the main branch features of the action decoder, balancing pre-training capability preservation with specialized factor learning.
[0026] like Figure 1 As shown, the main steps for implementing a robot control method based on a guided vision-language-action model are as follows: Step 1: Input preprocessing and multimodal feature extraction.
[0027] Step 1.1: Receive three multi-view RGB images (left and right wrist cameras and a third-party camera), natural language command text (e.g., "Put the cup on the tray"), and the robot's current joint state as the foundation for cross-modal fusion.
[0028] Step 1.2: For each image, extract the image token using the SigLIP visual encoder, and map the image tokens of each image to a unified space using a multimodal projector.
[0029] Step 1.3: Segment the language instructions and extract semantic features from the language tokens. Concatenate the language tokens, three-way image tokens, and action token placeholders into a unified sequence. Input this sequence into the backbone of the PaliGemma visual-language model for cross-modal feature fusion, and finally output the fused multimodal features. The action token is a learnable embedding vector used to represent the robot's action sequence. In a flow-matching-based VLA model (such as the baseline model π0), the action token corresponds to the noisy action sequence generated during denoising. In subsequent attention layers of the model, the action token is used to generate an action query vector through linear projection from its hidden state, which is used to retrieve visual and linguistic features.
[0030] Step 2: Parallel computation of three types of specialized attention. Based on the pre-trained VLA model, three types of functional specialized attention heads are introduced: object localization head, skill recognition head, and geometric perception head. This achieves multi-factor decoupling and explicit guidance, eliminating the need to train the entire model from scratch.
[0031] Step 2.1: Object positioning head, such as Figure 2 As shown, using the action query as the retrieval vector and the key / value extracted by the VLA model as the retrieval object, a global attention heatmap of the image is generated by scaling dot product attention calculation, which models the spatial relationship between action tokens and visual features.
[0032] The input image and stage-aware object description are used as inputs. Grounded SAM is used to generate ground-value region masks of stage-aware objects, which serve as the benchmark for supervision constraints.
[0033] Using a truth mask as a constraint, the attention heatmap is supervised by the mask negative log-likelihood loss, which forces the attention of non-interested regions to be set to 0, and only punishes the behavior of attention leaking to the outside of the object. Finally, the attention weights that focus on the object of interest are output.
[0034] Step 2.2: Skill recognition head, such as Figure 3 As shown, the action query is used as the retrieval vector and the key / value temporal visual features extracted by the VLA model are used as the retrieval objects. The global temporal representation that integrates action semantics is generated by scaling dot product attention calculation. The skill probability distribution is obtained by linear projection mapping, realizing end-to-end prediction from features to skill semantics.
[0035] The predicted skill probability distribution is used to calculate the KL divergence loss with soft labels based on time window statistics, enabling the model to learn to identify the current operation stage. Finally, the attention weights that encode the temporal skill semantics are output, achieving accurate identification of the current operation stage.
[0036] Step 2.3: Geometric sensing head, such as Figure 4 As shown, for the input RGB image, depth features are extracted using a frozen DepthAnything V3-small depth encoder. After spatial downsampling of the depth features by Token Merging, a dedicated key-value projector is constructed to map the downsampled depth features into keys and values. This projector contains independent linear transformation layers. The spatially downsampled depth features are first normalized by the layer, and then projected through the linear transformation layers to generate depth-aware keys and values. This constrains the head to focus only on depth features for 3D geometric reasoning, thereby decoupling geometric reasoning from semantic information.
[0037] Using the action query as the query vector and the key and value of the deep features as the retrieval objects, the association between the action semantics and 3D geometric features is modeled by scaling dot product attention calculation. Finally, the attention weights injected with geometric information are output to provide attention guidance that integrates 3D spatial information for downstream tasks.
[0038] Step 3: ControlNet style residual blending.
[0039] Step 3.1: Use the calculation results of the standard attention head as the main output to retain the original capabilities of the pre-trained VLA model; use the calculation results of the three specialized attention heads as the calculation guidance output to inject task-customized attention guidance information into the model.
[0040] Step 3.2: Concatenate the outputs of the three specialized attention heads (object localization head, skill recognition head, and geometry perception head), and then perform residual fusion of the guiding output and the backbone output through a zero-initialized linear layer. The fusion formula is "final output = backbone output + zero convolution (concatenation of the outputs of the three specialized attention heads)".
[0041] This fusion mechanism fully preserves the original capabilities of the pre-trained VLA model in the early stages of training. As training progresses, the spliced specialized features are gradually learned and injected into the backbone, achieving a unified fusion of multi-factor guidance information rather than independent fusion, which helps maintain the consistency of the feature space. Furthermore, as the model learns specialized guidance information as training advances, it can achieve synergistic optimization of pre-training capability preservation and task-customized enhancement, ultimately outputting multimodal features that fuse multi-dimensional guidance information for generating robot control commands.
[0042] Step 4: Motion generation and robot execution.
[0043] Step 4.1: Based on the fused feature representation, predict future action sequences (action blocks) through the Gemma-300M action expert module, and optimize the action generation process by using flow matching training objectives, so that the predicted action distribution gradually approximates the distribution of real demonstration data.
[0044] Step 4.2: Inference phase, the model outputs action blocks. After receiving the action blocks, the client upsamples the action to the control frequency through linear interpolation to generate smooth joint trajectories.
[0045] Step 4.3: The robot controller executes the interpolated action, and the environmental feedback (new image, new state) is used as the input for the next time step to form closed-loop control.
[0046] This invention introduces three specialized attention heads—object localization, skill recognition, and geometry perception—based on a pre-trained VLA model. This achieves multi-factor decoupling and explicit guidance, eliminating the need for retraining the entire model from scratch. The object localization head is specifically designed to focus on task-related object regions, achieving precise target object localization through mask-guided attention supervision, forcing the model's attention to key areas such as the target object and grasping points. The skill recognition head is dedicated to identifying the skill stage and temporal logic of the current operation, achieving stage perception through soft-label classification loss, effectively capturing stage structure information in long-term tasks. The geometry perception head is specifically designed to process 3D spatial geometry information, achieving explicit geometric inference through dedicated depth feature injection, significantly enhancing the model's control over depth information and inference accuracy. A ControlNet-style zero-initialization ZeroConv projection layer is used to residually fuse the output features of the specialized attention heads with the main branch features of the action decoder, balancing pre-training capability preservation with specialized factor learning.
[0047] Example 2 As a second aspect of the present invention, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the robot control method based on the VLA model of the bootstrapping mechanism as described above. In addition to the processors, memory, and interfaces described above, any data processing device in the embodiments may also include other hardware depending on the actual function of the data processing device, which will not be elaborated further.
[0048] Example 3 As a third aspect of the present invention, this application also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, implement the robot control method based on the VLA model of the bootstrapping mechanism described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.
[0049] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. A robot control method based on a VLA (Virtual Auxiliary Linger) model with a guidance mechanism, characterized in that, The method steps include: Collect images, language command text, and the current joint state of the robot received by the robot, and input them into the improved VLA model; The improved VLA model encodes images and language command text, and fuses them with the noise action sequence in the VLA model denoising process to extract multimodal features; Multimodal features are input into the action decoder of the improved VLA model. The action decoder includes a standard attention head and an object localization attention head, a skill recognition attention head, and a depth geometry attention head connected in parallel with the standard attention head. The outputs of each attention head are fused using style residuals via ControlNet. The object positioning attention head is used to locate the task target object through mask-guided attention supervision. The skill recognition attention head is used to identify the skill stage and temporal logic of the current operation through soft label classification loss; The depth geometric attention head is used to process three-dimensional spatial geometric information and inject depth features to achieve explicit geometric reasoning; Based on the features obtained after style residual fusion, the robot's future motion sequences are predicted and generated through the motion expert module.
2. The robot control method based on the VLA model of the guidance mechanism according to claim 1, characterized in that, The representation fusion is performed to extract multimodal features, as follows: It receives multiple RGB images from different perspectives, natural language command text, and the robot's current joint state as raw input; For each image, an image token is extracted by a visual encoder, and the image token is mapped to a unified space by a multimodal projector; Segment the natural language instruction text and extract the language token; The noise action sequence in the VLA model denoising process is used as the action token; The multi-path image tokens, language tokens, and action token placeholders are concatenated into a unified sequence, input into a visual language model for cross-modal feature fusion, and the fused multimodal features are output.
3. The robot control method based on the VLA model of the guidance mechanism according to claim 1, characterized in that, The improved VLA model maps action tokens to action queries in the attention mechanism through a linear projection layer.
4. The robot control method based on the VLA model of the guidance mechanism according to claim 1, characterized in that, The object localization attention head uses the action query as the retrieval vector and the key and value extracted by the VLA model as the retrieval objects. It generates a global attention heatmap of the image by scaling dot product attention calculation. The input image and stage-aware object description are used as input. Grounded SAM is used to generate ground truth region masks of stage-aware objects, which serve as the benchmark for supervision constraints. Using a truth mask as a constraint, the attention heatmap is supervised by the mask negative log-likelihood loss, which forces the attention of non-interested regions to be set to 0, and outputs the attention weights that focus on the objects of interest.
5. The robot control method based on the VLA model of the guidance mechanism according to claim 1, characterized in that, The skill recognition attention head uses the action query as the retrieval vector and the key and value extracted by the VLA model as the retrieval objects. It obtains the global representation by scaling dot product attention calculation and average pooling, and then performs linear projection mapping on the global representation to obtain the predicted skill probability distribution. The predicted skill probability distribution is used to calculate the KL divergence loss with the skill soft labels based on time window statistics, and the attention weights that encode the temporal semantics of skills are finally output.
6. The robot control method based on the VLA model of the guidance mechanism according to claim 1, characterized in that, The depth geometric attention head extracts depth features from the input image using a frozen Depth Anything V3-small depth encoder; After the depth features are downsampled in the Token Merging space, the downsampled depth features are mapped to keys and values through a key-value projector. Using the action query as the retrieval vector and the key and value of the deep feature map as the retrieval objects, the association between the action semantics and 3D geometric features is modeled by scaling dot product attention calculation, and finally the attention weights that inject geometric information are output.
7. The robot control method based on the VLA model of the guidance mechanism according to claim 1, characterized in that, The ControlNet performs style residual fusion as follows: The calculation results of the standard attention head are used as the main output; the calculation results of the object localization attention head, skill recognition attention head, and geometric perception attention head are used as the guiding output. The various pilot outputs are concatenated and processed through a zero-initialized linear layer, and then residually fused with the trunk output to obtain the final output of the action decoder.
8. The robot control method based on the VLA model of the guidance mechanism according to claim 1, characterized in that, For the future motion sequence generated by the motion expert model, the motion is upsampled to the control frequency through linear interpolation to generate a smooth joint trajectory; The robot controller executes the interpolated joint trajectory and feeds back the new image and joint state as input for the joint state of the next time step, forming a closed-loop control.
9. A robot control device based on a VLA (Virtual Automation) model, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the robot control method based on the VLA model of the guidance mechanism as described in any one of claims 1-8.
10. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the robot control method based on the VLA model of the guidance mechanism as described in any one of claims 1-8.