Robot control method and system based on visual chain-type thinking
By introducing explicit visual chain thinking and full attention mechanism, the system predicts future sub-target images to generate precise action sequences, solving the problem of insufficient temporal planning in existing VLA models for complex tasks and improving the intelligence and generalization ability of the robot control system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN KIM DAI INTELLIGENCE INNOVATION TECHNOLOGY CO LTD
- Filing Date
- 2026-02-03
- Publication Date
- 2026-05-12
AI Technical Summary
Existing VLA models lack the ability for temporal planning and advanced reasoning in complex tasks, and cannot effectively utilize unlabeled video data for precise motion planning in pixel space.
We introduce explicit visual chain thinking, using the prediction of future sub-target images as intermediate inference results to generate accurate action sequences. We also employ a full attention mechanism for action token prediction, combined with a causal attention mechanism for autoregressive generation, and utilize large-scale non-action video data for pre-training and adaptation training.
It significantly improves the robot's temporal planning and reasoning capabilities in complex tasks, enhances decision-making accuracy and generalization performance, and realizes a more intelligent and versatile robot control system.
Smart Images
Figure CN122008209A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of robot learning, computer vision, and natural language processing, specifically a robot control method and system based on visual chain thinking. Background Technology
[0002] In recent years, Visual-Language-Action (VLA) models have demonstrated great potential in learning generalized perceptual-motor control strategies by leveraging large-scale pre-trained Visual-Language Models (VLMs) and diverse robot demonstration data. However, current mainstream VLA models primarily focus on constructing direct mappings from observation to action, lacking explicit modeling of key intermediate reasoning steps in complex tasks. This deficiency leads to insufficient temporal planning and high-level reasoning capabilities in tasks requiring multi-step planning, understanding of complex environments, or prediction of future states, limiting their effectiveness in real-world complex scenarios.
[0003] In the field of natural language processing, chained reasoning prompts have become an effective paradigm for enhancing the reasoning capabilities of large language models. Applying this "thinking" mode to robot control in a visual way provides a new approach to integrating advanced reasoning capabilities into the closed loop of perception and action. Although existing research has attempted to introduce intermediate representations such as language descriptions, keypoints, or bounding boxes to enhance robot reasoning, these methods often require additional preprocessing and capture mostly abstract states, failing to directly guide precise action planning in pixel space. How to perform intermediate reasoning in a more natural and direct visual form, and effectively utilize massive amounts of unlabeled video data, remains a pressing technical problem to be solved.
[0004] Therefore, the present invention was developed based on the above-mentioned shortcomings. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a robot control method based on visual chain thinking. By introducing explicit visual chain thinking before the robot performs actions, it predicts future sub-target images as intermediate reasoning results and generates accurate action sequences based on these results. This invention also provides a system for implementing this method.
[0006] This invention is achieved through the following technical solution:
[0007] A robot control method based on visual chain thinking, characterized by the following steps:
[0008] Step S1: Obtain the robot's current visual observations and natural language commands;
[0009] Step S2: Based on the current visual observation and the natural language instruction, generate sub-target images representing intermediate states of the task in an autoregressive manner using a unified visual-language model;
[0010] Step S3: Based on the current visual observation, the task instruction, and the sub-target image, generate a sequence of actions for controlling the robot;
[0011] Step S4: Execute the action sequence to control the robot to transition to the motion state represented by the sub-target image.
[0012] The robot control method based on visual chain thinking as described above is characterized by the following steps performed after generating and executing the action sequence:
[0013] Step S5: Acquire new visual observations from the robot;
[0014] Step S6: Using the new visual observation as the current visual observation, repeat the steps described in claim 1 to perform closed-loop control until the task corresponding to the natural language instruction is completed.
[0015] The robot control method based on visual chain thinking as described above is characterized in that: the action sequence in step S3 includes action blocks of multiple consecutive time step actions.
[0016] The robot control method based on visual chain thinking as described above is characterized in that: the generation of the action sequence in step S3 uses a full attention mechanism for action token prediction.
[0017] The robot control method based on visual chain thinking as described above is characterized in that: the visual-language model is trained through a two-stage process of pre-training and adaptation. The pre-training stage uses a combined dataset containing robot demonstration data and motionless video data to train the model, and the training target is the sum of the visual generation loss generated by the sub-target image and the motion generation loss generated by the motion sequence. The adaptation stage uses specific task data of the target robot scene to fine-tune the pre-trained model.
[0018] The robot control method based on visual chain thinking described above is characterized in that: the formula for calculating the visual generation loss of the sub-target image generation is:
[0019] L visual =−∑ j log P δ (k jd |k j,<d )
[0020] Where P δk represents the predicted probability when the model parameter is δ. jd Let k represent the d-th discrete token of the j-th visual input. j,<d This represents all tokens up to the d-th token;
[0021] The formula for calculating the action generation loss of the action sequence is:
[0022] L action =− log P θ (at...a t+m |l,s t ,s t+n )
[0023] Where P θ This represents the predicted probability when the model parameter is θ, at...a t+m This represents a series of continuous actions from time t to t+m, where l represents a natural language instruction, and s represents... t s represents the visual observation at the current moment. t+n This represents the predicted sub-target image.
[0024] The robot control method based on visual chain thinking as described above is characterized in that: in the pre-training stage, the generation of sub-target images adopts a causal attention mechanism, and the generation of action sequences adopts a full attention mechanism.
[0025] The robot control method based on visual chain thinking as described above is characterized in that: the robot demonstration data is a selected subset of the Open X-Embodiment dataset, and the motionless video data includes the EPIC-KITCHEN-100 dataset and / or the Something-Something V2 dataset.
[0026] The robot control method based on visual chain thinking described above is characterized in that: the unified visual-language model is the VILA-U model.
[0027] A robot control system based on visual chain thinking, which implements the aforementioned robot control method based on visual chain thinking, is characterized by comprising:
[0028] The perception module is used to acquire the current visual observation of the environment;
[0029] The instruction receiving module is used to receive natural language task instructions;
[0030] The visual chain thinking and reasoning module, which is connected to the perception module and the instruction receiving module, can generate sub-target images representing the intermediate state of the task based on the current visual observation and the task instruction, and can generate action sequences based on the current visual observation, the task instruction and the sub-target images.
[0031] A control output module, which is connected to the visual chain thinking and reasoning module, is used to send the action sequence to the robot for execution;
[0032] The perception module acquires new visual observations after the robot performs an action, and provides these new visual observations as the subsequent current visual observations to the visual chain thinking and reasoning module to form a closed-loop control.
[0033] Compared with the prior art, the present invention has the following advantages:
[0034] This invention introduces explicit visual sub-target prediction as an intermediate step in chained thinking. By predicting a series of future image frames as explicit visual sub-targets, and generating refined short action sequences to progressively achieve these pre-defined targets, it significantly enhances the model's temporal planning, reasoning ability, and decision-making accuracy in complex tasks. This method effectively utilizes large-scale, motionless video data for pre-training, overcoming the bottleneck of scarce robot demonstration data, thereby greatly improving the model's generalization performance and execution success rate in real-world robot tasks. The system features a modular design and strong interpretability, providing a solid foundation for building more intelligent and general-purpose robot control systems. Attached Figure Description
[0035] Figure 1 This is an overall architecture diagram of the present invention;
[0036] Figure 2 This is a schematic diagram of the hybrid attention mechanism in the model of this invention;
[0037] Figure 3 This is an example diagram of the present invention performing tasks in simulated and real robotic environments. Detailed Implementation
[0038] The present invention will now be further described with reference to the accompanying drawings:
[0039] like Figure 1 As shown, a robot control method based on visual chain thinking includes the following steps:
[0040] Step S1: Obtain the robot's current visual observations and natural language commands;
[0041] Step S2: Based on the current visual observation and the natural language instruction, generate sub-target images representing intermediate states of the task in an autoregressive manner using a unified visual-language model;
[0042] Step S3: Based on the current visual observation, the task instruction, and the sub-target image, generate a sequence of actions for controlling the robot;
[0043] Step S4: Execute the action sequence to control the robot to transition to the motion state represented by the sub-target image.
[0044] Step S1 aims to acquire the current visual observation of the robot's current environment, which can be in the form of an image or video frame, and to acquire the natural language instructions issued by the user. These instructions specify the target task the robot needs to complete, such as "pick up a carrot and put it in a bowl." The sub-target image generated in Step S2 serves as an intermediate visual reasoning step for the robot to achieve its current task. It accurately depicts the intermediate planning state the robot expects to reach in pixel space, representing the result of the robot's visual thinking before executing the action. In Step S3, the generation of the action sequence uses a full attention mechanism for action token prediction. The generated action sequence contains action blocks of multiple consecutive time steps, rather than single actions predicted independently for each time step, ensuring the coherence and efficiency of action execution.
[0045] After generating and executing the above sequence of actions, the following steps are further performed to achieve complete closed-loop control:
[0046] Step S5: Acquire new visual observations from the robot;
[0047] Step S6: Using the new visual observation as the current visual observation, repeat the steps described in claim 1 to perform closed-loop control until the task corresponding to the natural language instruction is completed.
[0048] The unified vision-language model used in the above method is the VILA-U model, an advanced unified multimodal foundational model capable of simultaneously understanding and generating image and text tokens. Through an autoregressive next-to-the-token prediction framework, it achieves unified understanding and generation capabilities for video, images, and language. Its internal mechanism includes a unified vision tower that encodes visual input into discrete tokens aligned with textual information, and progressively predicts residual tokens using residual quantization and a depth transformer, thereby significantly improving the representational power of discrete visual features.
[0049] This visual-language model is trained through a two-stage process of pre-training and adaptation to ensure the model's generalization ability and scene adaptability.
[0050] Specifically, the pre-training phase aims to enable the model to learn visual dynamic common sense and basic task understanding and planning capabilities from massive amounts of data. The model is trained using a combined dataset containing robot demonstration data (Dr) and still video data (Dv), such as a selected subset of the Open X-Embodiment dataset, and the still video data (Dv) includes the EPIC-KITCHEN-100 dataset and / or the Something-Something V2 dataset. During training, the model simultaneously learns two objectives: first, to generate sub-target images using a causal attention mechanism; and second, to generate actions using a full attention mechanism.
[0051] In the generation of sub-target images, a causal attention mask is used to ensure that the generation of the current token depends only on previous tokens, conforming to the autoregressive generation characteristics. The formula for calculating the visual generation loss of sub-target image generation is:
[0052] L visual =−∑ j log P δ (k jd |k j,<d )
[0053] Where P δ k represents the predicted probability when the model parameter is δ. jd Let k represent the d-th discrete token of the j-th visual input. j,<d This represents all tokens up to the d-th token;
[0054] On the action sequence generation path, a full attention mechanism is used, allowing full interaction among all action tokens to generate coordinated and coherent multi-step actions. When generating a sequence of m actions from n predicted sub-target images, the action generation loss is calculated as follows:
[0055] L action =− log P θ (at...a t+m |l,s t ,s t+n )
[0056] Where P θ This represents the predicted probability when the model parameter is θ, at...a t+m This represents a series of continuous actions from time t to t+m, where l represents a natural language instruction, and s represents... t s represents the visual observation at the current moment. t+n This represents the predicted sub-target image, and the value of m is usually set to 10.
[0057] The overall training objective is the sum of the visual generation loss for sub-object image generation and the action generation loss for action sequence generation, i.e., L=L action +L visual This stage primarily optimizes the parameters of the model's LLM backbone network, projector, and depth transformer, while keeping the parameters of the vision tower fixed to stabilize training.
[0058] Specifically, the adaptation phase fine-tunes the pre-trained model using task-specific data from the target robot scenario. On the action sequence generation path, a full attention mechanism is used, allowing full interaction between all action tokens to generate coordinated and coherent multi-step actions. The training setup for this phase is similar to the pre-training phase, also optimizing the LLM backbone network, projector, and depth transformer to ensure the model can efficiently adapt to new real-world environments and tasks.
[0059] The present invention also provides a robot control system based on visual chain thinking, which is used to implement the above-mentioned robot control method based on visual chain thinking, including: a perception module, an instruction receiving module, a visual chain thinking reasoning module, and a control output module.
[0060] The perception module acquires current visual observations of the environment through visual sensing devices such as cameras and depth cameras;
[0061] The instruction receiving module receives natural language task instructions, such as "put the carrot in the bowl";
[0062] The visual chain-based reasoning module is connected to the perception module and the instruction receiving module. It is configured to generate sub-target images representing intermediate states of the task in an autoregressive manner based on the current visual observation and the task instruction. The generation process uses a causal attention mechanism to ensure temporal continuity. It can also decode action tokens based on the current visual observation, the task instruction, and the sub-target images using a full attention mechanism to generate action sequences. The control output module, connected to the visual chain-based reasoning module, is used to send the action sequences to the robot actuator to drive the robot to transition to the motion state represented by the sub-target images. Moreover, after the robot performs an action, the perception module acquires new visual observations and provides these new visual observations as the subsequent current visual observations to the visual chain-based reasoning module to form a closed-loop control.
[0063] In summary, this invention provides a groundbreaking, efficient, and interpretable solution for the field of robot learning. By deeply integrating the visual chain thinking reasoning mechanism into the VLA model, it significantly enhances the robot's ability to perform complex operational tasks. Figure 3The paper demonstrates task execution examples of the system applying this invention in the LIBERO simulation environment, Bridge-V2, and the Franka-Tabletop real robot platform, fully verifying its superior performance improvement and broad generalization ability. As shown in the figure, the system can first generate reasonable sub-target images according to instructions, then plan and execute a series of precise action sequences, such as moving, grasping, and pulling, and finally successfully complete the task, verifying the effectiveness and practicality of the invention.
Claims
1. A robot control method based on visual chain thinking, characterized in that, Includes the following steps: Step S1: Obtain the robot's current visual observations and natural language commands; Step S2: Based on the current visual observation and the natural language instruction, generate sub-target images representing intermediate states of the task in an autoregressive manner using a unified visual-language model; Step S3: Based on the current visual observation, the task instruction, and the sub-target image, generate a sequence of actions for controlling the robot; Step S4: Execute the action sequence to control the robot to transition to the motion state represented by the sub-target image.
2. The robot control method based on visual chain thinking according to claim 1, characterized in that, After generating and executing the action sequence, the following steps are also performed: Step S5: Acquire new visual observations from the robot; Step S6: Using the new visual observation as the current visual observation, repeat the steps described in claim 1 to perform closed-loop control until the task corresponding to the natural language instruction is completed.
3. The robot control method based on visual chain thinking according to claim 1, characterized in that: The action sequence in step S3 includes action blocks consisting of multiple consecutive time-step actions.
4. The robot control method based on visual chain thinking according to claim 1, characterized in that: In step S3, the generation of the action sequence uses a full attention mechanism for action token prediction.
5. The robot control method based on visual chain thinking according to claim 1, characterized in that: The vision-language model is trained through a two-stage process: a pre-training stage and an adaptation stage. The pre-training stage uses a combined dataset containing robot demonstration data and motionless video data to train the model, with the training target being the sum of the visual generation loss generated from the sub-target image and the motion generation loss generated from the motion sequence. The adaptation stage uses specific task data from the target robot scene to fine-tune the pre-trained model.
6. The robot control method based on visual chain thinking according to claim 5, characterized in that: The formula for calculating the visual generation loss of the sub-target image is as follows: L visual =−∑ j log P δ (k jd ∣k j,<d ); Where P δ k represents the predicted probability when the model parameter is δ. jd Let k represent the d-th discrete token of the j-th visual input. j,<d This represents all tokens up to the d-th token; The formula for calculating the action generation loss of the action sequence is: L action =− log P θ (at...a t+m ∣l,s t ,s t+n ); Where P θ This represents the predicted probability when the model parameter is θ, at...a t+m This represents a series of continuous actions from time t to t+m, where l represents a natural language instruction, and s represents... t s represents the visual observation at the current moment. t+n This represents the predicted sub-target image.
7. The robot control method based on visual chain thinking according to claim 5, characterized in that: In the pre-training phase, the generation of sub-target images adopts a causal attention mechanism, while the generation of action sequences adopts a full attention mechanism.
8. The robot control method based on visual chain thinking according to claim 5, characterized in that: The robot demonstration data is a selected subset of the Open X-Embodiment dataset, and the motionless video data includes the EPIC-KITCHEN-100 dataset and / or the Something-Something V2 dataset.
9. The robot control method based on visual chain thinking according to claim 1, characterized in that: The unified visual-language model is the VILA-U model.
10. A robot control system based on visual chain thinking, the system being used to implement the robot control method based on visual chain thinking as described in any one of claims 1-8, characterized in that, include: The perception module is used to acquire the current visual observation of the environment; The instruction receiving module is used to receive natural language task instructions; The visual chain thinking and reasoning module, which is connected to the perception module and the instruction receiving module, can generate sub-target images representing the intermediate state of the task based on the current visual observation and the task instruction, and can generate action sequences based on the current visual observation, the task instruction and the sub-target images. A control output module, which is connected to the visual chain thinking and reasoning module, is used to send the action sequence to the robot for execution; The perception module acquires new visual observations after the robot performs an action, and provides these new visual observations as the subsequent current visual observations to the visual chain thinking and reasoning module to form a closed-loop control.