A visual-language-action model training method based on stage reasoning and stage optimization
By decomposing the robot trajectory into multiple semantically defined stages and combining stage reasoning and optimization methods, the problem of evaluating action contribution in long-term tasks by robot behavior models is solved, thereby improving the stability and efficiency of task execution.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- EAST CHINA NORMAL UNIV
- Filing Date
- 2026-03-23
- Publication Date
- 2026-06-05
AI Technical Summary
Existing robot behavior models struggle to accurately assess the contribution of individual actions or local decisions to the overall success of long-term, multi-stage tasks, resulting in low learning efficiency and poor policy stability. Existing preference data collection methods are costly, have poor scalability, and are difficult to characterize the quality of complex operations.
By employing a phased reasoning and phased optimization approach, the robot trajectory is decomposed into multiple semantically defined phases. A multimodal large language model is used for phase description and autoregressive prediction. Combined with phase-level expert demonstration data and online reinforcement learning, state reward signals are constructed, and action generation strategies are optimized.
It improves the stability and success rate of robot execution in complex, multi-stage tasks, achieves stage consistency and causal rationality in action generation, and has a clear method structure and strong scalability.
Smart Images
Figure CN122154935A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of embodied intelligence technology, specifically a visual-language-action model training method based on staged reasoning and staged optimization. Background Technology
[0002] With the rapid development of artificial intelligence and robotics, large-scale vision-language-action models are emerging, enabling robots to perform complex tasks in unstructured environments by unifying visual perception, language understanding, and motion control. However, existing robot behavior models still face significant challenges when handling long-term, multi-stage, and contact-rich maneuvers. Currently, vision-language-action models are typically trained and optimized using methods such as supervised fine-tuning, reinforcement learning, or preference optimization. However, in long-term tasks, these methods generally rely on uniformly modeling and optimizing the entire motion trajectory, resulting in highly sparse reward signals and optimization objectives in the time dimension, coarse-grained credit allocation, and difficulty in accurately assessing the contribution of individual actions or local decisions to the overall task success. This reduces model learning efficiency and affects policy stability.
[0003] To alleviate the aforementioned problems, introducing human preference data is considered an effective approach. Existing preference data collection methods mainly include human annotation, online human intervention, and scoring methods based on manual rules or automatic scoring standards. However, these methods still have significant shortcomings: automatic scoring methods typically only reflect performance indicators with limited dimensions, making it difficult to characterize the quality of complex operations; online human intervention is costly and lacks scalability; and during preference annotation, because robot trajectories often consist of multiple stages with significant differences in semantics and difficulty, and different stages have different evaluation dimensions in terms of action consistency, stability, and completion quality, it is difficult for humans to provide clear and consistent preference judgments for the complete trajectory, resulting in ambiguous preference information and low utilization efficiency. In recent years, some research has begun to focus on stage-aware robot learning methods, introducing stage planning into the vision-language-action model or constructing reward models with stage-aware capabilities, aiming to improve the optimization performance of long-term tasks.
[0004] In summary, most existing technologies only introduce stage information in a single stage modeling, reward design, or policy learning process, and have not yet formed a unified training framework that synergistically combines stage reasoning, stage preference modeling, and online policy optimization, making it difficult to fully leverage the advantages of stage structures in long-term task learning. Summary of the Invention
[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a visual-language-action model training method based on stage-based reasoning and stage optimization. This method leverages the deep semantic reasoning capabilities of a multimodal large language model and introduces stage encoding to generate fine-grained descriptions for each action, constructing a stage-annotated expert demonstration dataset. Based on this, the pre-trained visual-language-action model is extended into a stage-based reasoning model with stage information as an intermediate reasoning condition. Stage information is autoregressively predicted by a self-attention module to determine the current stage semantics and stage completion progress, while a stage completion detection mechanism controls stage switching. Supervised learning ensures the model maintains consistency with the stage-level expert demonstration data in terms of stage information and action generation. Stage-level preference data annotated by human experts is introduced, and the model's action generation strategy is adjusted using a stage trajectory preference optimization method. Finally, a state reward signal based on stage progress is constructed, and combined with the model's interaction data with the environment, online reinforcement learning is used to optimize the model's strategy. This paper introduces a stage-based reasoning-based action generation method, decomposing long-term robot tasks into multiple semantically clear and causally ordered stages. Through the progressive reasoning and propagation of stage-level information, the model can acquire high-order task structure understanding across time with fewer optimization iterations, effectively alleviating the problems of fuzzy credit allocation and unstable optimization in long-sequence trajectories. Simultaneously, a stage-level "imitation-preference-interaction" optimization strategy is employed, enabling the model to continuously optimize action strategies while maintaining stage consistency and task interpretability. This achieves stable execution of complex, long-field-of-view robot operation tasks. The method has a clear structure, strong scalability, and promising engineering application prospects.
[0006] The objective of this invention is achieved as follows: a visual-language-action model training method based on stage inference and stage optimization. The method first utilizes a multimodal large language model to divide and semantically describe the robot trajectory into stages, obtaining stage-annotated data. Then, the visual-language-action model, extended by visual observations, language commands, and stage termination signals, is input and, through a stage-conditional self-attention mechanism, autoregressively predicts the current stage description and remaining stage completion as intermediate inference conditions, generating action sequences and stage completion signals based on these conditions. Next, supervised learning ensures the model output aligns with stage-level expert demonstration data. Stage trajectory preference data annotated by human experts is introduced, and a stage-level trajectory preference optimization method is used to directly adjust the model's policy distribution to favor stages with higher generation probabilities. Finally, a state-level reward signal based on stage progress is constructed, and the model is deployed in a simulation environment to interact with the environment. A proximal policy optimization method is used to perform online policy optimization of the visual-language-action model. Specifically, this method includes the following steps:
[0007] 1) Utilizing a multimodal large language model and a stage encoding method, stage segmentation and semantic description of the robot trajectory are completed, specifically including:
[0008] 1.1: Collect and standardize robot trajectory data from expert demonstrations. The trajectory data includes visual observations, motion sequences, and verbal commands.
[0009] 1.2: The trajectory is input into a multimodal large language model and divided into stages. Each stage is labeled with a 4-bit binary description, including whether it moves, the state of the mechanical gripper, and whether it contacts the object.
[0010] 1.3: Generate a fine-grained description for each action, including a stage description and whether the action is a stage termination signal;
[0011] 1.4: The processed trajectories are stored in stages to form a stage-annotated dataset containing fine-grained descriptions of visual observations, action sequences, language instructions, and actions, providing input for subsequent stage inference model training and preference optimization.
[0012] 2) The extended pre-trained vision-language-action model is a stage-based reasoning model for robot action generation that uses stage descriptions and stage remaining completion as intermediate reasoning conditions. Specifically, it includes:
[0013] 2.1: Visual observations, verbal instructions, and stage termination signals are input into a vision-language joint coding network to extract multimodal fusion state feature representations;
[0014] 2.2: Utilizing a self-attention mechanism to autoregressively predict the current stage description and remaining stage completion, as detailed below:
[0015] 2.2.1: Using multimodal state feature representations as input, a stage-conditional self-attention module is used to model the dependency relationship between the current state and historical states, highlighting actions and visual features related to the current stage;
[0016] 2.2.2: The current stage description is generated step by step in an autoregressive manner, and the remaining completion rate of the stage is predicted simultaneously to represent the completion progress of the current stage.
[0017] 2.3: The model predicts action sequences based on the generated stage information, as follows:
[0018] 2.3.1: The generated stage description, stage remaining completion degree, and multimodal state features are fused and input into the action generation network to generate the current action sequence via autoregression;
[0019] 2.3.2: Add a lightweight completion detection head to the feature layer of the action generation network to output the stage completion probability;
[0020] 2.3.3: When the probability of a stage completion is lower than a set threshold, a stage completion signal is generated, and the stage termination signal is used as model input to guide the switching of stage prediction.
[0021] 3) Supervised learning methods are used to ensure that the output of the stage-based reasoning visual-language-action model is consistent with the stage annotations in the stage-level expert demonstration data. Specifically, this includes:
[0022] 3.1: The expert demonstration data with stage annotations, including visual observations, language instructions, stage descriptions, action sequences and stage termination signals, are input into the pre-trained visual-language-action model as supervision signals for model training. The state at different time points within the same stage is trained in pairs. The supervision model predicts that the stage remaining completion degree of the earlier state is higher than that of the later state, thereby enhancing the model's understanding and perception of stage progress.
[0023] 3.2: By using a joint loss of stage description, action termination signal and stage remaining completion, the vision-language-action model is fine-tuned to ensure that action generation is consistent with stage trajectory, while improving the ability to perceive stage progress and task completion status.
[0024] 4) Utilizing preference data annotated by human experts, through stage-based trajectory preference optimization, the vision-language-action model can adjust its action generation strategy based on stage-level preference relationships. Specifically, this includes:
[0025] 4.1: Extract stage trajectories for the same task and the same stage from the expert demonstration data divided into stages, calculate the similarity between the first and last frames of the trajectory pairs, and sample the K stage pairs with the highest similarity to assist human experts in preference labeling.
[0026] 4.2: Human experts annotate the preferences of K stage pairs for each stage of each task, forming a stage-level preference dataset;
[0027] 4.3: In offline mode, based on stage-level preference data, a stage-level trajectory preference optimization algorithm is used for preference-supervised optimization, which enables the vision-language-action model to improve the generation probability of more preferred stage trajectories.
[0028] 5) Construct state-based reward signals and obtain real trajectory data through interaction with the environment to perform online policy optimization on the vision-language-action model, specifically including:
[0029] 5.1: Deploy the stage-based reasoning-based visual-language-action model, which has been optimized through imitation learning and stage preferences, into the simulation environment and allow it to interact with the environment in multiple rounds to collect real trajectory data;
[0030] 5.2: Extract the remaining completion of each state of each collected trajectory, and use the difference between the initial and final remaining completion of the stage to which the state belongs as the stage progress. The stage progress score of each state is the change of its remaining completion relative to the initial stage. The stage progress of all previous stages is added together to form the cumulative stage progress reward.
[0031] 5.3: Using an online policy optimization method, the stage progress reward is used as the input of the advantage function. The proximal policy optimization (PPO) algorithm is used to iteratively update the vision-language-action model to improve the long-term task completion capability and stage consistency of action generation.
[0032] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:
[0033] 1) This invention divides the expert demonstration trajectory into stages and semantic descriptions, explicitly decomposes the continuous action sequence into a representation with a stage structure, and extends the stage description and stage completion progress as intermediate reasoning conditions to the vision-language-action model, so that the model has the ability to perceive the semantics of the current stage and the execution progress while generating actions, thereby improving the stage consistency and causal rationality of action generation.
[0034] 2) This invention proposes a unified stage optimization framework that follows the strategy of "imitation-preference-interaction". First, it aligns the stage-level expert demonstration trajectory through imitation learning, then performs offline preference optimization using stage-level human preference data, and finally continuously optimizes the strategy in the environment through stage progress-driven online interactive reinforcement learning. This effectively alleviates the credit allocation problem in long-range trajectory optimization and significantly improves the robot's execution stability and success rate in complex multi-stage tasks. Attached Figure Description
[0035] Figure 1 This is a flowchart of the present invention;
[0036] Figure 2 is a schematic diagram of the specific operation of Example 1. Detailed Implementation
[0037] To effectively model the stage semantics and action generation process of robot execution trajectories, this invention proposes a stage division and stage semantic description method based on a multimodal large language model. This method explicitly decomposes continuous robot execution trajectories into stage structures with clear semantics and termination conditions. Then, a stage-based reasoning vision-language-action model is designed, using stage descriptions and remaining stage completion as intermediate reasoning conditions to autoregressively predict the current stage state and fuse multimodal state features to generate action sequences. Finally, a unified stage optimization framework is proposed, following a "imitation-preference-interaction" strategy: first, aligning with expert demonstration trajectories through imitation learning; then, utilizing stage-level human preference data for offline preference optimization; and finally, continuously optimizing in the environment through stage progress-driven online interactive reinforcement learning. This improves the stage consistency and causal rationality of action generation in long-term tasks.
[0038] See appendix Figure 1 A method for training a visual-language-action model based on stage-based reasoning and stage-based optimization includes the following steps:
[0039] 1) Using a multimodal large language model and a stage encoding method, the robot trajectory is segmented into stages and its semantic description is completed, as follows:
[0040] 1.1: Collect and standardize robot trajectory data from expert demonstrations, wherein the robot trajectory data includes: visual observations, action sequences, and verbal commands;
[0041] 1.2: Input the robot trajectory data into a multimodal large language model to divide it into stages, and use a 4-bit binary description label for each stage. The description label includes: whether it moves, the state of the mechanical gripper, and whether it contacts an object.
[0042] 1.3: Generate a fine-grained description for each action, the fine-grained description including: stage description and whether the action is a stage termination signal;
[0043] 1.4: The processed robot trajectory data is stored in stages to form a stage-labeled dataset, which provides input for subsequent stage inference model training and preference optimization. The stage-labeled data includes: visual observations, action sequences, language commands, and fine-grained descriptions of actions.
[0044] 2) The extended pre-trained vision-language-action model is a stage-based reasoning model for robot action generation that uses stage descriptions and stage remaining completion as intermediate reasoning conditions, as detailed below:
[0045] 2.1: Input visual observations, language instructions, and stage termination signals into a vision-language joint coding network to extract multimodal fusion state feature representations;
[0046] 2.2: Utilizing a self-attention mechanism to autoregressively predict the current stage description and remaining stage completion, as detailed below:
[0047] 2.2.1: Using multimodal state feature representations as input, a stage-conditional self-attention module is used to model the dependency relationship between the current state and historical states, highlighting actions and visual features related to the current stage;
[0048] 2.2.2: The current stage description is generated step by step using an autoregressive method, and the remaining completion rate of the stage is predicted simultaneously to represent the completion progress of the current stage;
[0049] 2.3: The model predicts action sequences based on the generated stage information, as follows:
[0050] 2.3.1: The generated stage description, stage remaining completion degree, and multimodal state features are fused and input into the action generation network to autoregressively generate the current action sequence;
[0051] 2.3.2: Add a lightweight completion detection head to the feature layer of the action generation network to output the stage completion probability;
[0052] 2.3.3: When the probability of a stage completion is lower than a set threshold, a stage completion signal is generated, and the stage termination signal is used as model input to guide the switching of stage prediction.
[0053] 3) Supervised learning methods are used to ensure that the output of the stage-based reasoning visual-language-action model is consistent with the stage annotations in the stage-level expert demonstration data, as follows:
[0054] 3.1: The expert demonstration data labeled with the stage is input into the pre-trained visual-language-action model as a supervision signal for model training. The state at different time points within the same stage is trained in pairs. The supervision model predicts that the remaining completion degree of the stage in the early stage is higher than that in the later stage, thereby enhancing the model's understanding and perception of the stage progress. The expert demonstration data includes: visual observation, language instructions, stage description, action sequence and stage termination signal.
[0055] 3.2: By using a joint loss of stage description, action termination signal and stage remaining completion, the vision-language-action model is fine-tuned to ensure that action generation is consistent with stage trajectory, while improving the ability to perceive stage progress and task completion status.
[0056] 4) Utilizing preference data annotated by human experts, and through stage trajectory preference optimization, the vision-language-action model can adjust its action generation strategy based on stage-level preference relationships, as detailed below:
[0057] 4.1: Extract stage trajectories for the same task and the same stage from the expert demonstration data divided into stages, calculate the similarity between the first and last frames of the trajectory pairs, and sample the K stage pairs with the highest similarity to assist human experts in preference labeling.
[0058] 4.2: Human experts annotate the preferences of K stage pairs for each stage of each task, forming a stage-level preference dataset;
[0059] 4.3: In offline mode, based on stage-level preference data, a stage-level trajectory preference optimization algorithm is used for preference-supervised optimization, which enables the vision-language-action model to improve the generation probability of more preferred stage trajectories.
[0060] 5) Construct a state-based reward signal and obtain real trajectory data through interaction with the environment to perform online policy optimization on the vision-language-action model, as detailed below:
[0061] 5.1: Deploy the stage-based reasoning-based visual-language-action model, which has been optimized through imitation learning and stage preferences, into the simulation environment and allow it to interact with the environment in multiple rounds to collect real trajectory data;
[0062] 5.2: Extract the remaining completion of each state of each collected trajectory, and use the difference between the initial and final remaining completion of the stage to which the state belongs as the stage progress. The stage progress score of each state is the change of its remaining completion relative to the initial stage. The stage progress of all previous stages is added together to form the cumulative stage progress reward.
[0063] 5.3: Using an online policy optimization method, the stage progress reward is used as the input of the advantage function. The proximal policy optimization (PPO) algorithm is used to iteratively update the vision-language-action model to improve the long-term task completion capability and stage consistency of action generation.
[0064] The invention will be further described below with reference to specific examples and accompanying drawings.
[0065] Example 1
[0066] See Figure 2 In this embodiment, robot demonstration data from the simulation environment is input into a multimodal large language model, and fine-grained annotation of actions is completed using a multi-dimensional stage coding method. The specific process is as follows:
[0067] 1) In the robot model architecture, the robot's current visual observations and language commands are input into the pre-trained OpenVLA-OFT model for multimodal feature extraction. Then, the stage-conditional self-attention module is used to autoregressively predict the current stage description and the remaining stage completion as intermediate inference conditions.
[0068] 2) The stage description, stage remaining completion degree and multimodal state features are fused and input into the action generation network. The action sequence is generated by autoregression and the stage completion signal is predicted by a lightweight completion detection head.
[0069] 3) In the optimization phase, this embodiment adopts a unified “imitation-preference-interaction” approach. In the imitation phase, the expert demonstration trajectory labeled at the stage level is aligned with the stage-level fine-tuning (SFT). In the preference phase, the stage-level human preference data is used to adjust the strategy using the stage preference optimization (TPO) to complete the preference optimization. In the interaction phase, the real trajectory obtained by the environmental interaction is continuously optimized using the stage progress-driven proximal policy optimization (PPO). This improves the consistency of action generation, causal rationality and execution stability of the model in complex multi-stage tasks.
[0070] The above description is only a preferred embodiment of the present invention. Modifications may be made within the scope defined by the claims of the present invention, but all such modifications shall fall within the protection scope of the present invention.
Claims
1. A method for training a visual-language-action model based on stage-based reasoning and stage-based optimization, characterized in that, The method includes the following steps: 1) Using a multimodal large language model and a stage coding method, the robot trajectory is segmented into stages and semantically described. 2) Extend the pre-trained vision-language-action model and use stage descriptions and stage remaining completion as intermediate inference conditions to create a stage-based inference model for robot action generation; 3) By using supervised learning methods, the output of the stage-based reasoning visual-language-action model is made consistent with the stage annotations in the stage-level expert demonstration data; 4) By utilizing preference data annotated by human experts and optimizing stage trajectory preferences, the visual-language-action model can adjust its action generation strategy based on stage-level preference relationships. 5) Construct state-based reward signals and obtain real trajectory data through interaction with the environment to optimize the vision-language-action model online.
2. The visual-language-action model training method based on stage reasoning and stage optimization according to claim 1, characterized in that, Step 1) specifically includes: 1.1: Collect and standardize robot trajectory data demonstrated by experts, the trajectory data including: visual observations, action sequences, and verbal commands; 1.2: Input the trajectory data into the multimodal large language model for stage division, and use 4 binary bits to describe each stage. The description includes: whether it moves, the state of the mechanical gripper, and whether it contacts the object. 1.3: Generate a fine-grained description for each action, including: a stage description and whether the action is a stage termination signal; 1.4: Store the processed trajectories in stages to form a stage-annotated dataset containing fine-grained descriptions of visual observations, action sequences, and language command actions.
3. The visual-language-action model training method based on stage reasoning and stage optimization according to claim 1, characterized in that, Step 2) specifically includes: 2.1: Input visual observations, language instructions, and stage termination signals into a vision-language joint coding network to extract multimodal fusion state feature representations; 2.2: Utilize self-attention mechanisms to autoregressively predict the current stage description and remaining stage completion. 2.3: The visual-language-action model uses generated stage information as a condition to predict action sequences.
4. The visual-language-action model training method based on stage reasoning and stage optimization as described in claim 1, characterized in that, Step 3) specifically includes: 3.1: Input the stage-annotated expert demonstration data into the pre-trained visual-language-action model as a supervision signal for model training, and perform pairwise training on the states at different time points within the same stage, so that the supervision model predicts the stage remaining completion degree of the earlier state higher than that of the later state. The expert demonstration data includes: visual observation, language instructions, stage description, action sequence and stage termination signal. 3.2: Fine-tune the vision-language-action model by using a joint loss of stage description, action termination signal, and stage remaining completion degree to ensure that action generation is consistent with stage trajectory.
5. The visual-language-action model training method based on stage reasoning and stage optimization as described in claim 1, characterized in that, Step 4) specifically includes: 4.1: Extract stage trajectories for the same task and the same stage from the expert demonstration data divided into stages, calculate the similarity between the first and last frames of the trajectory pairs, and sample the K stage pairs with the highest similarity to assist human experts in preference labeling. 4.2: Human experts are used to annotate the K stage pairs for each stage of each task, forming a stage-level preference dataset; 4.3: In offline mode, based on stage-level preference data, a stage-level trajectory preference optimization algorithm is used for preference-supervised optimization, which enables the vision-language-action model to improve the generation probability of more preferred stage trajectories.
6. The visual-language-action model training method based on stage reasoning and stage optimization as described in claim 1, characterized in that, Step 5) specifically includes: 5.1: Deploy the stage-based reasoning-based visual-language-action model, which incorporates imitation learning and stage preference optimization, into a simulation environment and allow it to interact with the environment in multiple rounds to collect real trajectory data; 5.2: Extract the remaining completion of each state of each trajectory, and use the difference between the initial and final remaining completion of the stage to which the state belongs as the stage progress. The stage progress score of each state is the change of its remaining completion relative to the initial stage. The stage progress of all previous stages is added together to form the cumulative stage progress reward. 5.3: Using an online policy optimization method, the stage progress reward is used as the input of the advantage function. The near-end policy optimization algorithm is used to iteratively update the vision-language-action model to improve the long-term task completion capability and stage consistency of action generation.
7. The visual-language-action model training method based on stage reasoning and stage optimization according to claim 3, characterized in that, Step 2.2 specifically includes: 2.2.1: Using multimodal state feature representations as input, a stage-conditional self-attention module is used to model the dependency relationship between the current state and historical states, highlighting actions and visual features related to the current stage; 2.2.2: The current stage description is generated step by step in an autoregressive manner, and the remaining completion rate of the stage is predicted simultaneously to represent the completion progress of the current stage.
8. The visual-language-action model training method based on stage reasoning and stage optimization according to claim 3, characterized in that, Step 2.3 specifically includes: 2.3.1: The generated stage description, stage remaining completion degree and multimodal state features are fused and input into the action generation network to generate the current action sequence through autoregression; 2.3.2: Add a lightweight completion detection head to the feature layer of the action generation network to output the stage completion probability; 2.3.3: When the probability of a stage completion is lower than a set threshold, a stage completion signal is generated, and the stage termination signal is used as model input to guide the switching of stage prediction.