A vla model autonomous generalization method, system, device and medium
By constructing a memory-reflex reward model and an experience-driven generator, and combining reinforcement learning and supervised fine-tuning, the problems of high data acquisition costs and large gap between simulation and reality in cross-scenario generalization of VLA models are solved, enabling autonomous reflection and continuous optimization, and improving the model's generalization ability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WESTLAKE UNIV
- Filing Date
- 2026-02-04
- Publication Date
- 2026-06-09
Smart Images
Figure CN121638318B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot control technology, and in particular to a method, system, device and medium for autonomous generalization of VLA models. Background Technology
[0002] With the rapid development of artificial intelligence and robotics, Vision-Language-Motion (VLA) models have become a core research area in embodied intelligence. VLA models aim to empower robots to understand natural language instructions and perform specific actions in complex environments by integrating multimodal perception information. However, existing VLA models still face many challenges in practical deployment, especially in cross-scenario generalization and efficient knowledge transfer.
[0003] Currently, the mainstream solutions for improving the operational capabilities of VLA models are mainly divided into two categories: supervised fine-tuning (SFT) and reinforcement learning (RL). Supervised fine-tuning methods heavily rely on high-quality expert demonstration data. However, obtaining large-scale and diverse real-world expert trajectories is not only extremely costly, but also limited by the scale of the data, resulting in insufficient scalability of the model. When the robot faces distributed external environments or unseen task logic, models trained solely on SFT are prone to failure and lack autonomous adaptability. Although reinforcement learning methods allow robots to achieve self-evolution through trial and error, large-scale real-world training is risky and inefficient. Most current RL training is limited to simulation environments and heavily relies on manually designed rule-based reward functions. When facing unknown or complex long-term tasks, rule-based rewards are often too sparse and cannot dynamically adjust the judgment criteria, making it difficult for the model to extract reusable knowledge from limited interaction experience.
[0004] Furthermore, in existing technologies, the training processes for simulation environments and real-world scenarios are often isolated from each other, lacking an efficient closed-loop transfer mechanism, resulting in a significant gap in generalization from simulation to reality. Although some human-machine collaborative solutions attempt to introduce human preferences for intervention, they still require a high degree of human supervision and external correction, and the models cannot achieve true autonomous reflection and adaptive evolution.
[0005] In summary, reducing data acquisition costs, constructing a dense reward mechanism with self-reflection capabilities, and narrowing the performance gap between simulation and reality have become key technical issues that urgently need to be addressed to improve the autonomous generalization capabilities of VLA models. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this invention provides a method for autonomous generalization of VLA models, aiming to solve the problems of existing technologies.
[0007] To achieve the above objectives, this application provides the following solution:
[0008] Firstly, this application provides an autonomous generalization method for a VLA model. The method includes acquiring visual observation data and natural language task descriptions of the robot's current environment; performing cross-modal feature fusion through a VLA policy model to generate action sequences and control the robot to execute tasks; collecting trajectory data during task execution and inputting it into a memory-reflection reward model, which analyzes the action sequences and outputs token-level dense rewards and structured reflection information; storing the trajectory data, token-level dense rewards, and structured reflection information in a dual-memory block, whereby the dual-memory block includes short-term memory for storing recent interaction data and long-term memory for storing high-value task priors; extracting task priors from the long-term memory using an experience-driven generator to generate trajectory data containing sub-objective chains, and iteratively correcting based on the execution results to obtain a high-quality trajectory dataset; updating the VLA policy model using a reinforcement learning algorithm based on the token-level dense rewards; and simultaneously updating the VLA policy model using a supervised fine-tuning algorithm based on the high-quality trajectory dataset.
[0009] Secondly, this application provides an autonomous generalization system for VLA models. The system includes a VLA policy model module configured to receive visual-language multimodal input, fuse visual features and linguistic intent, and output a sequence of robot control actions; a memory-reflection-reward model module connected to the VLA policy model module, configured to receive task trajectory data and output token-level dense rewards and structured reflection information, wherein the token-level dense rewards are used to evaluate the importance of each action marker in the action sequence; a dual-memory block module connected to the memory-reflection-reward model module, including short-term memory units and long-term memory units, wherein the short-term memory units store recent interaction data and reflection information, and the long-term memory units store high-value task priors selected from the short-term memory; and an experience-driven generator module connected to the dual-memory block module and the VLA policy model module, configured to extract task priors from the long-term memory units, generate and verify trajectory data, and use it for supervised fine-tuning of the VLA policy model.
[0010] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method described above when executing the program.
[0011] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.
[0012] Through the above technical solutions, the beneficial effects of this invention are as follows: by constructing a dynamic reward mechanism, the problems of reward sparsity and credit allocation in reinforcement learning are solved, thereby improving the model's performance in long-term tasks; by storing and reusing interactive experience through a dual-memory layer structure, the model can achieve autonomous reflection and continuous optimization, reducing reliance on human supervision; by designing an experience-driven data generation scheme, prior knowledge of the task can be extracted efficiently, reducing the cost of data collection in real-world scenarios and narrowing the generalization gap between simulation and reality; and a closed-loop self-evolutionary process of simulation exploration and real-world adaptation can be formed, improving the generalization performance of the VLA model in cross-task, cross-scenario, and cross-carrier scenarios. Attached Figure Description
[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention, and the embodiments in the accompanying drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0014] Figure 1 A flowchart illustrating the autonomous generalization method of the VLA model provided in an embodiment of this application;
[0015] Figure 2 This is a schematic diagram of the structure of a VLA model autonomous generalization system provided in an embodiment of this application;
[0016] Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application;
[0017] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0018] It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the application. Rather, these embodiments are provided to make the disclosure more thorough and complete, and to fully convey the scope of the disclosure to those skilled in the art.
[0019] The foregoing and other technical contents, features and effects of the present invention are described in conjunction with the appendix below. Figure 1-3 The detailed description of the embodiments will make this clear. All structural details mentioned in the following embodiments are based on the accompanying drawings.
[0020] Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings.
[0021] In one exemplary embodiment, such as Figure 1As shown, a method for autonomous generalization of VLA models is provided, including:
[0022] S101. Obtain visual observation data and natural language task description of the robot's current environment, perform cross-modal feature fusion through VLA policy model, generate action sequence and control the robot to perform task.
[0023] In this step, the VLA policy model is first initialized, which can be built upon a large-scale pre-trained base model. The robot acquires visual observation data of the current environment through its onboard sensors, including but not limited to RGB images, depth images, point cloud data, and semantic segmentation maps. Simultaneously, the system receives a natural language task description input by the user.
[0024] The visual encoding unit of the VLA policy model extracts visual features from the scene, while the language understanding unit parses the task intent and constraints. The cross-modal information fusion unit combines these two types of information to form a unified representation describing the current task context. Subsequently, the action decoding unit sequentially generates semantic action tag sequences (including movement, rotation, grasping, placement, alignment, obstacle avoidance, etc.) and further deduces them into specific robot control commands, such as the movement stride and joint angle changes of the end effector, driving the robot to perform the task.
[0025] S102. Collect trajectory data during task execution and input it into the memory reflection reward model. The memory reflection reward model analyzes the action sequence and outputs token-level dense rewards and structured reflection information.
[0026] In this step, the system records the complete trajectory data generated by the robot performing the task. This includes visual observation, verbal input, and action sequences. The Memory Reflex Reward Model (MRRM) utilizes a high-performance visual-verbal reasoning model to perform in-depth analysis of trajectories, identifying key nodes (such as successful steps, failed steps, and potentially risky steps) in the action sequence through a verbal-visual alignment mechanism.
[0027] MRRM calculates and outputs a "token-level" dense reward, which scores the importance of each action token in the action sequence and its contribution to task completion (e.g., ...). This allows for the identification of the precise location of the action that caused the task failure. Simultaneously, MRRM generates structured reflective information, including specific reasons for failure (such as "endpoint misalignment" or "low grasp height"), targeted improvement suggestions, and avoidance strategies.
[0028] S103. Store the trajectory data, token-level dense reward, and structured reflection information into a dual memory block, wherein the dual memory block includes short-term memory for storing recent interaction data and long-term memory for storing high-value task priors.
[0029] In this step, the dual memory block consists of short-term memory (STM) and long-term memory (LTM). The STM is mainly used to store data from the most recent rounds of interaction, including task descriptions, trajectory completion scores, reflection information given by MRRM, and key action points, so that the robot can quickly learn the failure modes of the latest task.
[0030] The LTM (Long-Term Knowledge Base) is used to store high-value, filtered experiences. The system periodically selects the highest-scoring and most stable experiences from the STM (Shortest-Term Knowledge Base) and writes them into the LTM through mechanisms such as cosine similarity retrieval, forming a long-term prior knowledge base for task execution. The MRRM (Most Reliable Prior Knowledge Base) also accumulates historical failure cases to form a reward prior library. When a certain failure pattern recurs, its penalty weight is automatically increased.
[0031] S104. Using an experience-driven generator, extract task priors from the long-term memory to generate trajectory data containing sub-target chains, and perform iterative correction based on the execution results to obtain a high-quality trajectory dataset.
[0032] In this step, the experience-driven generator extracts high-value task priors from the LTM, uses a visual language model to decompose complex task instructions into structured chains of sub-goals (e.g., "locate handle - adjust height - approach - grasp"), and maps each sub-goal to a robot control primitive that can be executed in the simulation environment.
[0033] The generator executes the generated sub-target sequences in the simulation environment and monitors execution deviations. If execution fails, the system automatically corrects the action flow by incorporating reflective information from MRRM feedback. This process performs a closed-loop iteration of "generation-verification-correction" until the trajectory reaches a stable and successful state. Finally, the successful trajectory is written into the generated dataset. In addition, the generator can incorporate random positional perturbations, velocity variations, and lighting simulations to enhance the real-world robustness of the data.
[0034] S105. Based on the token-level dense reward, update the VLA policy model using a reinforcement learning algorithm; simultaneously, based on the high-quality trajectory dataset, update the VLA policy model using a supervised fine-tuning algorithm.
[0035] This step achieves closed-loop self-evolutionary optimization of the strategy model, specifically including two aspects:
[0036] Reinforcement Learning (RL) Update: During the simulation exploration phase, the VLA policy model is updated using the Proximal Policy Optimization (PPO) algorithm. The optimization objective is to maximize the total reward. ,in As an environmental reward, The token-level dense reward output by MRRM enables the model to maintain stable performance in long-term tasks.
[0037] Supervised Fine-Tuning (SFT) Update: During the reality fine-tuning phase, the high-quality trajectory dataset generated in step S104 is updated. By combining VLA policy models with a small number of real-world trajectories, supervised fine-tuning (SFT) can be used to train the model (e.g., using a LoRA adapter), thereby improving the model's behavioral understanding and the transfer effect from simulation to reality.
[0038] In one specific embodiment, the output of token-level dense rewards and structured reflection information includes: performing language and visual alignment analysis on the trajectory data to identify successful steps, failed steps, and potentially risky steps in the action sequence; generating token-level dense rewards based on the importance of each action marker in the action sequence and whether it contributes to task completion, and generating structured reflection information for failed steps, including reasons for failure, improvement suggestions, and avoidance strategies; and increasing the penalty weight corresponding to a failure pattern when it recurs in multiple tasks.
[0039] First, a language-visual alignment analysis is performed. The MRRM memory reflex reward model receives complete trajectory data containing visual observations, language commands, and action sequences. Using a pre-built high-performance visual-language inference model, deep semantic analysis is performed on the trajectory, mapping discrete action frames to abstract language commands through a language-visual alignment mechanism. During this process, the model automatically identifies key execution nodes in the action sequence, classifying these nodes as successful steps (e.g., "successfully approaching the target"), failed steps (e.g., "grab slippage"), and potentially risky steps (e.g., "moving too fast causing end effect jitter").
[0040] Secondly, it generates token-level dense rewards. To address the issues of sparse rewards and difficulty in credit allocation in traditional reinforcement learning, MRRM does not simply provide a final score for the entire trajectory, but rather generates fine-grained "token-level" rewards for the task execution process. Specifically, the system progressively analyzes the importance of each action token in the action sequence and provides corresponding positive or negative evaluations based on whether the action propels the task state closer to the goal (e.g., calculating the probability of success). In this way, the system can accurately identify the specific action that caused the task to fail, rather than making a general evaluation based solely on the final result.
[0041] Next, structured reflection information is generated. For the identified failed steps, MRRM further generates explanatory structured reflection text. This information mainly includes three dimensions: failure cause: the physical or logical attribution of the failure phenomenon, such as "endpoint misalignment," "grab height too low," or "movement too fast causing offset"; improvement suggestions: specific corrective measures for the current error, such as "reducing approach speed" or "adding alignment steps before grabbing"; avoidance strategy: avoidance logic for similar situations in the future. This reflection information will be formatted and written into short-term memory to guide subsequent strategy modifications.
[0042] Finally, dynamic weight adjustments are made for failure modes. MRRM maintains an internal reward prior library to accumulate failure cases from historical tasks. The system continuously monitors the frequency of failure modes, and when a specific failure mode recurs in multiple tasks, the system automatically increases the penalty weight corresponding to that mode. This dynamic adjustment mechanism makes the policy model more sensitive to such high-frequency errors during subsequent training, thus proactively avoiding similar risks in future tasks, forming a reflexive reward system that becomes smarter with each failure.
[0043] In one specific embodiment, the dual memory card update and transfer mechanism includes: writing the description of the current task, trajectory score, reflection information and key action points into short-term memory; retrieving similar task experiences in short-term memory through cosine similarity, filtering high-value experiences and transferring them to long-term memory; the task priors in long-term memory include typical action sequences, physical constraint rules and semantic behavior rules.
[0044] First, the Short-Term Memory (STM) is written to and updated. After each round of task execution, the system automatically integrates the complete data packet for the current interaction cycle. This data packet specifically includes: a task description in natural language, a trajectory score calculated based on environmental feedback and MRRM scoring, structured reflection information generated by MRRM (including reasons for failure and improvement suggestions), and key action points during execution. The system uses a "first-in, first-out (FIFO)" or importance-based replacement strategy to write this information into the STM block. The STM is typically designed with a small capacity (e.g., storing the most recent 50 records) to retain the latest environmental feedback and trial-and-error experience, allowing the model to quickly correct consecutive failures caused by minor environmental changes or operational errors in the short term.
[0045] Secondly, experience retrieval and filtering based on cosine similarity are performed. To extract valuable knowledge from massive short-term interactions, the system periodically triggers a memory retrieval mechanism (e.g., after every 5 trajectory scrolls). Specifically, the system compares the feature vector of the current task with the historical task vectors stored in short-term memory, and uses the cosine similarity algorithm to calculate the similarity between tasks. Based on the similarity score, the system retrieves the historical experience most relevant to the current scene, and further combines it with the trajectory score to filter out the high-value experiences with the highest scores and most stable performance.
[0046] Finally, the transfer and prior knowledge construction of long-term memory (LTM) are performed. High-value experiences selected are transferred and solidified into long-term memory blocks (LTM), forming a reusable task prior library across tasks. The task priors in long-term memory are no longer simple trajectory replays, but rather abstracted general knowledge, specifically including: Typical Action Sequences: standard operating procedures for specific types of tasks, such as storing standardized steps for the door-opening task: locating the handle—adjusting the height—approaching—grabbing—rotating—pushing; Physical Constraint Laws: constraints on the object in the physical environment, such as "keeping the end level when grasping an object with a handle" or "limiting acceleration when moving a liquid container"; Semantic Behavioral Rules: high-level logical rules based on scene understanding, such as "confirming there are no obstacles in the target area before placing an object" or "handling fragile items with care." Through the above mechanism, the system realizes knowledge dimensionality upgrade from short-term trial and error to long-term cognition, enabling the VLA model to use the prior knowledge in LTM to achieve rapid generalization when facing new but similar tasks.
[0047] In one specific embodiment, obtaining a high-quality trajectory dataset includes: decomposing the task prior in the long-term memory into an executable sub-target sequence and mapping the sub-targets to robot control primitives; executing the generated sub-target sequence in a simulation environment and monitoring execution deviations; if execution fails, automatically correcting the action flow by combining the reflection information fed back by the memory reflection reward model, performing closed-loop iteration until the trajectory is stable and successful; and writing the successful trajectory into the high-quality trajectory dataset.
[0048] First, task decomposition and primitive mapping are performed. The experience-driven generator extracts abstract prior knowledge of the task from long-term memory (LTM). Utilizing a built-in visual language model (VLM), the generator decomposes complex, long-sequential task instructions (such as "clean up the clutter on the table") into a series of structured sub-objective sequences with a logical order. For example, it decomposes them into sub-objectives such as "identify clutter," "plan movement path," "grasp object," "find trash can," and "dispose." Subsequently, the generator further maps each sub-objective to low-level control primitives that the robot can directly execute, such as specific inverse kinematics instructions, force control parameters, or predefined skill operators, ensuring that the generated action sequence is physically executable.
[0049] Secondly, simulation verification and deviation monitoring are performed. The generated sub-target sequences and their corresponding control primitives are input into a highly realistic simulation environment for simulated execution. During the simulation, the robot's execution state is monitored in real time and compared with the expected sub-target state to calculate the execution deviation. Deviation monitoring includes spatial pose deviation, contact force feedback deviation, and task progress deviation. By conducting preliminary verification in the simulation environment, potentially feasible action schemes can be screened without consuming real-world resources.
[0050] Secondly, a closed-loop iterative correction based on reflective information is executed. If a failure is encountered during simulation (such as a collision, a missed grab, or failure to reach the target state), the attempt is not simply discarded, but a closed-loop iterative mechanism is triggered. The generator, combined with the Memory Reflection Reward Model (MRRM), automatically corrects the action flow based on the structured reflective information output for this failure (such as the grab point being too biased towards one side of the center of gravity). The correction process includes adjusting the parameters of the control primitives, optimizing the logical order of sub-objectives, or regenerating the action trajectory. This closed-loop process of "generation-verification-reflection-correction" continues until the task sequence can stably and continuously achieve success in the simulation.
[0051] Finally, a high-quality trajectory dataset is constructed. Once an action sequence passes the stability test in the simulation environment, the system records the complete data stream of that success, including multi-view visual observations, proprioceptive states, and precise action tokens, and writes it into the high-quality trajectory dataset. Because this data has undergone rigorous verification and multiple corrections in the simulation environment, its data distribution is closer to the level of expert demonstrations. In this way, this embodiment can automatically generate massive amounts of labeled, high-quality training data with extremely low manual costs, providing core support for the subsequent supervised fine-tuning (SFT) of the VLA policy model.
[0052] In one specific embodiment, generating trajectory data containing sub-target chains includes: adding random position perturbations, velocity changes, and illumination simulations when generating the trajectory; mapping the generated simulation prior to the real trajectory; and performing spatial perturbation processing by combining inverse kinematics calculations.
[0053] To improve the robustness of the VLA strategy model under different physical environments, the experience-driven generator actively introduces various randomization perturbations when generating trajectory data in the simulation environment. These include: positional random perturbation: when generating sub-target points, random coordinate offsets are added within the legal space to simulate subtle differences in object placement in reality; velocity variation simulation: a dynamic range is set for the robot's end effector's movement speed to generate motion trajectories covering different acceleration and deceleration configurations, enabling the model to adapt to tasks with varying dynamic response requirements; and lighting and visual simulation: when rendering trajectory images, the lighting intensity, light source position, and camera noise parameters of the simulation environment are randomly adjusted. Through these perturbation processes, the generator's dataset can cover a wider distribution space, effectively preventing the model from overfitting to a single simulation scene.
[0054] The simulation priors, extracted from long-term memory and verified through simulation, are transformed into guidance trajectories suitable for real-world deployment. During this transformation, the generator performs spatial adaptation calculations using the robot's inverse kinematics (IK) algorithm. Specifically, based on the physical constraints of the real robot, such as link lengths and joint limits, the system maps the end-effector pose points in the simulation space to continuous paths within the real joint space. To further narrow the generalization gap between simulation and reality during this mapping process, spatial perturbation is performed. This perturbation simulates potential mechanical errors or flexible deformations that may exist in the real robot by introducing minute joint angle compensations or impedance control parameter perturbations during the IK calculation process.
[0055] The trajectory data, after the aforementioned perturbation and mapping processes, not only retains the original successful logical chain but also possesses extremely high environmental diversity. This data, together with a small number of real-world trajectories, constitutes a fine-tuning dataset, which can induce the VLA policy model to learn the essential characteristic of being insensitive to environmental noise. This ensures that the model can still stably execute complex sub-target sequences in complex and variable lighting and physical contact environments in the real world.
[0056] In one specific embodiment, updating the VLA policy model includes: during the simulation exploration phase, the VLA policy model maximizes the total reward through the PPO algorithm, wherein the total reward is a weighted average of the environmental reward and the reward output by the memory reflex reward model; during the reality fine-tuning phase, the dataset generated by the experience-driven generator is combined with the real-world acquisition trajectory, and the VLA policy model is optimized through supervised fine-tuning.
[0057] Reinforcement learning optimization during the simulation exploration phase: In this phase, the VLA policy model undergoes large-scale parallel exploration in a highly controlled simulation environment. The Proximal Policy Optimization (PPO) algorithm is used as the core update algorithm. The goal of model optimization is to maximize the preset total reward function. The total reward consists of two weighted components: environmental rewards. : Sparse reward given by the simulation system based on whether the task is ultimately completed (e.g., whether the object is placed in the designated container); Memory reflection reward model reward : The token-level dense reward output by the aforementioned memory-reflection reward model. The total reward formula can be expressed as: = ,in These are weighting coefficients. By introducing... The model can obtain timely behavioral feedback in complex long-term tasks, effectively solving the problem that traditional reinforcement learning is difficult to converge in the early stages of a task due to the sparse rewards, and guiding the model to learn more logical intermediate actions.
[0058] Supervised learning optimization in the reality fine-tuning stage: After the model accumulates basic operational skills through the simulation stage, the system enters the reality fine-tuning stage, aiming to eliminate the differences between simulation and reality. In this stage, the system constructs a hybrid fine-tuning dataset. This dataset consists of two parts: a generated dataset: high-quality simulation / perturbation trajectories extracted from long-term memory and corrected by the aforementioned experience-driven generator; and a reality trajectory dataset: real interaction data collected on a real robot platform through a small number of expert demonstrations or autonomous attempts. Using this hybrid dataset, the parameters of the VLA policy model are updated using a supervised fine-tuning SFT algorithm (such as using efficient fine-tuning schemes like low-rank adaptive LoRA). In this way, the model can maintain its generalization ability by leveraging the large-scale diversity of the generated data, while using real-world data to align the differences between physical realism and visual perception, thereby achieving autonomous generalization in complex real-world scenarios.
[0059] Through the alternation or sequential iteration of the above two stages, the VLA strategy model can not only quickly learn new skills in simulation, but also maintain a very high success rate and stability in real-world deployment, ultimately achieving autonomous evolution of the model.
[0060] Compared with existing technologies, the proposed method for autonomous generalization of VLA models using an experience-driven generator has significant technical advantages. The proposed method was compared with several mainstream baseline methods on four LIBERO task suites (LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long). On all four task suites, the proposed method achieved an average task success rate of 84.9% in VLA-SE, ranking first among all methods. This represents an improvement of approximately 3.9 percentage points in average success rate compared to the best baseline method and approximately 12.5 percentage points compared to the lowest-performing baseline method. These results demonstrate that the proposed method maintains stable performance across different task types (spatial constraint tasks, object manipulation tasks, goal-oriented tasks, and long-term tasks), significantly improving the robot's generalization and robustness in multi-task and multi-scenario environments.
[0061] Based on the same inventive concept, this application also provides a VLA model autonomous generalization system. The solution provided by this system is similar to the solution described in the above method. Therefore, the specific limitations in one or more method embodiments provided below can be found in the limitations described above, and will not be repeated here.
[0062] In one exemplary embodiment, such as Figure 2 As shown, a VLA model autonomous generalization system is provided, including:
[0063] The VLA strategy model module is configured to receive visual-language multimodal input, fuse visual features and linguistic intent, and output a sequence of robot control actions.
[0064] The VLA policy model module described in this application is primarily responsible for fusing visual information, language descriptions, and historical interaction information to generate action sequences suitable for robot control. This module typically consists of a visual encoding unit, a language understanding unit, a cross-modal information fusion unit, and an action decoding unit, and can be built using existing large-scale pre-trained basic models, such as OpenVLA or IsaacGR00T.
[0065] During system operation, the robot first acquires visual observations from the environment, such as RGB images, depth images, point clouds, and semantic segmentation maps; simultaneously, it receives a natural language task description input by the user. The visual encoding unit processes the scene to obtain visual features including target location, object structure, and interaction relationships; the language understanding unit parses the task intent and constraints; and the cross-modal fusion unit combines the two types of information to form a unified representation describing the current task context.
[0066] After obtaining the cross-modal fusion results, the motion decoding unit generates a sequence of motion tags in sequence. These tags can include semantic operation instructions such as movement, rotation, grasping, placement, alignment, and obstacle avoidance. Based on the motion tags, the system further calculates specific robot control instructions, such as the movement stride of the end effector, the adjustment of gripper force, and the change of joint angles, to achieve motion execution.
[0067] During the policy optimization phase, this module can continuously upgrade the model by combining supervised fine-tuning and reinforcement learning. Supervised fine-tuning utilizes high-quality trajectory data created by the generator to improve the model's behavioral understanding ability; reinforcement learning adjusts the policy during interactions by incorporating reward signals (including environmental rewards and MRRM rewards), enabling the robot to maintain stable performance in long-term tasks. Through this design, the VLA policy model can not only understand complex natural language tasks but also continuously self-optimize during long-term interactions.
[0068] The memory reflection reward model module, connected to the VLA policy model module, is configured to receive task trajectory data and output token-level dense rewards and structured reflection information. The token-level dense rewards are used to evaluate the importance of each action token in the action sequence.
[0069] The Memory Reflex Reward Model (MRRM) is the core innovative module of this application. Its main function is to transform the originally sparse, end-feedback reward signal into a dense reward that is interpretable, continuous and traceable, thereby guiding the policy model to learn key behaviors in the task steps more effectively.
[0070] After the system executes a task, the robot generates a complete motion trajectory, including visual observation, verbal input, action sequence, and execution result. The MRRM module uses a high-performance visual-language reasoning model to perform in-depth analysis of the trajectory and identifies key nodes in the action sequence through a language-visual alignment mechanism, including successful steps, failed steps, and potentially risky steps.
[0071] MRRM generates "token-level" rewards for task execution, which involves progressively analyzing the importance of each action token in the action sequence and assigning positive or negative evaluations based on whether it contributes to task completion. This allows the system to identify the precise location of the action that caused the task to fail, rather than simply scoring based on whether the task was ultimately successful.
[0072] In addition, MRRM generates structured reflective information, including reasons for failure, improvement suggestions, and avoidance strategies. For example, for a "grabbing failure" task, MRRM might point out fine-grained reasons such as "endpoint misalignment," "grabbing height too low," or "movement too fast causing offset." This reflective information is written into short-term memory and also used for reward adjustments in subsequent tasks.
[0073] It's worth emphasizing that MRRM builds a reward prior library by accumulating historical failure cases. When a failure pattern recurs in multiple tasks, the system automatically increases the penalty weight of that failure pattern, enabling the policy model to proactively avoid similar risks in future tasks. Through this dynamic reward adjustment mechanism, MRRM forms a reflexive reward system that "gets smarter with each failure," thereby significantly improving the robot's stability and success rate in complex tasks.
[0074] The dual-memory block module, connected to the memory reflection reward model module, includes a short-term memory unit and a long-term memory unit. The short-term memory unit stores recent interaction data and reflection information, while the long-term memory unit stores high-value task priors selected from the short-term memory.
[0075] The dual-memory block module comprises two parts: Short-Term Memory (STM) and Long-Term Memory (LTM), serving as a crucial structure to support the robot's continuous self-evolutionary learning. The two memory units each handle different levels of knowledge storage functions and collaborate bidirectionally with the MRRM and the generator to form a complete memory loop system.
[0076] Short-Term Memory (STM) is primarily used to store data from recent rounds of interactions, including task descriptions, trajectory completion scores, reflection information from MRRM (Match Response Management), and key action points. The STM's capacity is designed to quickly acquire failure modes and successes from the latest tasks, enabling the robot to rapidly adjust its strategy within a short period. For example, when the robot repeatedly makes "grasp offset" errors in similar tasks, the STM will mark this pattern as high-risk and inject warning information into the policy model before the next similar task, thus avoiding repeated failures.
[0077] Long-term memory (LTM) is used to filter the best-performing and most stable high-value experiences from the standard memory (STM). These experiences typically include trajectories with high success rates, reliable action sequences, and strong deterministic physical constraints. The system selects the highest-scoring experiences from the STM and writes them into the LTM as a long-term prior knowledge base for task execution. The knowledge in the LTM has stronger generalization capabilities; for example, experiences such as "how to grasp an object with a handle" or "how to move a lightweight object from a table" can be transferred across tasks. The dual-memory block design allows the robot to utilize both short-term corrections and long-term stability judgments simultaneously during task execution, thereby achieving a unity of immediate adaptation and continuous learning.
[0078] The experience-driven generator module connects the dual-memory block module and the VLA policy model module. It is configured to extract task priors from long-term memory units, generate and verify trajectory data, and use them for supervised fine-tuning of the VLA policy model.
[0079] The experience-driven generator module is responsible for extracting task priors from the long-term memory (LTM), organizing them into a structured chain of sub-objectives, and generating high-quality trajectory data through simulation and real-world execution verification. This enables the robot to accumulate supervised training samples during continuous task execution, achieving self-evolving policy improvement.
[0080] First, the generator extracts high-value task experience from the LTM. This experience usually exists in the form of "task priors," including typical action sequences, physical constraints, spatial layout logic, and semantic behavior rules. For example, for the "opening a door" task, the priors may include structured steps such as "locating the handle—adjusting the height—approaching—grabbing—rotating—pushing."
[0081] Secondly, the generator uses a visual language model to decompose each task instruction into sub-goals, breaking down complex tasks into executable key steps. Each sub-goal is mapped to a robot control primitive that can be executed in the simulation, such as "move to the vicinity of the target", "adjust alignment", "slowly descend to grasp", etc.
[0082] The generator then executes the generated sub-target sequence in the simulation environment, monitoring for deviations or failures during execution. If execution fails, the system automatically corrects the action flow by incorporating reflective information from MRRM feedback, such as adjusting the grab angle, inserting additional alignment processes, or adding buffer pauses. The generator will continue to perform a closed-loop iteration of "generation-verification-correction" until the trajectory reaches a stable and successful state.
[0083] Ultimately, successful trajectories are written into the generated dataset to enhance supervised fine-tuning of the policy model. Furthermore, the generator supports data collection on real robots, improving the real-world robustness of the trajectories by incorporating random position perturbations, velocity variations, and lighting simulations.
[0084] By using an experience-driven generator, this application achieves an efficient combination of simulation and reality, enabling the robot to continuously generate high-quality data that supports self-evolutionary training while reducing the cost of manual demonstration.
[0085] In one specific embodiment, the memory reflex reward model module is constructed through a visual language reasoning model to identify fine-grained reasons for task failure and accumulate failure patterns to form a reward prior library; the VLA policy model module receives reward signals from the memory reflex reward model module for reinforcement learning and receives trajectory data from the experience-driven generator module for supervised fine-tuning.
[0086] The system described in this application was compared and evaluated with the Isaac-GR00T method on a real Uniree G1 humanoid robot platform, with the task settings consistent with the simulation experiments. The system deployment environment included: hardware configuration: an 8×A100 GPU cluster, with one dedicated GPU running MRRM inference via vLLM, and the remaining GPUs responsible for trajectory scrolling and PPO optimization; the real-world deployment used a Uniree G1 humanoid robot equipped with a Dex3-1 7-DOF robotic arm and an Intel RealSense D435i RGB-D camera; software environment: based on the Isaac Sim simulation platform, using the PyTorch framework, with PPO training configuration γ=0.98, clip ε=0.2, LoRA adapter rank=16, and an AdamW optimizer learning rate of 1×10⁻⁶. -4 .
[0087] The workflow includes: Initialization phase: Loading the pre-trained VLA base model (OpenVLA-7B / IsaacGR00T), initializing MRRM, dual memory blocks and experience-driven generator, setting the STM capacity to 50 and the LTM capacity to 10; Simulation exploration phase: The VLA policy model executes tasks in the simulation environment to generate trajectory data. MRRM receives trajectory and memory data, calculates token-level rewards and structured reflections, and updates STM; after every 5 trajectory rolls, MRRM triggers memory retrieval and reward optimization, and LTM updates high-value priors; the policy model maximizes the total reward through the PPO algorithm. The strategy is updated; in the data generation phase, the experience-driven generator extracts the task priors from the LTM, decomposes the sub-objectives to generate the initial trajectory, and after simulation verification and iterative correction, a high-quality trajectory dataset is formed. Real-world fine-tuning phase: Combined with a small number of real-world collected trajectories, the VLA strategy model is optimized through supervised fine-tuning and deployed to humanoid robots to perform real-world tasks; in the continuous evolution stage, real-world task execution data is fed back to MRRM to update memory blocks and prior libraries, and the simulation exploration and real-world fine-tuning process is repeated to achieve continuous self-evolution of the model.
[0088] This system supports multiple scheme adaptations: Reward mechanism adaptation: For short task scenarios, MRRM can be simplified to a "token-level reward + STM" structure; for long-term tasks, a complete MRRM and LTM prior library are enabled; Generator optimization: A lightweight sub-objective decomposition algorithm is used for low-resource scenarios, and a diffusion model can be introduced to improve trajectory generation quality for high-resource scenarios; Hardware adaptation: Single-arm robot scenarios do not require dual-arm coordination priors, and humanoid robot scenarios enhance the weight of spatial physical constraint priors.
[0089] In real-world robot testing, the Isaac-GR00T achieved an average success rate of approximately 12.0% and an average of 1220 reasoning steps. In contrast, our system, under the same task conditions, achieved an average success rate of 52.0% and an average of approximately 943 reasoning steps. This invention demonstrates an average success rate improvement of approximately 40 percentage points in a real-world environment, while reducing the average number of execution steps by approximately 23%, significantly decreasing the length of actions and time required to complete the task.
[0090] More importantly, in terms of performance drop from simulation to real deployment, the average success rate of this application decreased by only about 8 percentage points, while the performance drop of the baseline method was about 24 percentage points. This demonstrates that the closed-loop joint optimization and experience-driven generation mechanism constructed in this application can effectively mitigate the distribution differences between simulation and real environments, and improve the transferability and deployability of the strategy.
[0091] In summary, by introducing a memory-reflection reward model, dual memory blocks, and an experience-driven generator, this application significantly improves the success rate: In multi-task benchmark kits and humanoid robot manipulation tasks, this application achieves a higher success rate than existing technologies, particularly demonstrating a substantial advantage on real robot platforms; enhanced cross-task and cross-scenario generalization capabilities: This application maintains leading performance across different task types and difficulty levels, indicating strong transferability and versatility of its learned strategies; improved consistency from simulation to reality: This application effectively reduces Sim-to-Real performance drop, enabling more reliable transfer of strategies learned in simulation to real robots; and improved execution efficiency: While maintaining a high success rate, this application also reduces the average number of steps required to complete the task, improving overall reasoning and control efficiency. Therefore, this application possesses significant technical advantages and engineering application value in high-difficulty robot manipulation tasks, long-sequential tasks, and real-world scenario deployments.
[0092] In one exemplary embodiment, an electronic device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 3As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores underwater dynamic mud level measurement data based on a rotary encoder. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an underwater dynamic mud level measurement method based on a rotary encoder.
[0093] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0094] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.
[0095] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.
[0096] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.
[0097] It should be noted that the engineering cost investment information and data involved in this application (including but not limited to data used for analysis, data stored, data displayed, etc.) are all information and data authorized by the client or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0098] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (Read-Only Memory). Memory includes ROM, magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0099] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0100] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0101] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for autonomous generalization of VLA models, characterized in that, The method includes: The robot acquires visual observation data and natural language task descriptions of its current environment, performs cross-modal feature fusion through a VLA policy model, generates action sequences, and controls the robot to perform tasks. Trajectory data during task execution is collected and input into a memory-reflex reward model. The memory-reflex reward model analyzes the action sequence and outputs token-level dense rewards and structured reflection information. The token-level dense rewards refer to fine-grained reward signals generated by evaluating the importance of each action token in the action sequence and its contribution to task completion. The structured reflection information refers to formatted information generated for failed steps in task execution, including reasons for failure, improvement suggestions, and avoidance strategies. The trajectory data, token-level dense reward, and structured reflection information are stored in a dual memory block; the dual memory block includes short-term memory and long-term memory, wherein the short-term memory is used to store recent interaction data including the trajectory data, token-level dense reward, and structured reflection information, and the long-term memory is used to store high-value task priors filtered and extracted from the short-term memory. An experience-driven generator is used to extract task priors from the long-term memory, generate trajectory data containing sub-target chains, and perform iterative correction based on the execution results to obtain a high-quality trajectory dataset. During the simulation exploration phase, the VLA policy model is updated using a reinforcement learning algorithm based on the token-level dense reward. During the reality fine-tuning phase, the VLA policy model is updated using a supervised fine-tuning algorithm based on the high-quality trajectory dataset. The simulation exploration phase and the reality fine-tuning phase alternate and iterate, forming a closed-loop self-evolutionary update of the VLA policy model.
2. The method according to claim 1, characterized in that, The output token-level dense reward and structured reflection information include: Perform language visual alignment analysis on the trajectory data to identify successful steps, failed steps, and potentially risky steps in the action sequence; Based on the importance of each action token in the action sequence and whether it contributes to the completion of the task, a corresponding positive or negative evaluation is given, and a token-level dense reward is generated. For failed steps, structured reflection information is generated, including the reasons for failure, improvement suggestions, and avoidance strategies. When a certain failure mode occurs repeatedly in multiple tasks, the penalty weight corresponding to that failure mode is increased.
3. The method according to claim 2, characterized in that, The update and transfer mechanism of the dual memory blocks includes: Write the description of the current task, trajectory score, reflection information, and key action points into short-term memory; retrieve similar task experiences in short-term memory using cosine similarity, and filter and transfer high-value experiences to long-term memory; The task priors in the long-term memory include typical action sequences, physical constraints, and semantic behavior rules.
4. The method according to claim 3, characterized in that, The method for obtaining high-quality trajectory datasets includes: The task priors in the long-term memory are decomposed into a sequence of executable sub-goals, and the sub-goals are mapped to robot control primitives; The generated sub-target sequence is executed in the simulation environment, and the execution deviation is monitored. If the execution fails, the action process is automatically corrected by combining the reflection information fed back by the memory reflection reward model, and closed-loop iteration is performed until the trajectory is stable and successful. The successful trajectory is written into a high-quality trajectory dataset.
5. The method according to claim 4, characterized in that, The process of generating trajectory data containing sub-target chains includes: adding random position perturbations, velocity changes, and illumination simulations when generating the trajectory; mapping the generated simulation priors to the real trajectory; and performing spatial perturbation processing by combining inverse kinematics calculations.
6. The method according to claim 5, characterized in that, The update of the VLA strategy model includes: During the simulation exploration phase, the VLA policy model maximizes the total reward through the PPO algorithm. The total reward is a weighted average of the environmental reward and the reward output by the memory reflection reward model. In the real-world fine-tuning phase, the dataset generated by the experience-driven generator is combined with the real-world acquisition trajectory, and the VLA policy model is optimized through supervised fine-tuning.
7. A VLA model autonomous generalization system, characterized in that, The system includes: The VLA strategy model module is configured to receive visual-language multimodal input, fuse visual features and linguistic intent, and output a sequence of robot control actions. The memory-reflection reward model module, connected to the VLA policy model module, is configured to receive task trajectory data and output token-level dense rewards and structured reflection information. The token-level dense reward refers to a fine-grained reward signal generated by evaluating the importance of each action marker in the action sequence and its contribution to task completion on a marker-by-mark basis. This token-level dense reward is transmitted to the VLA policy model module as a reward signal for reinforcement learning updates. The structured reflection information refers to formatted information generated for failed steps in task execution, including reasons for failure, improvement suggestions, and avoidance strategies. This structured reflection information is transmitted to the dual-memory block module for storage. The dual-memory block module, connected to the memory reflection reward model module, includes a short-term memory unit and a long-term memory unit. The short-term memory unit stores recent interaction data and reflection information, while the long-term memory unit stores high-value task priors selected from the short-term memory. The experience-driven generator module connects the dual-memory block module and the VLA policy model module, and is configured to extract task priors from long-term memory units, generate and verify trajectory data, and use them for supervised fine-tuning of the VLA policy model. The VLA policy model module receives token-level dense rewards from the memory-reflection reward model module and updates the model using a reinforcement learning algorithm; it also receives high-quality trajectory datasets from the experience-driven generator module and updates the model using a supervised fine-tuning algorithm.
8. The system according to claim 7, characterized in that, The system includes: The memory reflex reward model module is constructed through a visual language reasoning model to identify fine-grained reasons for task failure and accumulate failure patterns to form a reward prior library. The VLA policy model module receives reward signals from the memory-reflection reward model module for reinforcement learning, and receives trajectory data from the experience-driven generator module for supervised fine-tuning.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 6.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.