Training methods, devices, electronic equipment, and storage media for visual language action models

Abstract

This application provides a training method, apparatus, electronic device, and storage medium for a visual language action model, relating to the field of artificial intelligence technology. The method includes: acquiring an expert demonstration dataset containing robot states and actions; fine-tuning a pre-trained visual language action model using offline reinforcement learning based on the expert demonstration dataset to obtain an initial policy model and an initial value evaluation model; acquiring online interaction data; mixing the online interaction data with the expert demonstration dataset to obtain a hybrid dataset; optimizing the initial value evaluation model based on the hybrid dataset to obtain a calibrated value evaluation model; and fine-tuning the initial policy model using online reinforcement learning based on the calibrated value evaluation model through a value-guided policy self-distillation mechanism to supervise the updating of the initial policy model. The method provided in this application ultimately achieves a reliable performance improvement of the visual language action model in robot control tasks.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, electronic device, and storage medium for a visual language action model. Background Technology

[0002] In the field of robot control, visual language action models are pre-trained through imitation learning, and then fine-tuned using online reinforcement learning to overcome the performance bottleneck of demonstration data. However, when transitioning directly from offline training to online fine-tuning, the pre-trained value evaluation model suffers from severe "initial forgetting" due to drastic changes in data distribution, leading to a collapse in policy performance. At the same time, the large amount of low-quality data generated during the online exploration phase can severely reduce learning efficiency.

[0003] Existing methods, in order to alleviate these problems, typically rely on expensive and scalable manual intervention to screen data or provide security constraints, which greatly limits their application in fully automated, high-sample-efficiency scenarios. Therefore, a solution to address these issues is urgently needed. Summary of the Invention

[0004] This application provides a training method, apparatus, electronic device, and storage medium for a visual language action model to address the deficiencies in the prior art.

[0005] This application provides a method for training a visual language action model, including: Obtain an expert demonstration dataset containing robot states and actions, and fine-tune the pre-trained visual language action model using offline reinforcement learning based on the expert demonstration dataset to obtain an initial policy model and an initial value evaluation model. Acquire online interaction data, mix the online interaction data with the expert demonstration dataset to obtain a hybrid dataset, and optimize the initial value assessment model based on the hybrid dataset to obtain a calibrated value assessment model; wherein, the online interaction data is generated by controlling the robot to perform online interactions in the environment using the initial strategy model; Based on the calibration value assessment model, the initial policy model is fine-tuned online through a value-guided policy self-distillation mechanism to supervise the updating of the initial policy model; wherein, the value-guided policy self-distillation mechanism includes: evaluating the value of historical interaction actions using the calibration value assessment model, comparing it with the value benchmark of new candidate actions generated by the initial policy model, and determining the target action based on the comparison result to update the policy model.

[0006] According to an embodiment of this application, a training method for a visual language action model is provided, wherein the initial policy model is fine-tuned online through a value-guided policy self-distillation mechanism based on the calibrated value assessment model to supervise the updating of the initial policy model, including: For historical states and corresponding historical actions sampled from online interaction history, the value of the historical actions is evaluated using the calibration value assessment model. The initial policy model is used to generate multiple new candidate actions for the historical state, and the calibration value evaluation model is used to calculate the average value of the multiple new candidate actions; The value of the historical actions is compared with the average value to generate a comparison result; Based on the comparison results, the historical action or a new candidate action with a value higher than the average value is determined as the target action.

[0007] According to an embodiment of this application, a training method for a visual language action model is provided. In the online reinforcement learning fine-tuning stage, a first experience pool and a second experience pool are maintained: the first experience pool is used to store expert demonstration datasets and successful trajectory data in subsequent online interactions, and the second experience pool is used to store trajectory data generated by all online interactions. The sampling of historical states and corresponding historical actions from online interaction history includes: The historical states and corresponding historical actions are sampled from the second experience pool.

[0008] According to an embodiment of this application, a training method for a visual language action model is provided, wherein determining the historical action or a new candidate action with a value higher than the average value as the target action based on the comparison result includes: If the comparison result indicates that the value of the historical action is not lower than the average value, the historical action is identified as the target action; If the comparison result indicates that the value of the historical action is lower than the average value, actions with a value higher than the average value are selected from the plurality of new candidate actions to form a high-value action set, and the actions in the high-value action set are identified as the target action.

[0009] According to an embodiment of this application, a training method for a visual language action model is provided. When the target action is an action in the set of high-value actions, the initial policy model is updated by using the actions in the set of high-value actions and the weights determined by the values ​​evaluated based on the calibration value assessment model as supervision targets.

[0010] According to an embodiment of this application, a training method for a visual language action model, wherein the initial policy model is fine-tuned online through a value-guided policy self-distillation mechanism based on the calibrated value assessment model to supervise the updating of the initial policy model, further includes: Each time the policy model is updated, data is sampled from the first experience pool and the second experience pool according to a preset balance ratio to form a training batch.

[0011] According to an embodiment of this application, a training method for a visual language action model is provided, wherein the pre-trained visual language action model is fine-tuned using offline reinforcement learning based on the expert demonstration dataset to obtain an initial policy model and an initial value evaluation model, including: The initial value assessment model is obtained by training the calibration Q-learning algorithm on the expert demonstration dataset; wherein, the calibration Q-learning algorithm adds a conservative regularization term to the standard time-series difference error objective to prevent excessive underestimation of the value of actions outside the distribution of non-expert data. The initial policy model is trained by combining the behavior cloning loss with the policy gradient loss guided by the initial value evaluation model.

[0012] This application also provides a training device for a visual language action model, comprising: The first fine-tuning module is used to acquire an expert demonstration dataset containing robot states and actions, and to perform offline reinforcement learning fine-tuning on the pre-trained visual language action model based on the expert demonstration dataset to obtain an initial policy model and an initial value evaluation model. The second calibration module is used to acquire online interaction data, mix the online interaction data with the expert demonstration dataset to obtain a mixed dataset, and optimize the initial value assessment model based on the mixed dataset to obtain a calibrated value assessment model; wherein, the online interaction data is generated by controlling the robot to perform online interactions in the environment using the initial strategy model; The third fine-tuning module is used to perform online reinforcement learning fine-tuning of the initial policy model based on the calibration value assessment model through a value-guided policy self-distillation mechanism, so as to supervise the updating of the initial policy model; wherein, the value-guided policy self-distillation mechanism includes: using the calibration value assessment model to evaluate the value of historical interaction actions, comparing it with the value benchmark of new candidate actions generated by the initial policy model, and determining the target action to update the policy model based on the comparison result.

[0013] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the training method for the visual language action model as described above.

[0014] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the training method for the visual language action model as described above.

[0015] This application also provides a computer program product, including a computer program that, when executed by a processor, implements a training method for a visual language action model as described above.

[0016] This application provides a training method, apparatus, electronic device, and storage medium for a visual language action model. The method involves acquiring an expert demonstration dataset containing robot states and actions, and fine-tuning a pre-trained visual language action model using offline reinforcement learning based on the expert demonstration dataset to obtain an initial policy model and an initial value evaluation model. It then acquires online interaction data, mixes the online interaction data with the expert demonstration dataset to obtain a hybrid dataset, and optimizes the initial value evaluation model based on the hybrid dataset to obtain a calibrated value evaluation model. The online interaction data is generated by controlling the robot to interact online in an environment using the initial policy model. Based on the calibrated value evaluation model, the initial policy model is fine-tuned using an online reinforcement learning mechanism guided by value to supervise the updating of the initial policy model. The value-guided policy self-distillation mechanism includes: evaluating the value of historical interaction actions using the calibrated value evaluation model and comparing it with the value benchmark of new candidate actions generated by the initial policy model; determining the target action based on the comparison result to update the policy model. Therefore, this embodiment introduces a value pre-calibration stage after offline fine-tuning. It uses online interactive data generated from the initial strategy and offline expert data to optimize the value assessment model, enabling it to adapt to the online strategy state before entering online fine-tuning. This stabilizes the value function distribution and avoids strategy collapse caused by value scale mismatch. Furthermore, through the value-guided strategy self-distillation mechanism based on this calibrated value assessment model, high-value actions can be intelligently selected as learning targets when updating the strategy using online interactive data. This reduces the interference of low-quality exploration data on strategy updates, thereby significantly improving the stability and sample efficiency of strategy optimization without human intervention. Ultimately, this achieves a reliable performance improvement of the visual language action model in robot control tasks. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the training method for the visual language action model provided in this application embodiment.

[0019] Figure 2 This is a schematic diagram of the value-guided strategy self-distillation mechanism provided in the embodiments of this application.

[0020] Figure 3 This is a complete flowchart of the training method for the visual language action model provided in the embodiments of this application.

[0021] Figure 4 This is a schematic diagram of the structure of the training device for the visual language action model provided in the embodiments of this application.

[0022] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the embodiments of this application.

[0024] Before describing the embodiments of this application, the terms and concepts involved in the embodiments of this application will be explained illustratively.

[0025] VGPD: Value-Guided Policy self-Distillation. This is a mechanism proposed in this application during the online fine-tuning phase. It uses a calibrated Q-function as a filter to select exploration and expert data generated by the policy, ensuring that only high-value actions are used for policy updates.

[0026] The following describes, with reference to the accompanying drawings, a training method, apparatus, electronic device, and storage medium for a visual language action model according to an embodiment of this application.

[0027] Currently, Visual-Language-Action (VLA) models are primarily post-trained through Imitation Learning (IL). To overcome the "imitation ceiling" limited by the quality of demonstration data, researchers have introduced Reinforcement Learning (RL) fine-tuning to further improve performance through interaction with the environment. While RL fine-tuning holds promise, it faces significant sample inefficiency in practical applications, particularly in robotic manipulation. The main reasons are as follows: 1. Catastrophic "Initial Forgetting": When a policy transitions from offline pre-training (which typically uses conservative value estimation) to online fine-tuning, the offline Q-function can be misled by the new online data due to the mismatch in Q-value scaling, leading to severe performance crashes.

[0028] 2. Low strategy update efficiency: During the online exploration phase, the strategy generates a large amount of low-quality exploration data. Related methods struggle to effectively filter this data, leading to low learning efficiency.

[0029] 3. Reliance on expensive human intervention: In order to address the above-mentioned instability and ensure safe exploration, related methods (such as ConRFT) usually require expensive and difficult-to-scale human-in-the-loop intervention.

[0030] Based on this, embodiments of this application propose a training method for a visual language action model to solve at least one of the above problems.

[0031] Figure 1 This is a flowchart illustrating the training method for the visual language action model provided in this application embodiment, as shown below. Figure 1 As shown, the method includes the following: Step 100: Obtain an expert demonstration dataset containing robot states and actions, and fine-tune the pre-trained visual language action model using offline reinforcement learning based on the expert demonstration dataset to obtain an initial policy model and an initial value evaluation model.

[0032] Specifically, an expert demonstration dataset refers to a collection of data recorded when a robot is controlled by a human expert or a mature strategy to perform a specific task. It contains state information (such as joint angles, end effector position, and visual images) and corresponding motion commands (such as joint speed and end effector displacement) during the robot's execution process, and is usually stored in the form of trajectory sequences.

[0033] A pre-trained visual-language-action model (VLA) is a neural network model that has been pre-trained on large-scale multimodal data and can directly output robot action sequences based on visual observations and verbal commands. Examples include the VLA model based on the Transformer architecture, which already possesses basic environmental perception and command understanding capabilities. Offline reinforcement learning fine-tuning refers to optimizing this pre-trained model using only a static expert demonstration dataset without real-time interaction with the environment. Through algorithms (such as calibration Q-learning), an initial value assessment model (or Critic network) for predicting the long-term value of actions and an initial policy model (or Actor network) for decision-making are trained simultaneously. The aim is to allow the policy to initially learn expert behavior and establish a conservative value estimate.

[0034] Step 200: Obtain online interaction data, mix the online interaction data with the expert demonstration dataset to obtain a mixed dataset, and optimize the initial value assessment model based on the mixed dataset to obtain a calibrated value assessment model; wherein, the online interaction data is generated by controlling the robot to perform online interactions in the environment using the initial strategy model.

[0035] Specifically, online interactive data refers to the interaction records generated when the initial policy model, fine-tuned as described above, is deployed to a real or simulated environment and the policy autonomously controls the robot to explore. This data includes both successful trajectories and failed attempts. By mixing this fresh online data with existing expert data to form a hybrid dataset, and further optimizing the value assessment model on this dataset, value pre-calibration can be achieved. The core of this step is to use the actual policy performance (success and failure samples) reflected in the online data to correct the distribution bias of the value function during pure offline training. This allows for a more accurate assessment of the state-action pairs generated by the current policy itself, thus smoothly transitioning to the online learning stage and avoiding the "initial forgetting" phenomenon caused by abrupt changes in value scale.

[0036] Step 300: Based on the calibration value assessment model, fine-tune the initial policy model online through a value-guided policy self-distillation mechanism to supervise the updating of the initial policy model; wherein, the value-guided policy self-distillation mechanism includes: evaluating the value of historical interaction actions using the calibration value assessment model, comparing it with the value benchmark of new candidate actions generated by the initial policy model, and determining the target action based on the comparison result to update the policy model.

[0037] Specifically, during the online learning process, for a certain historical interaction action (i.e., an action performed in a past state) stored in the policy history, a pre-calibrated calibration value assessment model is used to evaluate its value score; at the same time, the current policy model (i.e., the initial policy model, which is itself being updated) is used to generate multiple new action options (i.e., new candidate actions) for the same historical state, and these new actions are also scored using the calibration value assessment model, and their average score is calculated as the value benchmark.

[0038] By comparing the value of historical actions with this benchmark: if a historical action has a higher value, it is used as a learning target, essentially a form of self-imitation; if the average value of newly generated actions is higher, high-value actions superior to the benchmark are selected as learning targets. This mechanism acts like a built-in "intelligent filter," enabling the policy to continuously and automatically identify and distill better action patterns from its own historical experience and immediate exploration, thereby achieving efficient and stable online policy optimization and improvement without external manual screening or intervention. The entire method systematically solves the stability and efficiency challenges in VLA model reinforcement fine-tuning through the close integration of three stages: "offline fine-tuning - value pre-calibration - online self-distillation."

[0039] The above describes the steps of the training method for the visual language action model provided in the embodiments of this application. As can be seen from the above description, according to the training method for the visual language action model provided in the embodiments of this application, an expert demonstration dataset containing robot states and actions is obtained, and the pre-trained visual language action model is fine-tuned using offline reinforcement learning based on the expert demonstration dataset to obtain an initial policy model and an initial value evaluation model; online interaction data is obtained, and the online interaction data is mixed with the expert demonstration dataset to obtain a mixed dataset, and the initial value evaluation model is optimized based on the mixed dataset to obtain a calibrated value evaluation model; wherein, the online interaction data is generated by controlling the robot to perform online interactions in the environment using the initial policy model; based on the calibrated value evaluation model, the initial policy model is fine-tuned using online reinforcement learning through a value-guided policy self-distillation mechanism to supervise the updating of the initial policy model; wherein, the value-guided policy self-distillation mechanism includes: evaluating the value of historical interaction actions using the calibrated value evaluation model, comparing it with the value benchmark of new candidate actions generated by the initial policy model, and determining the target action based on the comparison result to update the policy model. Therefore, this embodiment introduces a value pre-calibration stage after offline fine-tuning. It uses online interactive data generated from the initial strategy and offline expert data to optimize the value assessment model, enabling it to adapt to the online strategy state before entering online fine-tuning. This stabilizes the value function distribution and avoids strategy collapse caused by value scale mismatch. Furthermore, through the value-guided strategy self-distillation mechanism based on this calibrated value assessment model, high-value actions can be intelligently selected as learning targets when updating the strategy using online interactive data. This reduces the interference of low-quality exploration data on strategy updates, thereby significantly improving the stability and sample efficiency of strategy optimization without human intervention. Ultimately, this achieves a reliable performance improvement of the visual language action model in robot control tasks.

[0040] Based on the above embodiments, in this embodiment, step 100 involves offline reinforcement learning fine-tuning of the pre-trained visual language action model based on the expert demonstration dataset to obtain an initial policy model and an initial value evaluation model, including: Step 110: Train the calibration Q-learning algorithm on the expert demonstration dataset to obtain the initial value assessment model; wherein, the calibration Q-learning algorithm adds a conservative regularization term to the standard time-series difference error objective to prevent excessive underestimation of the value of actions outside the distribution of non-expert data.

[0041] Step 120: Train the initial policy model by combining the behavior cloning loss and the policy gradient loss guided by the initial value evaluation model.

[0042] Specifically, the calibrated Q-learning algorithm is a value function training method designed specifically for offline reinforcement learning. The standard temporal difference error objective refers to the fundamental Q-learning objective in reinforcement learning, which updates the parameters of the value assessment model by minimizing the temporal difference error between the current value prediction and the value based on the next state target. The conservative regularization term prevents the value assessment model from overly pessimistically underestimating new actions that the policy might generate (i.e., actions outside the expert data distribution) that are not present in the dataset. In practice, this regularization term typically adds a penalty to the loss function, for example, encouraging the value assessment model to estimate the value of policy-generated actions at least as low as a certain lower bound on the value estimates of real actions in the dataset. This ensures that the learned value function is more robust when transitioning from offline to online, avoiding initial policy degradation due to value scale mismatch.

[0043] Furthermore, the behavior cloning loss is a supervised learning loss that directly mimics the state-action pairs in expert demonstration data. By minimizing the difference between the policy model's output action and the expert's recorded action (such as negative log-likelihood), it provides a stable and reliable learning signal for the policy. Meanwhile, the policy gradient loss, guided by the initial value assessment model, introduces the optimization objective of reinforcement learning. Specifically, this loss encourages the policy model to generate actions deemed high-value by the value assessment model, typically achieved by calculating the expected value of the policy output action and taking its negative gradient.

[0044] The training method for the visual language action model provided in this application ensures the policy's faithful imitation of expert behavior and learning stability through behavior cloning, while the value-guided policy gradient provides the policy with the potential to surpass expert data and develop towards better performance, laying a good foundation for subsequent online fine-tuning that combines conservatism and improvement potential.

[0045] Based on the above embodiments, in this embodiment, step 300, based on the calibration value assessment model, performs online reinforcement learning fine-tuning of the initial policy model through a value-guided policy self-distillation mechanism to supervise the updating of the initial policy model, including: Step 310: For the historical states and corresponding historical actions sampled from the online interaction history, evaluate the value of the historical actions using the calibration value assessment model.

[0046] It should be noted that during the online reinforcement learning fine-tuning phase, a first experience pool and a second experience pool are maintained: the first experience pool is used to store the expert demonstration dataset and the successful trajectory data in subsequent online interactions, and the second experience pool is used to store the trajectory data generated by all online interactions.

[0047] Step 310 specifically includes: The historical states and corresponding historical actions are sampled from the second experience pool.

[0048] Specifically, the system maintains two experience pools with different semantics: the first experience pool serves as a high-quality reference library, storing not only the original expert demonstration data but also continuously collecting and storing trajectory data judged as successful in online interactions, which represent reliable behavioral patterns known to the strategy; the second experience pool serves as a complete exploration archive of the strategy, recording all trajectories generated by online interactions without filtering, regardless of success or failure, thus ensuring the diversity and comprehensiveness of the learning data.

[0049] Step 310 involves sampling a historical state and its corresponding historical action from the second experience pool representing the exploration history, serving as the starting point for evaluation and improvement. Subsequently, the calibrated value assessment model, optimized during the value pre-calibration phase, is used to score this historical action, yielding a quantified value (i.e., the Q-value), representing the model's estimate of the expected long-term return for this historical action in the historical state.

[0050] Step 320: Generate multiple new candidate actions for the historical state using the initial policy model, and calculate the average value of the multiple new candidate actions using the calibration value evaluation model.

[0051] Step 330: Compare the value of the historical actions with the average value to generate a comparison result.

[0052] Specifically, the initial policy model to be updated (i.e., the policy itself) is used to reason about the same sampled historical state, generating multiple new candidate actions. These actions represent new decisions the policy might make in the same situation based on its current knowledge. Similarly, a calibration value assessment model is used to evaluate the value of each of these new candidate actions, and their average value is calculated. This average value is used as a dynamic benchmark to measure the policy's current average performance level in that state. Step 330 compares the value of historical actions with this dynamic benchmark to generate a comparison result. This comparison essentially determines whether the actions in the historical record are better, equivalent, or worse than the policy's current average performance level.

[0053] Step 340: Based on the comparison results, determine the historical action or a new candidate action with a value higher than the average value as the target action.

[0054] Step 340 specifically includes: Step 341: If the comparison result indicates that the value of the historical action is not lower than the average value, the historical action is identified as the target action.

[0055] Step 342: If the comparison result indicates that the value of the historical action is lower than the average value, select actions with a value higher than the average value from the plurality of new candidate actions to form a high-value action set, and determine the actions in the high-value action set as the target action.

[0056] Specifically, if the value of a historical action is not lower than the average value, it indicates that the historical action is still competitive, or even superior, under the current policy standard. Therefore, it is directly identified as a learning target, which is equivalent to the policy consolidating and imitating its past successful experiences. If the value of a historical action is lower than the average value, it indicates that after learning, the policy's current average generation ability is better than that historical record. In this case, the system does not simply discard the sample, but performs an intelligent screening, selecting actions with values ​​higher than the average value from the newly generated candidate actions, forming a high-value action set. The actions in this set represent the better solutions that the policy can think of for this state under the current understanding, solutions that are superior to its own average level. Finally, the actions in this high-value action set are identified as the target.

[0057] Furthermore, if the target action is an action in the set of high-value actions, the initial policy model is updated using the weights determined by the actions in the set of high-value actions and the values ​​evaluated based on the calibration value assessment model as the supervision targets.

[0058] Specifically, when it is determined that the target for policy update comes from a set of high-value actions, this embodiment employs a refined weighted distillation mechanism to guide the update of the policy model.

[0059] Specifically, the system assigns importance weights to each action based on the specific value score (i.e., Q-value) calculated by the calibration value assessment model. Typically, this is achieved through a softmax operation: actions with higher value scores have greater weight in the supervision objective. Finally, the weighted distribution formed by this set of high-value actions and their corresponding value weights is used as the supervision objective, and the initial policy model is updated by minimizing the distillation loss. The distillation loss measures the difference between the current policy model's output action distribution and the aforementioned weighted target distribution. By minimizing this loss through backpropagation, the policy model is trained to make its probability distribution of output actions in corresponding historical states as close as possible to the target distribution defined by high-value actions and their value weights.

[0060] Furthermore, step 300 also includes: Each time the policy model is updated, data is sampled from the first experience pool and the second experience pool according to a preset balance ratio to form a training batch.

[0061] Specifically, during each policy model update, the system synchronously samples data from a second experience pool containing successful expert demonstrations and a first experience pool containing the policy's own exploration experience, at a preset equilibrium ratio (e.g., 1:1 or a ratio dynamically adjusted according to the task), constructing a training batch. This equilibrium ratio aims to balance the signals from imitation learning and reinforcement learning: on the one hand, the expert state-action pairs sampled from the second experience pool provide high-quality, high-reward behavioral demonstrations for the policy, guiding it to imitate the expert's successful patterns; on the other hand, the policy's own interaction data sampled from the first experience pool provides diverse feedback during the exploration process, especially since failure experiences are crucial for learning robust policies. Mixing these two types of data in a single batch allows the policy model to simultaneously receive strong supervisory signals from expert demonstrations and reinforcement learning gradients from environmental interactions in a single update.

[0062] The following embodiment provides a detailed description of the value-guided strategy self-distillation mechanism provided in this application.

[0063] Figure 2 This is a schematic diagram of the value-guided strategy self-distillation mechanism provided in the embodiments of this application, as shown below. Figure 2 The diagram illustrates a Value-Driven Policy Distillation (VGPD) framework, which optimizes policy decisions by combining the policy's own experience with expert demonstrations. The policy buffer on the left of the diagram stores the states s generated by policy interactions. p This includes the corresponding failure transition data and success transition data, while the expert buffer stores successful transition pairs from the online demo. e ,a e Strategy π(s) receives state s p And generate multiple candidate actions a1, a2, ..., a n Ultimately, action a is selected. p Execution. The VGPD module utilizes the experience of two buffers and passes it through the value function Q. means By evaluating and guiding actions, strategies can learn from successful demonstrations by experts while also adjusting and optimizing based on one's own failures, thus achieving a more stable and efficient learning process.

[0064] The training method for the visual language action model provided in this application embodiment realizes a dynamic, value-driven self-distillation: the strategy not only imitates its past good behavior, but also learns better solutions from its own imaginary space when it realizes the inadequacy of past behavior, thereby achieving continuous, autonomous and efficient strategy iterative optimization.

[0065] Figure 3 This is a complete flowchart of the training method for the visual language action model provided in the embodiments of this application. The following is a combination of... Figure 3 The training method for the visual language action model provided in the embodiments of this application will be described in detail.

[0066] like Figure 3 The diagram illustrates the complete process of the three-stage fine-tuning framework. First, in the Offline Stage, the Vision-Language-Action Model simultaneously trains the critic network (updating the action value Q(s,a) through critic updates) and the policy network (updating by behavioral clones to mimic expert actions) using offline demo data. Next, in the Pre-Calibration Stage, the policy begins interacting in the environment, storing its resulting success transitions and failure transitions along with the offline demo data in a mixed buffer. This buffer is then used to further optimize the critic network (critic updates) and the policy (expert behavioral clone updates) to calibrate the value function distribution and smoothly transition to the online environment. Finally, in the Online Stage, the framework maintains two independent buffers: an Expert Buffer storing offline demos and successful online trajectories, and a Policy Buffer storing all online interaction trajectories of the policy. In each training step, the system draws data from two buffers to form batches through equal sampling. The policy network generates actions based on the state and performs temporal difference updates (TD updates) using the Q-grad (value gradient) provided by the calibrated critic network. At the same time, the core value-guided policy self-distillation (VGPD) mechanism starts to operate. It uses the critic network to evaluate and select high-value actions generated by the policy itself, and updates the policy through distillation to make the policy imitate these high-quality actions, thereby achieving efficient and stable online reinforcement learning fine-tuning without human intervention.

[0067] The training method for the visual language action model provided in this application introduces a value pre-calibration stage, effectively integrating offline expert data and preliminary online interaction data. This pre-corrects the distribution bias of the value function, fundamentally eliminating the "initial forgetting" phenomenon commonly encountered during fine-tuning from offline to online, ensuring a smooth start and high stability in the training process. Based on this, the designed value-guided strategy self-distillation mechanism utilizes the calibrated value function as an intelligent filter to dynamically evaluate and select high-value actions from historical experience and immediate exploration. This allows the strategy to continuously imitate and surpass its own performance, significantly improving the sample efficiency and strategy optimization quality of online learning without any human intervention. Ultimately, this method enables the trained model to achieve a significant leap in success rate in various robot operation tasks (e.g., near 100% success rate in simulation and real environments), while also drastically reducing the number of execution steps required to complete the task. It exhibits more decisive and efficient action sequences, providing a stable, efficient, and universal solution for achieving high-performance, fully autonomous robot reinforcement learning fine-tuning.

[0068] The training apparatus for the visual language action model provided in the embodiments of this application will be described below. The training apparatus for the visual language action model described below and the training method for the visual language action model described above can be referred to in correspondence.

[0069] Figure 4 This is a schematic diagram of the structure of the training device for the visual language action model provided in the embodiments of this application, as shown below. Figure 4 As shown in the embodiment of this application, the training device for the visual language action model includes: The first fine-tuning module 401 is used to acquire an expert demonstration dataset containing robot states and actions, and to perform offline reinforcement learning fine-tuning on the pre-trained visual language action model based on the expert demonstration dataset to obtain an initial policy model and an initial value evaluation model. The second calibration module 402 is used to acquire online interaction data, mix the online interaction data with the expert demonstration dataset to obtain a mixed dataset, and optimize the initial value assessment model based on the mixed dataset to obtain a calibrated value assessment model; wherein, the online interaction data is generated by controlling the robot to perform online interactions in the environment using the initial strategy model; The third fine-tuning module 403 is used to perform online reinforcement learning fine-tuning of the initial policy model based on the calibration value assessment model through a value-guided policy self-distillation mechanism, so as to supervise the updating of the initial policy model; wherein, the value-guided policy self-distillation mechanism includes: using the calibration value assessment model to evaluate the value of historical interaction actions, comparing it with the value benchmark of new candidate actions generated by the initial policy model, and determining the target action based on the comparison result to update the policy model.

[0070] The training device for the visual language action model provided in this application embodiment acquires an expert demonstration dataset containing robot states and actions, and performs offline reinforcement learning fine-tuning on the pre-trained visual language action model based on the expert demonstration dataset to obtain an initial policy model and an initial value evaluation model; acquires online interaction data, mixes the online interaction data with the expert demonstration dataset to obtain a hybrid dataset, and optimizes the initial value evaluation model based on the hybrid dataset to obtain a calibrated value evaluation model; wherein, the online interaction data is generated by controlling the robot to perform online interactions in the environment using the initial policy model; based on the calibrated value evaluation model, the initial policy model is fine-tuned online through a value-guided policy self-distillation mechanism to supervise the updating of the initial policy model; wherein, the value-guided policy self-distillation mechanism includes: evaluating the value of historical interaction actions using the calibrated value evaluation model, comparing it with the value benchmark of new candidate actions generated by the initial policy model, and determining the target action based on the comparison result to update the policy model. Therefore, this embodiment introduces a value pre-calibration stage after offline fine-tuning. It uses online interactive data generated from the initial strategy and offline expert data to optimize the value assessment model, enabling it to adapt to the online strategy state before entering online fine-tuning. This stabilizes the value function distribution and avoids strategy collapse caused by value scale mismatch. Furthermore, through the value-guided strategy self-distillation mechanism based on this calibrated value assessment model, high-value actions can be intelligently selected as learning targets when updating the strategy using online interactive data. This reduces the interference of low-quality exploration data on strategy updates, thereby significantly improving the stability and sample efficiency of strategy optimization without human intervention. Ultimately, this achieves a reliable performance improvement of the visual language action model in robot control tasks.

[0071] Based on the above embodiments, in this embodiment, the third fine-tuning module 403 is specifically used for: For historical states and corresponding historical actions sampled from online interaction history, the value of the historical actions is evaluated using the calibration value assessment model. The initial policy model is used to generate multiple new candidate actions for the historical state, and the calibration value evaluation model is used to calculate the average value of the multiple new candidate actions; The value of the historical actions is compared with the average value to generate a comparison result; Based on the comparison results, the historical action or a new candidate action with a value higher than the average value is determined as the target action.

[0072] Based on the above embodiments, in this embodiment, during the online reinforcement learning fine-tuning stage, a first experience pool and a second experience pool are maintained: the first experience pool is used to store expert demonstration datasets and successful trajectory data in subsequent online interactions, and the second experience pool is used to store trajectory data generated by all online interactions; The device further includes a sampling module, specifically used for: The historical states and corresponding historical actions are sampled from the second experience pool.

[0073] Based on the above embodiments, in this embodiment, the device further includes a determining module, specifically used for: If the comparison result indicates that the value of the historical action is not lower than the average value, the historical action is identified as the target action; If the comparison result indicates that the value of the historical action is lower than the average value, actions with a value higher than the average value are selected from the plurality of new candidate actions to form a high-value action set, and the actions in the high-value action set are identified as the target action.

[0074] Based on the above embodiments, in this embodiment, when the target action is an action in the set of high-value actions, the weights determined by the actions in the set of high-value actions and the values ​​evaluated based on the calibration value assessment model are used as the supervision targets, and the initial policy model is updated through distillation loss.

[0075] Based on the above embodiments, in this embodiment, the third fine-tuning module 403 is further used for: Each time the policy model is updated, data is sampled from the first experience pool and the second experience pool according to a preset balance ratio to form a training batch.

[0076] Based on the above embodiments, in this embodiment, the first fine-tuning module 401 is specifically used for: The initial value assessment model is obtained by training the calibration Q-learning algorithm on the expert demonstration dataset; wherein, the calibration Q-learning algorithm adds a conservative regularization term to the standard time-series difference error objective to prevent excessive underestimation of the value of actions outside the distribution of non-expert data. The initial policy model is trained by combining the behavior cloning loss with the policy gradient loss guided by the initial value evaluation model.

[0077] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device can be a robot or other electronic device, and may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 550. The processor 510, communication interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions from the memory 530 to execute a training method for the visual language action model, including: Obtain an expert demonstration dataset containing robot states and actions, and fine-tune the pre-trained visual language action model using offline reinforcement learning based on the expert demonstration dataset to obtain an initial policy model and an initial value evaluation model. Acquire online interaction data, mix the online interaction data with the expert demonstration dataset to obtain a hybrid dataset, and optimize the initial value assessment model based on the hybrid dataset to obtain a calibrated value assessment model; wherein, the online interaction data is generated by controlling the robot to perform online interactions in the environment using the initial strategy model; Based on the calibration value assessment model, the initial policy model is fine-tuned online through a value-guided policy self-distillation mechanism to supervise the updating of the initial policy model; wherein, the value-guided policy self-distillation mechanism includes: evaluating the value of historical interaction actions using the calibration value assessment model, comparing it with the value benchmark of new candidate actions generated by the initial policy model, and determining the target action based on the comparison result to update the policy model.

[0078] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application embodiment, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in at least one embodiment of this application embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0079] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the training method for the visual language action model provided by the above methods, including: Obtain an expert demonstration dataset containing robot states and actions, and fine-tune the pre-trained visual language action model using offline reinforcement learning based on the expert demonstration dataset to obtain an initial policy model and an initial value evaluation model. Acquire online interaction data, mix the online interaction data with the expert demonstration dataset to obtain a hybrid dataset, and optimize the initial value assessment model based on the hybrid dataset to obtain a calibrated value assessment model; wherein, the online interaction data is generated by controlling the robot to perform online interactions in the environment using the initial strategy model; Based on the calibration value assessment model, the initial policy model is fine-tuned online through a value-guided policy self-distillation mechanism to supervise the updating of the initial policy model; wherein, the value-guided policy self-distillation mechanism includes: evaluating the value of historical interaction actions using the calibration value assessment model, comparing it with the value benchmark of new candidate actions generated by the initial policy model, and determining the target action based on the comparison result to update the policy model.

[0080] In another aspect, embodiments of this application also provide a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a training method for the visual language action model provided by the methods described above, including: Obtain an expert demonstration dataset containing robot states and actions, and fine-tune the pre-trained visual language action model using offline reinforcement learning based on the expert demonstration dataset to obtain an initial policy model and an initial value evaluation model. Acquire online interaction data, mix the online interaction data with the expert demonstration dataset to obtain a hybrid dataset, and optimize the initial value assessment model based on the hybrid dataset to obtain a calibrated value assessment model; wherein, the online interaction data is generated by controlling the robot to perform online interactions in the environment using the initial strategy model; Based on the calibration value assessment model, the initial policy model is fine-tuned online through a value-guided policy self-distillation mechanism to supervise the updating of the initial policy model; wherein, the value-guided policy self-distillation mechanism includes: evaluating the value of historical interaction actions using the calibration value assessment model, comparing it with the value benchmark of new candidate actions generated by the initial policy model, and determining the target action based on the comparison result to update the policy model.

[0081] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0082] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of this application, and are not intended to limit them; although the embodiments of this application have been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A training method for a visual language action model, characterized in that, include: Obtain an expert demonstration dataset containing robot states and actions, and fine-tune the pre-trained visual language action model using offline reinforcement learning based on the expert demonstration dataset to obtain an initial policy model and an initial value evaluation model. Acquire online interaction data, mix the online interaction data with the expert demonstration dataset to obtain a hybrid dataset, and optimize the initial value assessment model based on the hybrid dataset to obtain a calibrated value assessment model; wherein, the online interaction data is generated by controlling the robot to perform online interactions in the environment using the initial strategy model; Based on the calibration value assessment model, the initial policy model is fine-tuned online through a value-guided policy self-distillation mechanism to supervise the updating of the initial policy model; wherein, the value-guided policy self-distillation mechanism includes: evaluating the value of historical interaction actions using the calibration value assessment model, comparing it with the value benchmark of new candidate actions generated by the initial policy model, and determining the target action based on the comparison result to update the policy model.

2. The training method for the visual language action model according to claim 1, characterized in that, The step of fine-tuning the initial policy model online through a value-guided policy self-distillation mechanism based on the calibration value assessment model, in order to supervise the updating of the initial policy model, includes: For historical states and corresponding historical actions sampled from online interaction history, the value of the historical actions is evaluated using the calibration value assessment model. The initial policy model is used to generate multiple new candidate actions for the historical state, and the calibration value evaluation model is used to calculate the average value of the multiple new candidate actions; The value of the historical actions is compared with the average value to generate a comparison result; Based on the comparison results, the historical action or a new candidate action with a value higher than the average value is determined as the target action.

3. The training method for the visual language action model according to claim 2, characterized in that, During the online reinforcement learning fine-tuning phase, a first experience pool and a second experience pool are maintained: the first experience pool is used to store the expert demonstration dataset and successful trajectory data in subsequent online interactions, and the second experience pool is used to store trajectory data generated by all online interactions. The sampling of historical states and corresponding historical actions from online interaction history includes: The historical states and corresponding historical actions are sampled from the second experience pool.

4. The training method for the visual language action model according to claim 3, characterized in that, The step of determining the historical action or a new candidate action with a value higher than the average value as the target action based on the comparison result includes: If the comparison result indicates that the value of the historical action is not lower than the average value, the historical action is identified as the target action; If the comparison result indicates that the value of the historical action is lower than the average value, actions with a value higher than the average value are selected from the plurality of new candidate actions to form a high-value action set, and the actions in the high-value action set are identified as the target action.

5. The training method for the visual language action model according to claim 4, characterized in that, When the target action is an action in the set of high-value actions, the initial policy model is updated using the weights determined by the actions in the set of high-value actions and the values ​​evaluated based on the calibration value assessment model as the supervision targets, and distillation loss is used to update the initial policy model.

6. The training method for the visual language action model according to claim 3, characterized in that, The step of fine-tuning the initial policy model online through a value-guided policy self-distillation mechanism based on the calibration value assessment model to supervise the updating of the initial policy model further includes: Each time the policy model is updated, data is sampled from the first experience pool and the second experience pool according to a preset balance ratio to form a training batch.

7. The training method for the visual language action model according to claim 1, characterized in that, The offline reinforcement learning fine-tuning of the pre-trained visual language action model based on the expert demonstration dataset to obtain an initial policy model and an initial value evaluation model includes: The initial value assessment model is obtained by training the calibration Q-learning algorithm on the expert demonstration dataset; wherein, the calibration Q-learning algorithm adds a conservative regularization term to the standard time-series difference error objective to prevent excessive underestimation of the value of actions outside the distribution of non-expert data. The initial policy model is trained by combining the behavior cloning loss with the policy gradient loss guided by the initial value evaluation model.

8. A training device for a visual language action model, characterized in that, include: The first fine-tuning module is used to acquire an expert demonstration dataset containing robot states and actions, and to perform offline reinforcement learning fine-tuning on the pre-trained visual language action model based on the expert demonstration dataset to obtain an initial policy model and an initial value evaluation model. The second calibration module is used to acquire online interaction data, mix the online interaction data with the expert demonstration dataset to obtain a mixed dataset, and optimize the initial value assessment model based on the mixed dataset to obtain a calibrated value assessment model; wherein, the online interaction data is generated by controlling the robot to perform online interactions in the environment using the initial strategy model; The third fine-tuning module is used to perform online reinforcement learning fine-tuning of the initial policy model based on the calibration value assessment model through a value-guided policy self-distillation mechanism, so as to supervise the updating of the initial policy model; wherein, the value-guided policy self-distillation mechanism includes: using the calibration value assessment model to evaluate the value of historical interaction actions, comparing it with the value benchmark of new candidate actions generated by the initial policy model, and determining the target action to update the policy model based on the comparison result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the training method for the visual language action model as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method for the visual language action model as described in any one of claims 1 to 7.

Images