VLA action optimization method and device based on action regret estimation, and electronic equipment

Abstract

The application discloses a VLA action optimization method and device based on action regret estimation, and electronic equipment, comprising: obtaining the original action output by the VLA in the current control period as an anchor point action; constructing a safe action neighborhood based on the anchor point action, generating counterfactual candidate actions; scoring the state transition consistency of the anchor point action and each candidate action to obtain a scoring set; calculating the action regret value according to the difference between the optimal candidate score and the anchor point score, and generating an action refinement gating coefficient; when the regret value meets the preset refinement condition, the anchor point action and the optimal candidate action are fused based on the gating coefficient to obtain a refined action and execute it. The method takes the original action as a reliable anchor point, estimates the action optimization space through counterfactual candidate action comparison, realizes action refinement in the reasoning phase without fine-tuning the VLA model, thereby improving the stability, safety and task completion rate of the agent action execution.

Description

Technical Field

[0001] This application belongs to the field of embodied intelligence, specifically relating to a VLA action optimization method, apparatus, and electronic device based on action regret estimation. Background Technology

[0002] With the development of embodied intelligence and agent learning technologies, the Vision-Language-Action Model (VLA model) is gradually being applied to industrial agent control scenarios. The VLA model can generate agent actions based on visual observation information and natural language task instructions, enabling agents to have strong task understanding and action generation capabilities in tasks such as handling, grasping, assembling, and sorting.

[0003] In practical deployments, VLA models are typically used as pre-trained master control models, with their parameters generally kept frozen to avoid the security risks and deployment costs associated with online updates. However, the operating environment of industrial intelligent agents is complex and variable. Factors such as changes in lighting, camera viewpoint shifts, fluctuations in target material positions, gripper wear, and the introduction of new objects can all cause the actions output by the VLA model to no longer be optimal for the current scenario. Even if the actions output by the VLA model are still executable, there may be issues such as end-effector position deviations, poor gripping angles, inappropriate gripper opening and closing amounts, or unstable transport paths, thus affecting the success rate of task execution.

[0004] In existing technologies, one type of method improves the adaptability of the VLA model by retraining or fine-tuning it. However, this approach requires additional data and incurs high computational costs, and may compromise the original model's general capabilities. Other methods improve action output by directly optimizing within the action space. However, without constraints on the original VLA model's actions, the corrected actions may deviate from the original strategy, leading to control instability or security risks.

[0005] Therefore, there is an urgent need for a method that can use the original actions of the VLA model as credible anchor points, construct counterfactual candidate actions in the neighborhood of safe actions, and estimate the action regret value through self-supervised action scoring. This would enable the system to determine whether there is room for optimization in the current VLA model actions, and generate refined actions during inference under the premise of satisfying safety constraints, thereby improving the action stability, safety and task completion rate of the agent in complex industrial tasks. Summary of the Invention

[0006] To address the problems existing in the prior art, this application provides a VLA action optimization method, apparatus, and electronic device based on action regret estimation. The method uses the original action output by the VLA model as a reliable anchor action, generates counterfactual candidate actions within the safe action neighborhood of the anchor action, and scores the anchor action and counterfactual candidate actions using a self-supervised action evaluation function. Furthermore, it calculates the action regret value based on the difference between the optimal counterfactual candidate action score and the anchor action score, determines whether action refinement is required and the degree of refinement based on the action regret value, and finally generates refined actions during inference through gating fusion. This achieves online optimization of agent actions without fine-tuning the VLA main model, improving the stability, safety, and task execution success rate of industrial intelligent agents in complex environments.

[0007] According to a first aspect of the embodiments of this application, a VLA action optimization method based on action regret estimation is provided, comprising: Obtain the original action output by the VLA model in the current control cycle, and determine the original action as the anchor action for action refinement during inference; A safe action neighborhood is constructed based on the anchor point action, and multiple counterfactual candidate actions are generated within the safe action neighborhood; The state transition consistency scores are calculated for the anchor point action and each counterfactual candidate action to obtain the anchor point action score and the candidate action score set. Based on the difference between the optimal score in the candidate action score set and the anchor point action score, the action regret value is calculated, and the action refinement gating coefficient is generated based on the action regret value. When the action regret value meets the preset refinement conditions, the anchor action and the optimal counterfactual candidate action are gated and fused based on the action refinement gating coefficient to obtain the inference-time refined action, and the agent is controlled to execute the inference-time refined action.

[0008] According to a second aspect of the application embodiments, a VLA motion optimization apparatus based on motion regret estimation is provided, comprising: Anchor Action Module is used to obtain the original action output by the VLA model in the current control cycle and determine the original action as the anchor action for action refinement during inference. The action constraint module is used to construct a safe action neighborhood based on the anchor point action, and generate multiple counterfactual candidate actions within the safe action neighborhood; The self-supervised evaluation module is used to perform state transition consistency scoring on the anchor action and each counterfactual candidate action, respectively, to obtain the anchor action score and the candidate action score set. The regret estimation module is used to calculate the action regret value based on the difference between the best score in the candidate action score set and the anchor point action score, and to generate the action refinement gating coefficient based on the action regret value. The action refinement execution module is used to perform gated fusion of the anchor point action and the optimal counterfactual candidate action based on the action refinement gating coefficient when the action regret value meets the preset refinement conditions, so as to obtain the inference-time refined action, and control the agent to execute the inference-time refined action.

[0009] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0010] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, having stored thereon computer instructions that, when executed by a processor, implement the method described in the first aspect.

[0011] Compared with the prior art, the embodiments of the present invention have at least the following beneficial effects: (1) Using the original action output by VLA as the anchor action avoids the control instability problem caused by directly replacing VLA actions. (2) Using the safe action neighborhood to generate counterfactual candidate actions reduces the risk of exceeding the speed, pose or gripper control boundary during action optimization. (3) Using the self-supervised action evaluation function to score the anchor action and counterfactual candidate actions, the effect of comparing the merits of actions without the need for truth action labels is obtained, improving the usability of action evaluation in the inference stage. (4) Using the difference between the score of the optimal counterfactual candidate action and the score of the anchor action to calculate the action regret value increases the quantification of the optimizable space of the original VLA action. (5) Using the action regret value to generate action refinement gating coefficients, and gating fusion of the anchor action and the optimal counterfactual candidate action enhances the smooth output of refined actions during inference, improving the agent's action execution accuracy while maintaining the stability of the original VLA policy. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating a VLA motion optimization method based on motion regret estimation, according to an exemplary embodiment.

[0013] Figure 2 This is a block diagram illustrating a VLA motion optimization device based on motion regret estimation according to an exemplary embodiment.

[0014] Figure 3This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0015] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0016] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0017] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0018] Figure 1 This is a flowchart illustrating a VLA (Vehicle Action Optimization) method based on action regret estimation, according to an exemplary embodiment. In this example, the invention can be applied to a controller module of an agent equipped with a VLA model in an industrial setting. The agent can be any agent equipped with a VLA model, such as an industrial robotic arm or a logistics forklift. The agent can be used to perform tasks such as assembly, gripping, sorting, or handling. Taking the material handling task of an industrial agent as an example, it is not limited to this; it can also be used for other agent action tasks driven by a large language model, such as robotic arm gripping tasks or industrial vehicle navigation tasks. Reference Figure 1 The present invention provides a VLA action optimization method based on action regret estimation, which may include the following steps: S1: Obtain the original action output by the VLA model in the current control cycle, and determine the original action as the anchor action for action refinement during inference; In this example, the industrial agent is used to move target materials from conveyor belts, workbenches, or racks to designated bins, pallets, or assembly areas. During the inference phase, the agent receives language instructions for the current task and outputs the initial action for the current control cycle based on the VLA model. This embodiment does not directly classify the initial action as an anomalous action, but rather uses it as an anchor action for subsequent action regret assessment and action optimization.

[0019] S1 may include the following sub-steps: S11: Obtain the observed state of the agent in the current control cycle and the language instructions for the current task; Specifically, in the current control period (t), the agent acquires the current observation state. , denoted as:

[0020] in, This represents the currently observed image captured by the industrial camera. This indicates the current pose of the agent's end effector. This indicates the current open / closed state of the gripper. The current observation state is used to characterize the agent's external visual environment and its own motion state at the current moment.

[0021] At the same time, the intelligent agent receives the language instructions for the current task, denoted as... In this case, the language instructions can be:

[0022] S12: Input the observation state and the language command into VLA to obtain the original action output by VLA in the current control cycle; Specifically, the current observation state and the language instructions Input the deployed VLA model to obtain the raw actions for the current control cycle. :

[0023] in, This represents a visual language action model.

[0024] In the scenario of material handling by an industrial intelligent agent, the original action can be represented as:

[0025] in,( ) represent the displacement increments of the agent's end effector in three-dimensional space, This indicates the attitude adjustment amount of the end effector. This indicates the control amount for opening and closing the grippers.

[0026] S13: Determine the original action as the anchor action for action refinement during inference, and output the observation state, language command and anchor action; Specifically, in order to perform inference-time action optimization without negating the validity of the original VLA action, this embodiment determines the original action output by the VLA as the anchor point action. :

[0027] The anchor point action serves as the baseline action for subsequent construction of the safe action neighborhood, generation of counterfactual candidate actions, and calculation of action regret value.

[0028] Therefore, the final output is the anchor point motion sample. :

[0029] The sample is input to step S2 to construct a safe action neighborhood around the anchor action and generate a set of counterfactual candidate actions.

[0030] In this embodiment, while keeping the VLA parameters frozen and without retraining the main model, the original VLA actions are used as a reliable action benchmark, providing a stable reference for subsequent action optimization based on regret self-supervision, and avoiding control instability caused by directly replacing VLA actions.

[0031] S2: Construct a safe action neighborhood based on the anchor point action, and generate multiple counterfactual candidate actions within the safe action neighborhood; this may include the following sub-steps: S21: Receive the observation status, language command and anchor point action output in step S1, and determine the safe action neighborhood corresponding to the anchor point action based on the agent action constraints and the end effector motion range; Specifically, to ensure that the subsequently generated counterfactual candidate actions do not deviate too far from the original VLA actions, nor exceed the safe execution range of the industrial intelligent agent, this embodiment constructs a safe action neighborhood based on the anchor point actions. The safe action neighborhood is denoted as... Represent the candidate action as The security field can be defined as:

[0032] in, Indicates candidate actions, This represents the neighborhood radius around which perturbations are allowed during actions around the anchor point. This represents the safe action space of the intelligent agent.

[0033] Meanwhile, the intelligent agent's safe action space It can be represented as:

[0034] in, and These represent the lower and upper limits of each dimension of the action, respectively. This indicates the linear velocity at the end of the action. Indicates the maximum permissible end linear velocity. This indicates the rate of change of posture corresponding to the action. This indicates the maximum permissible rate of attitude change.

[0035] S22: Perturbate and sample the anchor point action within the neighborhood of the safe action to generate multiple counterfactual candidate actions; Specifically, a perturbation is added within the neighborhood of the security action to generate multiple counterfactual candidate actions. The k-th counterfactual candidate action can be represented as:

[0036] in, This represents the disturbance term applied to the anchor point movement, the disturbance term satisfying... This ensures that counterfactual candidate actions are located within the safe neighborhood of the anchor action.

[0037] Further, form a set of counterfactual candidate actions. Represented as:

[0038] in This is represented as the Kth counterfactual candidate action generated in the previous control cycle t. Through the above method, this embodiment does not search for actions without constraints throughout the entire action space, but rather constructs counterfactual candidate actions within the safe neighborhood corresponding to the original VLA action. This ensures that subsequent action regret value calculations can both discover whether there are better actions near the anchor action and guarantee that the generated candidate actions meet the safe execution constraints in the industrial intelligent agent's material handling task. S3: Perform state transition consistency scoring on the anchor action and each counterfactual candidate action to obtain the anchor action score and candidate action score set; this step may include the following sub-steps: S31: The anchor point action and the counterfactual candidate action constitute a set of actions to be evaluated. For each action in the set, the forward dynamic evaluation network is used to predict the next state feature corresponding to each action. Specifically, the anchor point action and the set of counterfactual candidate actions are combined to form the action set U to be evaluated:

[0039] Right now:

[0040] The set of actions to be evaluated is used to calculate the state transition consistency score of anchor actions and each counterfactual candidate action through a self-supervised action evaluation function.

[0041] Based on the current observation state, task language instructions, and the action to be evaluated, a self-supervised action evaluation function is used to predict the state transition consistency that the action may cause. The current observation state... Task language instructions And for each action input in the set of actions to be evaluated, a forward dynamics evaluation network is used to predict the next state features. :

[0042] in, , This represents the forward dynamic evaluation network. Represents the actions within the action set U to be evaluated. The corresponding features for predicting the next state.

[0043] In the industrial intelligent agent material handling scenario of this case, the predicted next state features include the position of the target material relative to the gripper, the contact state between the gripper and the material, the position of the material relative to the target hopper, and the pose change trend of the intelligent agent's end effector.

[0044] S32: The inverse dynamics evaluation network is used to reconstruct the action based on the current state characteristics and the predicted next state characteristics to obtain the reconstructed action; Specifically, the current observation status Task language instructions and predicting the features of the next state Input the inverse dynamics evaluation network to reconstruct the action corresponding to the state transition, and obtain the reconstructed action. :

[0045] in, This indicates the inverse dynamics evaluation network. This represents the action obtained by reconstructing the network through inverse dynamics evaluation.

[0046] S33: Calculate the state transition consistency loss based on the corresponding action in the set of actions to be evaluated and the reconstructed action; Specifically, based on the corresponding actions in the set of actions to be evaluated With the reconstruction action The difference between them is used to calculate the action cycle consistency loss. :

[0047] The action cycle consistency loss is used to measure the action cycle consistency loss. Whether the state transition remains consistent after forward state prediction and backward action reconstruction. If the predicted state transition corresponding to a certain action can be stably reconstructed by the backward dynamics network, it indicates that the action has high state transition consistency under the current observation state and language instruction conditions.

[0048] S34: Obtain the self-supervised action score for each action based on the state transition consistency loss, and output the anchor action score and the candidate action score set. Specifically, the features of the next state will be predicted. With task language instructions Input the task progress evaluation network to obtain actions. Corresponding predictive task progress gain :

[0049] in, This represents the task progress evaluation network. Indicates action The predicted progress gain for the current transport task.

[0050] In material handling scenarios, if an action brings the target material closer to the gripper, makes the contact between the gripper and the material more stable, or brings the material closer to the target bin, then its predicted task progress gain is higher.

[0051] Based on the action cycle consistency loss (Loss) and the predicted task progress gain Construct a self-supervised action evaluation function:

[0052] in, Indicates action Self-monitored action scoring, The weighting coefficient representing the cycle consistency loss is set to 0.1 in this case.

[0053] Therefore, through the self-supervised action evaluation function Anchor point motion scores were calculated separately. and counterfactual candidate action scoring From the scores of each counterfactual candidate action, a set of candidate action scores is obtained. :

[0054] The final output includes anchor point motion scores and a set of candidate motion scores. , used to calculate action regret value and generate action refinement gating coefficient.

[0055]

[0056] In this embodiment, instead of directly determining whether the anchor action output by the VLA model is incorrect, the anchor action is compared with the counterfactual candidate actions in its safe neighborhood under the same self-supervised evaluation function, thereby providing a quantifiable basis for the subsequent calculation of action regret value.

[0057] S4: Calculate the action regret value based on the difference between the optimal score in the candidate action score set and the anchor point action score, and generate an action refinement gating coefficient based on the action regret value; this step may include the following sub-steps: S41: Determine the highest-scoring optimal counterfactual candidate action and its optimal score from the candidate action score set; Determine the optimal score from the candidate action score set. :

[0058] And determine the corresponding optimal counterfactual candidate action. .

[0059] S42: Calculate the difference between the optimal score and the anchor point action score to obtain the action regret value; Specifically, a motion regret value is calculated based on the difference between the optimal score and the anchor point motion score. :

[0060] The action regret value is used to characterize whether there are candidate actions with higher self-supervised scores relative to the original VLA action within the safe neighborhood of the anchor action.

[0061] like This indicates that the score of the anchor point action is not lower than that of all counterfactual candidate actions, suggesting that the original VLA action already has a superior performance in the current security neighborhood and does not require action refinement. like This indicates that there are counterfactual candidate actions with scores higher than the anchor action, suggesting that there is still room for optimization in the current VLA original action.

[0062] S43: Generate an action refinement gating coefficient based on the relationship between the action regret value and the preset refinement threshold; Specifically, a refining threshold is set. The value is 0.08, which is the action regret value compared to the preset refinement threshold. A comparison is made. When the action regret value exceeds a preset refinement threshold, i.e. When the condition for refining the action is met, it is determined that the action is satisfied.

[0063] To avoid abrupt changes in actions between the anchor action and the optimal counterfactual candidate action, this embodiment generates action refinement gating coefficients based on the action regret value. :

[0064] in, , This represents the Sigmoid function. This represents the gating slope coefficient.

[0065] when hour, A value close to 0 indicates that the system tends to maintain the anchor point action; when And when the regret value of the action is large, The gradual increase indicates that the system allows for the incorporation of more information about the optimal counterfactual candidate actions.

[0066] If satisfied This indicates that there are better safety candidate actions near the anchor point action. The system generates an action refinement gating coefficient greater than zero and passes the optimal counterfactual candidate action to step S5.

[0067] Therefore, the final output of step S4 is:

[0068] in, This indicates the regret value of the action. Indicates the motion refinement gating coefficient. This represents the optimal counterfactual candidate action. This indicates the anchor point action.

[0069] In this embodiment, instead of directly judging whether a VLA action is erroneous based on an absolute score, the optimization space of the original VLA action is quantified by comparing the score difference between the anchor action and the counterfactual candidate action in the safe neighborhood. This enables action optimization judgment during inference based on regret self-supervision.

[0070] S5: When the action regret value meets the preset refinement condition, based on the action refinement gating coefficient, the anchor action and the optimal counterfactual candidate action are gated and fused to obtain the inference-time refined action, and the agent is controlled to execute the inference-time refined action; this step may include the following sub-steps: S51: Determine whether the action regret value meets the preset refinement conditions; when the action regret value does not meet the preset refinement conditions, use the anchor point action as the refinement action during reasoning; when When this occurs, it indicates that the anchor action has no significant room for optimization within the current safe action neighborhood. In this case, the system does not adjust the original VLA action and uses the anchor action as the refined action during inference. S52: When the action regret value meets the preset refinement condition, the action refinement gating coefficient is used to perform gating fusion on the anchor point action and the optimal counterfactual candidate action to obtain the refined action during inference. when When the optimal counterfactual candidate action has a higher self-supervised action score than the anchor action, the system performs gating fusion on the anchor action and the optimal counterfactual candidate action based on the action refinement gating coefficient to obtain the refined action during inference. :

[0071] when When the size is smaller, the refined actions during reasoning are closer to the anchor actions; when When the value is large, the refined action during reasoning incorporates more information about the optimal counterfactual candidate action.

[0072] In industrial intelligent agent material handling scenarios, if the anchor point action Represented as:

[0073] Optimal counterfactual candidate action Represented as:

[0074] Then refine the actions during reasoning. It can be represented as:

[0075] To further ensure that the refined actions during inference satisfy the agent's execution safety constraints, this embodiment can perform a secure projection again after gating fusion:

[0076] in, This represents the safe action space of the intelligent agent.

[0077] If the refined actions during inference already satisfy the safe action space constraints, then the final action is output. : .

[0078] S53: Send the inference-time refinement action to the agent controller and control the agent to execute the inference-time refinement action; Specifically, the final output action is sent to the agent controller:

[0079] in, Indicates intelligent agent controller, This represents the control commands used to drive the actuators of an intelligent agent.

[0080] In this case, during the task of moving blue material to the right-side bin, if the anchor point action output by the VLA causes the gripper to deviate slightly from the material center, and there is a counterfactual candidate action that brings the gripper closer to the material center without violating safety constraints, then the action regret value exceeds the preset refinement threshold. In this case, the system generates refined actions during inference through gating fusion, allowing the agent's end effector to make minor corrections while maintaining the original VLA action direction, thereby improving the success rate of material gripping and handling.

[0081] In this embodiment, instead of directly replacing the original actions output by VLA, the original VLA actions are used as a reliable anchor point. Based on the action regret value, it is determined whether to introduce the optimal counterfactual candidate actions. Then, through gating fusion and secure projection, refined actions are generated during inference, thereby optimizing actions in the inference stage while maintaining the stability of the VLA main model.

[0082] Figure 2 This is a block diagram illustrating a VLA motion optimization device based on motion regret estimation according to an exemplary embodiment. (Reference) Figure 2 The device includes: Anchor point action module 1 is used to obtain the original action output by the VLA model in the current control cycle, and to determine the original action as the anchor point action for action refinement during inference. Action constraint module 2 is used to construct a safe action neighborhood based on the anchor point action, and generate multiple counterfactual candidate actions within the safe action neighborhood; The self-supervised evaluation module 3 is used to perform state transition consistency scoring on the anchor point action and each counterfactual candidate action respectively, to obtain the anchor point action score and the candidate action score set. The regret estimation module 4 is used to calculate the action regret value based on the difference between the best score in the candidate action score set and the anchor point action score, and to generate the action refinement gating coefficient based on the action regret value. The action refinement execution module 5 is used to perform gated fusion of the anchor point action and the optimal counterfactual candidate action based on the action refinement gating coefficient when the action regret value meets the preset refinement conditions, so as to obtain the inference-time refined action, and control the agent to execute the inference-time refined action.

[0083] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement a VLA action optimization method based on action regret estimation as described above. Figure 3 The diagram shown is a hardware structure diagram of any device with data processing capabilities, where a VLA motion optimization device based on motion regret estimation provided by an embodiment of the present invention is located. (Except for...) Figure 3 In addition to the processor and memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0084] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the VLA action optimization method based on action regret estimation as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0085] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0086] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A VLA action optimization method based on action regret estimation, characterized in that, include: Obtain the original action output by the VLA model in the current control cycle, and determine the original action as the anchor action for action refinement during inference; A safe action neighborhood is constructed based on the anchor point action, and multiple counterfactual candidate actions are generated within the safe action neighborhood; The state transition consistency scores are calculated for the anchor point action and each counterfactual candidate action to obtain the anchor point action score and the candidate action score set. Based on the difference between the optimal score in the candidate action score set and the anchor point action score, the action regret value is calculated, and the action refinement gating coefficient is generated based on the action regret value. When the action regret value meets the preset refinement conditions, the anchor action and the optimal counterfactual candidate action are gated and fused based on the action refinement gating coefficient to obtain the inference-time refined action, so as to control the agent to execute the inference-time refined action.

2. The VLA action optimization method based on action regret estimation according to claim 1, characterized in that, Obtain the raw action output by the VLA model in the current control cycle, and determine the raw action as the anchor action for action refinement during inference, including: Acquire the agent's observed state and the language instructions for the current task during the current control cycle; The observed state and the language command are input into the VLA model to obtain the original action output by the VLA model in the current control cycle; The original action is identified as the anchor action for action refinement during reasoning.

3. The VLA action optimization method based on action regret estimation according to claim 1, characterized in that, A safe action neighborhood is constructed based on the anchor point action, and multiple counterfactual candidate actions are generated within the safe action neighborhood, including: Receive observation status, language commands and anchor point actions, and determine the safe action neighborhood corresponding to the anchor point action based on the agent's action constraints and the end effector's motion range; The anchor point action is perturbed and sampled within the neighborhood of the safe action to generate multiple counterfactual candidate actions.

4. The VLA action optimization method based on action regret estimation according to claim 1, characterized in that, State transition consistency scores are applied to the anchor action and each counterfactual candidate action to obtain anchor action scores and candidate action score sets, including: The anchor point action and the counterfactual candidate action constitute a set of actions to be evaluated. For each action in the set, the forward dynamic evaluation network is used to predict the next state feature corresponding to each action. The inverse dynamics evaluation network is used to reconstruct the action based on the characteristics of the current state and the characteristics of the next state, thus obtaining the reconstructed action; Calculate the state transition consistency loss based on the corresponding action in the set of actions to be evaluated and the reconstructed action; The self-supervised action score for each action is obtained based on the state transition consistency loss, and the anchor action score and candidate action score set are output.

5. The VLA action optimization method based on action regret estimation according to claim 1, characterized in that, Based on the difference between the optimal score in the candidate action score set and the anchor point action score, an action regret value is calculated, and an action refinement gating coefficient is generated based on the action regret value, including: From the set of candidate action scores, determine the highest-scoring optimal counterfactual candidate action and its optimal score; The difference between the optimal score and the anchor point action score is calculated to obtain the action regret value; Based on the relationship between the action regret value and the preset refinement threshold, an action refinement gating coefficient is generated.

6. The VLA action optimization method based on action regret estimation according to claim 1, characterized in that, When the action regret value meets the preset refinement conditions, based on the action refinement gating coefficient, the anchor action and the optimal counterfactual candidate action are gated and fused to obtain the inference-time refined action, thereby controlling the agent to execute the inference-time refined action, including: Determine whether the action regret value meets the preset refinement conditions. If the action regret value does not meet the preset refinement conditions, the anchor point action is used as the refined action during reasoning. When the action regret value meets the preset refinement condition, the action refinement gating coefficient is used to perform gating fusion on the anchor point action and the optimal counterfactual candidate action to obtain the inference-time refined action. The inference-time refinement action is sent to the agent's controller to control the agent to execute the inference-time refinement action.

7. A VLA motion optimization device based on motion regret estimation, characterized in that, include: Anchor Action Module is used to obtain the original action output by the VLA model in the current control cycle and determine the original action as the anchor action for action refinement during inference. The action constraint module is used to construct a safe action neighborhood based on the anchor point action, and generate multiple counterfactual candidate actions within the safe action neighborhood; The self-supervised evaluation module is used to perform state transition consistency scoring on the anchor action and each counterfactual candidate action, respectively, to obtain the anchor action score and the candidate action score set. The regret estimation module is used to calculate the action regret value based on the difference between the best score in the candidate action score set and the anchor point action score, and to generate the action refinement gating coefficient based on the action regret value. The action refinement execution module is used to perform gated fusion of the anchor point action and the optimal counterfactual candidate action based on the action refinement gating coefficient when the action regret value meets the preset refinement conditions, so as to obtain the inference-time refined action, and control the agent to execute the inference-time refined action.

8. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction performs the method as described in any one of claims 1-6.

Images