VLA online self-correction method and device based on visual question and answer verification, and electronic equipment

Abstract

The application discloses a VLA online self-correction method and device based on visual question and answer verification, and electronic equipment, comprising: receiving a language instruction of a current task, and analyzing and generating an ordered sub-target and a task progress graph; triggering abnormal detection according to the deviation degree of the current observation feature and the expected feature of the current sub-target; performing zero-sample visual question and answer verification on the current observation frame; performing time sequence accumulation determination on the abnormal confidence of the continuous control period; and based on the abnormal confidence, gating and fusing the original action and the compensation action of the VLA model to obtain a correction action; and using the state transition data generated by executing the correction action and the task completion progress score given by the VLA model to learn and update the residual strategy network. The method can realize abnormal identification, visual verification, action correction and online self-learning of the long time sequence operation process of the intelligent agent without fine-tuning the VLA model, and improves the stability and correction ability of the VLA when executing complex tasks.

Description

Technical Field

[0001] This application belongs to the field of embodied intelligence, specifically relating to a VLA online error correction method, device, and electronic device based on visual question answering verification. Background Technology

[0002] In recent years, with the development of embodied intelligence and intelligent robot learning technologies, Vision-Language-Action Models (VLA models) have been increasingly applied to industrial intelligent agent control scenarios. VLA models can receive natural language task instructions and visual observation information, and directly generate agent actions to perform complex tasks such as assembly, grasping, sorting, and handling. Compared to traditional intelligent agent control methods based on manual rules or fixed programs, VLA models have stronger task generalization capabilities and greater convenience in human-computer interaction.

[0003] However, in long-term operational tasks of industrial intelligent agents, VLA models still suffer from problems such as uncontrollable execution processes, difficulty in timely detection of abnormal states, and the easy accumulation of erroneous actions. Specifically, VLA models typically generate actions based on current observations in an end-to-end manner, but they lack an explicit representation of task execution progress, making it difficult to determine whether a sub-objective has been completed by the current action, and also making it difficult to detect abnormal situations such as grasping failure, target slippage, positioning deviation, occlusion interference, or placement errors in a timely manner. Once a previous action deviates, subsequent actions may continue to be executed based on the erroneous state, leading to task failure.

[0004] In existing technologies, some methods constrain agent tasks by manually setting task flows or state machines. However, these methods rely on manual rule design and are difficult to adapt to diverse natural language instructions and changes in open scenarios. Some methods monitor the agent's execution state by training anomaly detection models or success discrimination models, but these usually require a large amount of labeled data and have limited generalization ability to new tasks, objects, or environments. Other methods improve the execution robustness of VLA models by fine-tuning or retraining them, but this approach has high training costs, long deployment cycles, and may compromise the general capabilities already possessed by the original VLA model.

[0005] Therefore, there is an urgent need for a method that can combine the semantic understanding capabilities of visual language models to monitor task progress, verify visual questions and answers, determine abnormal timing, and correct residual actions during the VLA execution process. This would enable the agent to promptly detect execution anomalies and perform online self-correction without fine-tuning the VLA main model, thereby improving the stability, reliability, and task completion rate in complex industrial operations. Summary of the Invention

[0006] To address the problems existing in the prior art, this application provides a VLA online self-correction method, device, and electronic device based on visual question-answering verification. The method uses a VLA model to parse language instructions into a task progress graph containing ordered sub-goals and visual verification conditions. During VLA model execution, feature deviation detection and zero-shot visual question-answering verification are combined to generate anomaly confidence scores, and temporal anomaly integrals are used to determine whether to enter error correction mode. In error correction mode, while keeping the VLA model parameters frozen, a residual policy network generates compensation actions. These compensation actions are then gated and fused based on the anomaly confidence scores to obtain corrective actions for controlling the agent. Furthermore, the state transition data generated from executing the corrective actions, task completion progress scores, and anomaly confidence scores are used to perform online reinforcement learning updates on the residual policy network, thereby achieving anomaly detection, semantic verification, action correction, and online self-learning during agent execution.

[0007] According to a first aspect of the embodiments of this application, a VLA online self-correction method based on visual question answering verification is provided, comprising: Receive the language instructions for the current task of the intelligent agent, use the VLA model to perform offline parsing of the language instructions, and generate a task progress map including ordered sub-goals and the corresponding visual verification conditions of each sub-goal; During the process of the VLA model generating and executing actions step by step, the current sub-target to be verified is determined based on the task progress map, and anomaly trigger detection is performed by using the deviation between the current observed features of the VLA model and the expected feature distribution of the current sub-target to be verified, and anomaly trigger signal is output. In response to the abnormal trigger signal, the visualization verification conditions corresponding to the current observation frame and the current sub-target to be verified are input into the VLA model for zero-sample visual question answering verification, and the abnormal confidence level with continuous values ​​is output based on the verification result. The anomaly confidence is exponentially decayed and weighted within the sliding time window to obtain the time-series anomaly integral, and the time-series anomaly integral is used to determine whether to enter the error correction mode. In response to the error correction mode instruction, the residual policy network of the VLA model generates a compensation action based on the current observation features and intermediate layer features, and generates a gating coefficient based on the anomaly confidence to obtain a correction action, which drives the agent to execute the correction action and stores the corresponding state transition data in the experience replay buffer. The VLA model is updated online using the state transition data in the experience replay buffer as training samples and the anomaly confidence level as the sampling priority.

[0008] According to a second aspect of the application embodiments, an apparatus for a VLA online self-correction method based on visual question-answering verification is provided, comprising: The task progress generation module is used to receive the language instructions of the agent's current task, use the VLA model to parse the language instructions offline, and generate a task progress map including ordered sub-objectives and the corresponding visual verification conditions of each sub-objective. An anomaly trigger detection module is used to determine the current sub-target to be verified based on the task progress map during the step-by-step generation and execution of actions by the VLA model, and to perform anomaly trigger detection by utilizing the deviation between the current observed features of the VLA model and the expected feature distribution of the current sub-target to be verified, and output anomaly trigger signal. The visual question-answering verification module is used to respond to the abnormal triggering signal, input the visual verification conditions corresponding to the current observation frame and the current sub-target to be verified into the VLA model to perform zero-sample visual question-answering verification, and output the abnormal confidence level with continuous values ​​based on the verification result. The timing anomaly determination module is used to perform exponential decay weighted accumulation of the anomaly confidence within a sliding time window to obtain the timing anomaly integral, and to determine whether to enter the error correction mode based on the timing anomaly integral. The residual action error correction module is used to respond to the indication of the error correction mode. The residual policy network of the VLA model generates compensation actions based on the current observation features and intermediate layer features, and generates gating coefficients based on the anomaly confidence to obtain correction actions. It drives the agent to execute correction actions and stores the corresponding state transition data in the experience replay buffer. The reinforcement learning update module is used to perform online reinforcement learning updates on the VLA model using the state transition data in the experience replay buffer as training samples and the anomaly confidence as the sampling priority.

[0009] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0010] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0011] Compared with the prior art, the embodiments of the present invention have at least the following beneficial effects: First, this invention uses a VLA model to parse language instructions into a task progress graph containing ordered sub-goals and visual verification conditions. During VLA model execution, it combines feature deviation detection, zero-shot visual question answering verification, and temporal anomaly integration to determine abnormal states, improving the monitorability and anomaly detection accuracy of long-term tasks. Simultaneously, while maintaining the VLA model parameters frozen, this invention utilizes a residual policy network to generate compensating actions and performs gated fusion based on anomaly confidence, enabling online action error correction without disrupting the original VLA model strategy. Finally, this invention uses the VLA model task completion progress score as a reward signal and anomaly confidence as a sampling priority to perform online reinforcement learning updates on the residual policy network, thereby continuously improving the agent's stability, error correction capability, and task completion rate in complex tasks. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating an online self-correcting method for VLA based on visual question answering verification, according to an exemplary embodiment.

[0013] Figure 2 This is a block diagram illustrating a VLA online self-correcting device based on visual question-and-answer verification according to an exemplary embodiment.

[0014] Figure 3 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0015] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0016] Figure 1 This is a flowchart illustrating an online self-correction method for VLA based on visual question-answering verification, according to an exemplary embodiment. In this example, the invention can be applied to a controller module of an agent equipped with a VLA model in an industrial setting. The agent can be any agent equipped with a VLA model, such as an industrial robotic arm or a logistics forklift. The agent can perform tasks such as assembly, grasping, sorting, or handling. At the current time step, the agent acquires an RGB image of the work area through a visual sensor and, combined with the current task description, physical constraint information, causal relationship information, and historical action sequences, constructs a unified input sequence for subsequent VLA model optimization. The agent's subsequent actions are optimized using a self-attention fusion method based on twin structure masks. The online self-correction method for VLA based on visual question-answering verification may include the following sub-steps: S1: Receive the language instructions for the agent's current task, and use the VLA model to perform offline parsing of the language instructions to generate a task progress map including ordered sub-goals and corresponding visual verification conditions for each sub-goal; this step includes the following sub-steps: S11: Receive the language instruction for the agent's current task, input the language instruction into the pre-trained VLA model, and parse it to obtain multiple sub-objectives required to complete the current task; Specifically, this embodiment takes an industrial robotic arm as an example. The intelligent agent receives the language instructions for the current task, denoted as:

[0017] in, Language instructions indicating the current task The first in the language instruction Each word element, The number of lexical units representing language instructions.

[0018] In an industrial sorting scenario, the language commands for the current task can be:

[0019] In an industrial assembly scenario, the language commands for the current task can be:

[0020] Subsequently, the language instructions are input into the pre-trained Visual Language Model (VLA) for offline task parsing, resulting in multiple sub-objectives required to complete the current task:

[0021] in, This represents the task parsing function for a pre-trained visual language model. This represents a set consisting of multiple sub-objectives. The set of sub-objectives can be represented as:

[0022] in, Indicates the first Sub-goals Indicates the number of sub-targets.

[0023] Taking "grabbing the red workpiece on the conveyor belt and placing it into the left-side material box" as an example, the VLA model can parse the following sub-objectives:

[0024] S12: Based on the execution order relationship between the multiple sub-targets, generate an ordered sequence of sub-targets, and generate a visual verification condition for each sub-target in the sub-target sequence to determine whether the sub-target has been completed; Specifically, based on the execution order of the multiple sub-targets, the multiple sub-targets are sorted to generate an ordered sequence of sub-targets:

[0025] The sorting relationships can be determined by the VLA model based on action dependencies, spatial relationships, and industrial operation process constraints in the language instructions. For example, gripping a workpiece must occur before releasing it, and positioning a target must occur before the robotic arm approaches the target.

[0026] After obtaining the sub-target sequence, for each sub-target in the sequence, a visual verification condition is generated to determine whether the sub-target has been completed. For the first sub-target... Sub-goals The corresponding visual verification conditions are represented as follows:

[0027] in, Indicates the first The set of visual verification conditions corresponding to each sub-target. Indicates the first The first visual verification condition corresponding to each sub-objective This indicates the number of visual verification conditions corresponding to this sub-goal.

[0028] In this case study's VLA industrial scenario, for the sub-objectives:

[0029] Sub-targets For example, its visual verification conditions may include, but are not limited to:

[0030] S13: Associate the sub-target sequence with the visual verification conditions corresponding to each sub-target to obtain a task progress map; Specifically, the ordered sub-target sequence is associated with the corresponding visual verification conditions for each sub-target to obtain a task progress graph:

[0031] in, This represents a task progress chart. Indicates the first One sub-goal to be executed. This represents the visual verification conditions corresponding to the sub-objective. This indicates the execution order of the sub-target within the task.

[0032] In this embodiment, before the agent actually performs its actions, the abstract language instructions are converted into a task progress map with a clear execution order and visually verifiable conditions using the VLA model. This enables the subsequent VLA execution process to perform sub-target tracking, anomaly detection, and error correction control based on the task progress map, thereby avoiding the problems of unexplainable task progress and difficulty in timely detection of abnormal states caused by relying solely on end-to-end action generation.

[0033] S2: During the step-by-step generation and execution of actions by the VLA model, the current sub-target to be verified is determined based on the task progress map, and anomaly trigger detection is performed using the deviation between the current observed features of the VLA model and the expected feature distribution of the current sub-target to be verified, and an anomaly trigger signal is output; this step includes the following sub-steps: S21: Obtain the task progress map and the execution status of the VLA model in the current control cycle, determine the completed sub-objectives based on the execution status, and determine the current sub-objectives to be verified from the task progress map; Specifically, in the current control period t, based on the set of sub-objectives already completed by the agent... Determine the current sub-objective to be verified:

[0034] in, This indicates the sub-target currently awaiting verification.

[0035] In this case, the sub-target to be verified for the crawling task is:

[0036] S22: Obtain the current observation frame of the agent and extract the visual features output by the VLA model visual encoder for the current observation frame; Specifically, the agent acquires the current observation frame. And input the VLA visual encoder Extracting visual features :

[0037] S23: Obtain the expected feature distribution corresponding to the current sub-target to be verified, and calculate the deviation of the visual feature from the expected feature distribution; Specifically, for each sub-target in the task progress chart, the system pre-calculates the distribution of its visual features under normal completion status.

[0038] For the current sub-target to be verified The corresponding expected feature distribution is expressed as:

[0039] in This represents the mean value of the features under the normal completion state of the current sub-target to be verified. This represents the corresponding covariance matrix. The expected feature distribution can be obtained from statistical analysis of historical successful execution samples or constructed from offline teaching data.

[0040] Subsequently, the deviation of the current observed feature from the expected feature distribution is calculated, using Mahalanobis distance as the deviation calculation index:

[0041] in, This indicates the feature deviation of the current control cycle. When the agent is in a normal execution state, the visual features... It should be close to the desired feature distribution, therefore Keep it small; when the agent fails to grasp, loses the target, shifts position, or experiences environmental disturbances, i.e., when it cannot acquire visual features that meet the conditions, This will deviate from the normal distribution, thus leading to Significantly increased.

[0042] S24: When the deviation does not exceed the first threshold, the current execution state is determined to be normal and the VLA model action generation process continues; when the deviation exceeds the first threshold, an abnormal trigger signal and the current observation frame are output.

[0043] Specifically, This indicates a preset first threshold. The deviation is compared with the first threshold, and when... The current execution status is determined to be normal; when At that time, it was determined that there was an abnormal risk.

[0044] when When an abnormal trigger signal is output. And simultaneously output the current observation frame As input for the subsequent VLA model visual verification module; when When an abnormal trigger signal is output. The system continues to execute the VLA action generation process.

[0045] For example, in a transport task, the current sub-goal to be verified is:

[0046] Under normal circumstances, When an abnormal trigger signal is output. The system operates normally, but when a workpiece slips, the features extracted by the vision encoder deviate significantly from the normal grasping state. At this time... The system immediately outputs an abnormality trigger signal and sends the current observation frame to step S3 for further visual question-and-answer verification.

[0047] In this embodiment, the expected feature distribution corresponding to the current sub-objective to be verified in the task progress graph is used to supervise the VLA execution process online. Low-overhead anomaly detection is achieved without frequent calls to the VLA model, and subsequent visual verification is triggered only when an anomaly risk is detected, thereby improving the overall system operating efficiency and anomaly detection capability.

[0048] S3: In response to the anomaly trigger signal, input the visualization verification conditions corresponding to the current observation frame and the current sub-target to be verified into the VLA model for zero-shot visual question answering verification, and output the anomaly confidence level with continuously taking values ​​based on the verification result; this step includes the following sub-steps: S31: Receive the abnormal trigger signal and the current observation frame, and in response to the abnormal trigger signal, read the visual verification conditions corresponding to the current sub-target to be verified from the task progress graph; Specifically, the abnormal trigger signal output in step S2 is represented as follows:

[0049] Among them, when When the time is right, it indicates that the current execution status is normal and there is no need to perform the visual question-and-answer verification in step S3.

[0050] when When this occurs, it indicates that there is an abnormal risk in the current control cycle, and it is necessary to call the VLA model for visual verification; the system receives the current observation frame output in step S2. And based on the current sub-target to be verified Read the set of visual verification conditions corresponding to the sub-objective from the task progress map generated in step S1. :

[0051] in, This represents the set of visual verification conditions corresponding to the current sub-target to be verified. This represents the k-th visual verification condition, where K represents the number of visual verification conditions.

[0052] In this case, when the industrial intelligent agent performs the grasping action, the current sub-objective to be verified is:

[0053] The corresponding visual verification conditions can be:

[0054] S32: Construct the current observation frame and the visualization verification conditions into a visual question-and-answer input, and input it into the VLA model for zero-sample verification; Specifically, the current observation frame and each visualization verification condition are constructed as visual question-and-answer inputs. For the Kth visualization verification condition, a visual question-and-answer prompt is constructed:

[0055] in, This represents the constructor for visual question-and-answer prompts.

[0056] In this case, regarding the conditions:

[0057] The following visual question-and-answer prompts can be constructed:

[0058] Input the current observation frame and visual question-and-answer prompts into the VLA model:

[0059] in, This represents the VLA model's judgment result for the k-th visual validation condition. This indicates the confidence level corresponding to the judgment result.

[0060] S33: Based on the zero-sample verification results, determine whether each of the visualization verification conditions is met, and calculate and output the outlier confidence of continuously taking values ​​based on the satisfaction of each visualization verification condition. Specifically, when the VLA model determines that the corresponding visualization verification conditions are met, The value is close to 1; when the VLA model determines that the corresponding visualization verification condition is not met, Close to 0.

[0061] To obtain the anomaly confidence score, the satisfaction confidence scores of each visual verification condition are converted into anomaly contribution values. :

[0062] The abnormal contribution values ​​are weighted and fused to obtain the abnormal confidence level for the current control cycle. :

[0063] in, This indicates the outlier confidence level for consecutive values. Let represent the weight of the k-th visual verification condition, satisfying:

[0064] The weight The importance of the visualization verification conditions can be preset according to their importance, or they can be automatically generated by the VLA model based on the importance of the current sub-objective.

[0065] In this case, for the sub-objective "grippers picking up red workpieces", if the confidence levels corresponding to the three visualization verification conditions are respectively:

[0066] Human-defined weights:

[0067] The anomaly confidence level is:

[0068] This result indicates that the current observation frame is highly likely not to meet the visualization completion conditions for the sub-target "grippers picking up the red workpiece".

[0069] In this embodiment, the VLA model is called for visual question-and-answer verification only when an anomaly risk is triggered in step S2. This avoids the computational overhead caused by frequently calling the VLA model in each control cycle, and also utilizes the zero-sample visual understanding capability of the VLA model to perform semantic-level confirmation of the current execution state of the industrial intelligent agent, thereby improving the accuracy and interpretability of anomaly judgment.

[0070] S4: Within the sliding time window, the anomaly confidence level is exponentially decayed and weighted to obtain the time-series anomaly integral, and based on the time-series anomaly integral, it is determined whether to enter error correction mode; this step includes the following sub-steps: S41: Receive the abnormal confidence level, write the abnormal confidence level of the current control cycle into the sliding time window, and perform exponentially decaying weighted summation on the abnormal confidence levels of multiple control cycles within the sliding time window to obtain the time-series abnormality integral. Specifically, step S3 outputs the anomaly confidence level in the current control period t. in, The larger the value, the more likely the agent's execution state is to be abnormal within the current control cycle.

[0071] The system writes the anomaly confidence level of the current control cycle into a sliding time window of length W:

[0072] in, W represents the anomaly confidence window up to the current control period (t), and W represents the sliding window length.

[0073] To ensure that anomaly confidence scores closer to the current time have a higher impact, this embodiment performs an exponentially decaying weighted summation on the anomaly confidence scores within the sliding time window to obtain the time-series anomaly integral. :

[0074] in, This represents the exponential decay coefficient. When i=0, it corresponds to the abnormal confidence level of the current control cycle. The weight of is the highest; as i increases, the weight of the abnormal confidence corresponding to the earlier control period gradually decreases.

[0075] S42: Determine whether the time-series anomaly integral exceeds a preset judgment threshold. When the time-series anomaly integral exceeds the preset judgment threshold, output an indication to enter the error correction mode. When the time-series anomaly integral does not exceed the preset judgment threshold, continue the VLA model action generation process.

[0076] Specifically, the integral of the time-series anomaly is compared with a preset judgment threshold. Compare, when satisfied When this occurs, it indicates that the agent has exhibited a high risk of anomalies over multiple consecutive control cycles, and the system outputs an indication to enter error correction mode.

[0077] When satisfied If the error is detected, it indicates that the anomaly may be a temporary disturbance or misjudgment. The system will not enter error correction mode and will continue to execute the VLA action generation process.

[0078] In this embodiment, single-frame anomaly judgment is transformed into temporal anomaly integral judgment within a continuous time window, so that the triggering of the error correction mode considers both the anomaly intensity and the anomaly persistence, thereby improving the stability and noise resistance of online error correction for industrial intelligent agents.

[0079] S5: In response to the indication of the error correction mode, the residual policy network of the VLA model generates a compensation action based on the current observation features and intermediate layer features, and generates a gating coefficient based on the anomaly confidence to obtain a correction action, driving the agent to execute the correction action, and storing the corresponding state transition data in the experience replay buffer; this step includes the following sub-steps: S51: Receive the instruction to enter error correction mode and the anomaly confidence level; Specifically, the error correction mode indication output in step S4 is represented as follows:

[0080] Among them, when When this occurs, it indicates that the system has entered error correction mode; when When this occurs, it indicates that the system continues to execute the VLA original action generation process.

[0081] S52: While keeping the VLA model parameters frozen, obtain the current observation features and the intermediate layer features of the VLA model, and input the current observation and the intermediate layer features of the VLA model into the residual policy network to generate compensation actions; Specifically, upon entering error correction mode, the current observation is acquired. and VLA model intermediate layer features ,in, It can be the intermediate layer features of the VLA visual encoder, the cross-modal fusion layer features, or the hidden state features before the action decoder.

[0082] Meanwhile, the VLA model outputs the original action in the current control cycle. .

[0083] For industrial intelligent agents, the original motion may include the pose increment of the robotic arm end effector, the gripper opening / closing control quantity, or the joint control quantity. In this case, the output original motion... It can be represented as:

[0084] in,( This represents the position increment of the end effector in three-dimensional space. Indicates the attitude increment. This indicates the opening and closing amount of the grippers.

[0085] The current observations and VLA intermediate layer features are input into the residual policy network to obtain the compensation action. :

[0086] in, Represents the residual policy network. This represents the network parameters of the residual policy.

[0087] S53: Generate a gating coefficient based on the anomaly confidence level, wherein the gating coefficient is initially zero and increases as the anomaly confidence level increases; add the original action output by the VLA model to the compensation action weighted by the gating coefficient to obtain the correction action; Specifically, based on the anomaly confidence level output in step S3 Generate gating coefficients :

[0088] in, This represents the Sigmoid function, where k represents the gating slope parameter. This indicates the threshold for gating activation.

[0089] To ensure that the system does not interfere with the original VLA action when no anomaly is detected, the gating coefficient satisfies:

[0090] And it increases with increasing anomaly confidence:

[0091] Subsequently, the original action output by the VLA model is added to the compensated action after being weighted by the gating coefficients to obtain the corrected action. :

[0092] S54: Control the intelligent agent to execute the correction action, and store the state transition data generated by executing the correction action into the experience playback buffer; Specifically, the corrective action is sent to the agent controller to drive the robotic arm or end effector to perform the error correction action for the current control cycle.

[0093] After the control agent performs the corrective action, the system records the state transition data corresponding to that control cycle. :

[0094] in, This indicates the agent's state before executing the corrective action. This indicates the agent's state after the corrective action is performed. This indicates the anomaly confidence level corresponding to the state transition.

[0095] Subsequently, the state transition data is stored in the experience replay buffer. :

[0096] The experience replay buffer is used in step S6 to update the residual policy network through online reinforcement learning.

[0097] In this embodiment, without changing the parameters of the VLA main model, gating compensation is performed on the original VLA actions only through the residual policy network. This enables the system to perform local error correction when anomalies occur, while keeping the original VLA policy undisturbed when execution is normal, thus balancing the stability, error correction capability, and online deployment security of the VLA model.

[0098] S6: Using the state transition data in the experience replay buffer as training samples and the anomaly confidence level as the sampling priority, perform online reinforcement learning updates on the VLA model; this step includes the following sub-steps: S61: Receive state transition data from the experience replay buffer and determine the sampling priority of the state transition data based on the anomaly confidence level; Specifically, to prioritize samples with higher anomaly levels and greater error-correction value, this embodiment determines the sampling priority of each state transition data based on the anomaly confidence level. For the i-th state transition data... :

[0099] Its sampling priority is defined as :

[0100] in, To prevent constants with zero priority.

[0101] S62: Sample training samples from the experience replay buffer according to the sampling priority, call the VLA model to score the task completion progress corresponding to the training samples, and obtain a reward signal; Specifically, the sampling priority is normalized to a sampling probability:

[0102] in, This represents the probability that the i-th state transition data is sampled. This represents the priority adjustment coefficient. When... When the value is large, samples with higher anomaly confidence have a higher sampling probability.

[0103] The training sample set is sampled from the empirical replay buffer according to the sampling probability. :

[0104] Where b represents the batch sampling quantity.

[0105] Subsequently, the pre-trained Visual Language Analysis (VLA) model is invoked to score the task completion progress corresponding to the training samples. For the i-th state transition sample, the VLA model outputs a task completion progress score based on the observed state after the correction action, the current task progress map, and the current sub-objective to be verified, thus obtaining a reward signal. :

[0106] Where P represents the task progress graph. This indicates the current sub-target to be verified corresponding to the state transition sample.

[0107] When the agent performs a corrective action and gets closer to the sub-goal completion state, the VLA model outputs a higher task completion progress score. When the corrective action causes the agent to move away from the sub-goal or the task to fail, the VLA model outputs a lower task completion progress score.

[0108] S63: Based on the training samples and the reward signal, the VLA model residual policy network is updated by online reinforcement learning to obtain the updated residual policy network, and the network is used to generate compensation actions in subsequent control cycles; Specifically, taking the policy gradient form as an example, the residual policy network parameters The optimization objective can be expressed as :

[0109] in, Represents the residual policy network. Indicates VLA intermediate layer features, This indicates a compensatory action.

[0110] Accordingly, residual policy network parameters Update as follows:

[0111] in, This represents the learning rate.

[0112] After one or more online updates, the updated residual policy network is obtained:

[0113] The updated residual policy network is returned to step S5, where it continues to generate compensation actions based on the current observations and VLA intermediate layer features in subsequent control cycles.

[0114] In this embodiment, the task completion progress score provided by the VLA model is used as a reinforcement learning reward signal, and the anomaly confidence is used to prioritize the sampling of experience samples. This enables the residual policy network to focus on learning execution segments with high anomaly and high error correction value, thereby continuously improving the online error correction capability of the industrial intelligent agent without updating the main parameters of the VLA model.

[0115] Corresponding to the aforementioned embodiments of the VLA online self-correction method based on visual question answering verification, this application also provides a VLA online self-correction method based on visual question answering verification. Figure 2This is a block diagram illustrating a VLA online self-correcting device based on visual question-and-answer verification, according to an exemplary embodiment. (Refer to...) Figure 2 The device includes: Task progress generation module 1 is used to receive the language instructions of the agent's current task, perform offline parsing of the language instructions using the VLA model, and generate a task progress map including ordered sub-objectives and the corresponding visual verification conditions of each sub-objective. The anomaly trigger detection module 2 is used to determine the current sub-target to be verified based on the task progress map during the process of the VLA model being generated and executing actions step by step, and to perform anomaly trigger detection by utilizing the deviation between the current observed features of the VLA model and the expected feature distribution of the current sub-target to be verified. The visual question answering verification module 3 is used to respond to the abnormal triggering signal, input the visual verification conditions corresponding to the current observation frame and the current sub-target to be verified into the VLA model to perform zero-sample visual question answering verification, and output the abnormal confidence level with continuous values ​​based on the verification result. The timing anomaly determination module 4 is used to perform exponential decay weighted accumulation of the anomaly confidence within a sliding time window to obtain the timing anomaly integral, and to determine whether to enter the error correction mode based on the timing anomaly integral. The residual action correction module 5 is used to respond to the indication of the correction mode, generate compensation actions by the residual policy network of the VLA model based on the current observation features and intermediate layer features, and generate gating coefficients by combining the anomaly confidence to obtain correction actions, drive the agent to execute correction actions, and store the corresponding state transition data into the experience replay buffer. The reinforcement learning update module 6 is used to perform online reinforcement learning updates on the VLA model using the state transition data in the experience replay buffer as training samples and the anomaly confidence as the sampling priority.

[0116] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-described visual question-answering verification-based VLA online self-correction method. Figure 3 The diagram shown is a hardware structure diagram of any data processing-capable device where a self-attention fusion optimization device based on twin structure masks provided in an embodiment of the present invention is located. (Except for...) Figure 3 In addition to the processor and memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0117] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the VLA online self-correction method based on visual question-answering verification as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0118] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0119] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A VLA online self-correction method based on visual question answering verification, characterized in that, include: Receive the language instructions for the current task of the intelligent agent, use the VLA model to parse the language instructions offline, and generate a task progress map including ordered sub-goals and the corresponding visual verification conditions of each sub-goal; During the process of the VLA model generating and executing actions step by step, the current sub-target to be verified is determined based on the task progress map, and anomaly trigger detection is performed by using the deviation between the current observed features of the VLA model and the expected feature distribution of the current sub-target to be verified, and anomaly trigger signal is output. In response to the abnormal trigger signal, the visualization verification conditions corresponding to the current observation frame and the current sub-target to be verified are input into the VLA model for zero-sample visual question answering verification, and the abnormal confidence level with continuous values ​​is output based on the verification result. The anomaly confidence is exponentially decayed and weighted within the sliding time window to obtain the time-series anomaly integral, and the time-series anomaly integral is used to determine whether to enter the error correction mode. In response to the error correction mode instruction, the residual policy network of the VLA model generates a compensation action based on the current observation features and intermediate layer features, and generates a gating coefficient based on the anomaly confidence to obtain a correction action, which drives the agent to execute the correction action and stores the corresponding state transition data in the experience replay buffer. The VLA model is updated online using the state transition data in the experience replay buffer as training samples and the anomaly confidence level as the sampling priority.

2. The VLA online self-correction method based on visual question answering verification according to claim 1, characterized in that, The system receives language instructions for the agent's current task, performs offline parsing of these instructions using a VLA model, and generates a task progress graph including ordered sub-goals and corresponding visual verification conditions for each sub-goal. Receive the language instruction for the agent's current task, input the language instruction into the pre-trained VLA model, and parse it to obtain multiple sub-objectives required to complete the current task; Based on the execution order of the multiple sub-targets, an ordered sequence of sub-targets is generated, and a visual verification condition is generated for each sub-target in the sequence to determine whether the sub-target has been completed. By associating the sub-target sequence with the corresponding visual verification conditions for each sub-target, a task progress map is obtained.

3. The VLA online self-correction method based on visual question answering verification according to claim 1, characterized in that, During the step-by-step generation and execution of actions by the VLA model, the current sub-target to be verified is determined based on the task progress map, and anomaly trigger detection is performed using the deviation between the current observed features of the VLA model and the expected feature distribution of the current sub-target to be verified, outputting anomaly trigger signals, including: Obtain the task progress map and the execution status of the VLA model in the current control cycle, determine the completed sub-objectives based on the execution status, and determine the current sub-objectives to be verified from the task progress map; Obtain the current observation frame of the agent and extract the visual features output by the VLA model visual encoder for the current observation frame; Obtain the expected feature distribution corresponding to the current sub-target to be verified, and calculate the deviation of the visual feature from the expected feature distribution; When the deviation does not exceed the first threshold, the current execution state is determined to be normal and the VLA model action generation process continues; when the deviation exceeds the first threshold, an abnormal trigger signal and the current observation frame are output.

4. The VLA online self-correction method based on visual question answering verification according to claim 1, characterized in that, In response to the anomaly trigger signal, the visual verification conditions corresponding to the current observation frame and the current sub-target to be verified are input into the VLA model for zero-shot visual question answering verification, and the anomaly confidence level with continuously taking values ​​is output based on the verification result, including: Receive the anomaly trigger signal and the current observation frame, and in response to the anomaly trigger signal, read the visual verification conditions corresponding to the current sub-target to be verified from the task progress graph; The current observation frame and the visualization verification conditions are constructed as visual question-and-answer inputs, and then input into the VLA model for zero-sample verification. Based on the zero-sample verification results, each visualization verification condition is checked to determine whether it is met. Based on the satisfaction of each visualization verification condition, the outlier confidence of continuously taking values ​​is calculated and output.

5. The VLA online self-correction method based on visual question answering verification according to claim 1, characterized in that, Within a sliding time window, the anomaly confidence level is exponentially decayed and weighted to obtain a temporal anomaly integral. Based on this temporal anomaly integral, it is determined whether to enter error correction mode, including: Receive the anomaly confidence level, write the anomaly confidence level of the current control cycle into the sliding time window, and perform exponentially decaying weighted summation on the anomaly confidence levels of multiple control cycles within the sliding time window to obtain the time series anomaly integral. Determine whether the time-series anomaly integral exceeds a preset threshold. If the time-series anomaly integral exceeds the preset threshold, output an indication to enter error correction mode; if the time-series anomaly integral does not exceed the preset threshold, continue the VLA model action generation process.

6. The VLA online self-correction method based on visual question answering verification according to claim 1, characterized in that, In response to the indication of the error correction mode, the residual policy network of the VLA model generates a compensation action based on the current observation features and intermediate layer features, and combines the anomaly confidence to generate gating coefficients to obtain a correction action, driving the agent to execute the correction action, and storing the corresponding state transition data in the experience replay buffer, including: Receive the instruction to enter error correction mode and the anomaly confidence level; While keeping the VLA model parameters frozen, the current observation features and the intermediate layer features of the VLA model are obtained, and the current observation and the intermediate layer features of the VLA model are input into the residual policy network to generate compensation actions; Gating coefficients are generated based on the anomaly confidence level, wherein the gating coefficients are initially zero and increase as the anomaly confidence level increases; the original action output by the VLA model is added to the compensated action weighted by the gating coefficients to obtain the corrected action; The control agent executes the corrective action and stores the state transition data generated by the corrective action into the experience replay buffer.

7. The VLA online self-correction method based on visual question answering verification according to claim 1, characterized in that, Using the state transition data in the experience replay buffer as training samples and the anomaly confidence level as the sampling priority, the VLA model is updated online through reinforcement learning, including: Receive state transition data from the experience replay buffer and determine the sampling priority of the state transition data based on the anomaly confidence level; Training samples are sampled from the experience replay buffer according to the sampling priority, and the VLA model is called to score the task completion progress corresponding to the training samples to obtain a reward signal. Based on the training samples and the reward signal, the VLA model residual policy network is updated through online reinforcement learning to obtain the updated residual policy network, which is then used to generate compensation actions in subsequent control cycles.

8. A VLA online self-correcting device based on visual question-answering verification, characterized in that, include: The task progress generation module is used to receive the language instructions of the agent's current task, use the VLA model to parse the language instructions offline, and generate a task progress map including ordered sub-objectives and the corresponding visual verification conditions of each sub-objective. An anomaly trigger detection module is used to determine the current sub-target to be verified based on the task progress map during the step-by-step generation and execution of actions by the VLA model, and to perform anomaly trigger detection by utilizing the deviation between the current observed features of the VLA model and the expected feature distribution of the current sub-target to be verified, and output anomaly trigger signal. The visual question-answering verification module is used to respond to the abnormal triggering signal, input the visual verification conditions corresponding to the current observation frame and the current sub-target to be verified into the VLA model to perform zero-sample visual question-answering verification, and output the abnormal confidence level with continuous values ​​based on the verification result. The timing anomaly determination module is used to perform exponential decay weighted accumulation of the anomaly confidence within a sliding time window to obtain the timing anomaly integral, and to determine whether to enter the error correction mode based on the timing anomaly integral. The residual action error correction module is used to respond to the indication of the error correction mode. The residual policy network of the VLA model generates compensation actions based on the current observation features and intermediate layer features, and generates gating coefficients based on the anomaly confidence to obtain correction actions. It drives the agent to execute correction actions and stores the corresponding state transition data in the experience replay buffer. The reinforcement learning update module is used to perform online reinforcement learning updates on the VLA model using the state transition data in the experience replay buffer as training samples and the anomaly confidence as the sampling priority.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-7.

Images