VLA-oriented fault-aware autonomous correction recovery method, device, equipment and medium
By using offline multimodal large language model analysis and online monitor coordination, retry and reset augmentation datasets are generated, and an expert policy library is trained. This solves the problem of autonomous recovery of robot operation models under minor perturbations and environmental changes, and achieves accurate fault differentiation and autonomous recovery.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN TIANYUANXING TECHNOLOGY CO LTD
- Filing Date
- 2026-06-02
- Publication Date
- 2026-07-03
Smart Images
Figure CN122334338A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for fault perception autonomous correction and recovery for VLA. Background Technology
[0002] In the field of robotics, vision-language-action models, by integrating perception, language, and control capabilities, have shown potential for performing long-duration tasks in open worlds. However, in practical deployments, these models often fail due to minor perturbations, accidental contact, or execution deviations, making it difficult to achieve reliable autonomous operation.
[0003] In existing technologies, some methods utilize multimodal large language models to provide high-level fault analysis, but they rely on a fixed skill set and lack adaptability to low-level execution biases. Another type of method uses reinforcement learning to explicitly learn recovery behaviors, but its sample efficiency is low, making it difficult to apply to long-term tasks in real-world scenarios. Data-driven methods improve robustness through large-scale data augmentation; however, because the training data is mainly composed of successful samples and the trajectories are monotonous, the model cannot cope with faults caused by catastrophic changes in environmental conditions (such as objects tipping over). None of these methods have achieved a unified fault recovery mechanism that can handle both minor posture perturbations and destructive changes in environmental conditions, meaning that robots still require human intervention to recover from failures. Summary of the Invention
[0004] This invention provides a fault-aware autonomous correction and recovery method, device, equipment, and medium for VLA (Variable Actuation Model). The technical problem it aims to solve is: how to provide a solution that can effectively improve the autonomous recovery capability of robot operation models under various fault scenarios.
[0005] In a first aspect, the present invention provides a fault perception autonomous correction and recovery method for VLA, comprising: in response to receiving a trigger command for a pre-trained vision-language-action model, controlling the vision-language-action model to perform multiple rounds on a specified task, and collecting execution video and trajectory data generated during the execution process;
[0006] The multimodal large language model is used as an offline fault analyzer to analyze the execution video. Based on the preset in-distribution / out-of-distribution error classification system, structured error metadata is output for each fault. The error metadata includes error type, reset target and fault timestamp.
[0007] If the error type is an in-distribution error, then the retry enhancement process is executed: based on the trajectory data, a disturbance-bridging segment is injected between task segments to generate a retry enhancement dataset. The disturbance-bridging segment includes a disturbance action that moves the robot arm to an out-of-distribution posture and a bridging action that connects the robot from the disturbed posture to the effective starting posture of the next task segment.
[0008] If the error type is an out-of-distribution error, then the reset skill learning process is executed: based on the fault timestamp in the error metadata, the environmental state is extracted from the trajectory data, a preset number of human demonstration data for the reset target are collected, and the perturbation-bridging segment is applied to the human demonstration data to enhance it, generating a reset enhancement dataset;
[0009] Based on the retry enhancement dataset and the reset enhancement dataset, train an expert policy library containing a main task adapter and at least one reset skill adapter;
[0010] During the inference phase, the execution process of the vision-language-action model is observed in real time by an online multimodal large language model monitor, and the adapter switching in the expert policy library is coordinated based on the observation results to execute the main task or reset the skill.
[0011] Secondly, the present invention also provides a fault-aware autonomous correction and recovery device for VLA, which includes a unit for performing the above-described method.
[0012] Thirdly, the present invention also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.
[0013] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.
[0014] This invention provides a method, apparatus, device, and medium for fault perception and autonomous correction and recovery in VLA (Variable Language Algorithm). Specifically, this invention analyzes execution videos using an offline multimodal large language model, outputting structured error metadata based on a pre-defined in-distribution / out-of-distribution classification system, achieving accurate differentiation between perturbation-type and catastrophic faults. For in-distribution errors, a retry augmentation dataset is generated using a perturbation-bridging segment, endowing the model with autonomous recovery capabilities for posture perturbations. For out-of-distribution errors, fault states are extracted using MLLM guidance and human demonstration data is collected, combined with perturbation-bridging segment augmentation to generate a reset augmentation dataset, enabling the model to master environmental recovery skills. Furthermore, by training an expert policy library containing the main task and reset skill adapters, and with real-time observation of the execution process and dynamic coordination of adapter switching by an online MLLM monitor, a complete mechanism from fault perception to closed-loop autonomous recovery is constructed. This method effectively improves the autonomous recovery capability of robot operation models under various fault scenarios, solving the problem of the lack of a unified fault recovery mechanism in existing technologies. Attached Figure Description
[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 A flowchart illustrating a fault-aware autonomous correction and recovery method for VLA provided in an embodiment of the present invention;
[0017] Figure 2 This is a flowchart illustrating the overall process architecture of the FLARE framework provided in this embodiment of the invention.
[0018] Figure 3 This is a sensitivity analysis diagram showing the impact of perturbation and bridging hyperparameters on performance and data generation success rate in an embodiment of the present invention.
[0019] Figure 4 This is a visual illustration of a skill failure reset case in an embodiment of the present invention;
[0020] Figure 5 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation
[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0022] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.
[0023] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0024] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0025] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."
[0026] Please see Figure 1 This invention provides a fault-aware autonomous correction and recovery method for VLA (Variable Availability) systems, which includes the following steps:
[0027] S1, in response to receiving a trigger command for the pre-trained vision-language-action model, controls the vision-language-action model to perform multiple rounds on a specified task, and collects the execution video and trajectory data generated during the execution process.
[0028] In specific implementation, in response to receiving a trigger command for a pre-trained visual-language-action (VLA) model, the system controls the VLA model to perform multiple rounds on a specified task, and collects the execution video and trajectory data generated during the execution process. The trigger command can be in various forms, such as system startup, user voice command, or reaching a preset time point; this invention is not specifically limited to any one of these. The visual-language-action (VLA) model is an embodied intelligent model that integrates visual perception, language understanding, and action control, such as OpenVLA; this invention is not specifically limited to any one of these. After receiving the trigger command, the VLA model performs multiple rounds of attempts on a preset operation task (such as making coffee, stacking blocks, etc.), including successful and failed trajectories. The system forms an initial fault data pool by recording the video data and action sequences generated by the VLA model during runtime.
[0029] S2, using a multimodal large language model as an offline fault analyzer, analyzes the execution video and outputs structured error metadata for each fault based on a preset in-distribution / out-of-distribution error classification system. The error metadata includes error type, reset target, and fault timestamp.
[0030] In specific implementation, a multimodal large language model (MLLM) is used as an offline fault analyzer to analyze the execution videos. Based on a preset in-distribution / out-of-distribution error classification system, structured error metadata is output for each fault. The MLLM can simultaneously understand visual and textual information; for example, Gemini 2.5-Pro is used, but this invention is not specifically limited to this model. The system inputs a large number of collected execution videos into the MLLM and, in conjunction with preset prompts, enables it to diagnose each fault.
[0031] The pre-defined in-distribution / out-of-distribution error classification system is the core criterion: In-distribution error refers to a situation where the environmental state is still within the valid range (e.g., the object is in the correct position), but the robot's posture deviates from the normal posture in the demonstration data (e.g., abnormal posture caused by hand tremors). This is because the monotonicity of the trajectory in the demonstration data causes the model to learn a false correlation between the robot's posture and the environmental state, thus making it unable to recover from the new posture. Out-of-distribution error refers to a situation where a catastrophic failure occurs in the environment (e.g., a cup tipping over, an object getting stuck), making it impossible to recover through normal task strategies.
[0032] The multimodal large language model outputs structured error metadata based on this system. The error metadata includes error type (i.e., in-distribution or out-of-distribution), reset target (the name of the object to be reset, such as "coffee cup" or "coffee powder box"), error group (a semantic set of one or more related objects that constitute the fault state), and fault timestamp (the specific video frame or time when the fault occurred).
[0033] In some preferred embodiments, the structured error metadata is in JSON format, and the error type, reset target, error group, and fault timestamp contained in the structured error metadata are all stored in the form of structured fields.
[0034] In practice, the structured error metadata is in JSON format. JSON is a lightweight data exchange format whose structure consists of key-value pairs, making it easy for humans to read and write, as well as for machines to parse and generate. During the offline fault analysis phase, the system inputs preset prompts into the multimodal large language model, requiring it to output the analysis results in JSON format. Upon receiving input containing the execution video, the multimodal large language model diagnoses each fault case according to preset reasoning logic and generates structured text conforming to JSON syntax.
[0035] The structured error metadata, including error type, reset target, error group, and fault timestamp, is stored in structured fields. Specifically, the JSON object contains four top-level fields: the first field stores the error type, with a value representing an identifier indicating an in-distribution or out-of-distribution fault; the second field stores the reset target, with a value of a string representing the name of a single object that needs to be reset; the third field stores the error group, with a value of an array of strings, where each element represents the name of a related object constituting the fault state; and the fourth field stores the fault timestamp, with a value representing the frame number or time point in the video where the fault occurred. For example, a fault analysis result for a "coffee machine" task would contain these four fields, recording the fault type as "OOD," the reset target as "coffee_pod," the error group as "coffee_pod" and "coffee_machine_bracket," and the fault timestamp as "1523." In this way, complex, unstructured fault videos are transformed into machine-readable, semantically clear structured data. The error metadata in JSON format will be passed to subsequent data augmentation and training processes as a key input to guide the reset of skill learning.
[0036] S3, if the error type is an in-distribution error, then execute the retry enhancement process: based on the trajectory data, inject a disturbance-bridging segment between task segments to generate a retry enhancement dataset. The disturbance-bridging segment includes a disturbance action to move the robot arm to an out-of-distribution posture and a bridging action to connect the robot from the disturbed posture to the effective starting posture of the next task segment.
[0037] In specific implementation, if the error type is an in-distribution error, a retry enhancement process is executed: based on the trajectory data, a perturbation-bridging segment is injected between task segments to generate a retry enhancement dataset. The perturbation-bridging segment is the core enhancement method of this scheme, comprising two stages: First, when the task reaches a certain stage (e.g., after "grabbing the cup"), a perturbation action is executed. This perturbation action moves the robot arm to an arbitrary out-of-distribution posture through random joint velocities or end effector movements, thereby breaking the monotonicity of the original trajectory. Subsequently, a bridging action is executed. This action uses kinematic interpolation (e.g., linear interpolation for position, spherical linear interpolation for rotation, not specifically limited in this invention) to reconnect the robot from the perturbed posture to the initial posture required for the subsequent task segment (e.g., "placing the cup"). The bridging segment is combined with the subsequent task segment to form a new training sequence. The generated trajectory is replayed to eliminate failed trajectories caused by collisions or other reasons. This is equivalent to providing the model with a large number of demonstrations of "recovering from incorrect postures and continuing the task," thereby training a model with built-in retry capabilities.
[0038] In some preferred embodiments, the step of injecting perturbation-bridging segments between task segments based on the trajectory data to generate a retry augmentation dataset includes: synthesizing baseline trajectories from a library of sub-task segments demonstrated by humans; inserting the perturbation-bridging segments between task segments of the baseline trajectories, wherein the perturbation phase moves the robot arm to an out-of-distribution pose through random motion, and the bridging phase generates motion trajectories from the perturbation-post-pose to subsequent task segments through kinematic interpolation; and combining the bridging segments with subsequent task segments into a new training sequence, which is then added to the retry augmentation dataset.
[0039] In practice, the first step is to synthesize a baseline trajectory from a library of sub-task segments demonstrated by humans. The system pre-divides successfully demonstrated human trajectories into independent sub-task segments, such as "grabbing a cup," "moving the cup to the coffee machine," and "placing the cup." When data needs to be generated for a new scenario (such as different object placement positions), the system selects a suitable sub-task segment from the library based on the task logic and performs kinematic transformations to match the new object posture. These segments are then sequentially stitched together to form a baseline trajectory. While this baseline trajectory can complete the task, it still exhibits trajectory monotonicity, meaning there is a strong correlation between the robot's posture and task progress.
[0040] Furthermore, the perturbation-bridging segments are inserted between the task segments of the baseline trajectory. To achieve true robustness, the system injects perturbation-bridging segments at the connection points between task segments (e.g., after "grabbing the cup" and before executing "moving the cup") during the execution flow of the baseline trajectory. Each perturbation-bridging segment consists of two parts: a perturbation phase and a bridging phase. The perturbation phase moves the robot arm to an arbitrary, unseen off-distribution pose through random motion (such as random joint velocities or end effector motion), thereby breaking the coupling between pose and state. For example, before executing "moving the cup," the model might first make an anomalous arm-waving motion. The bridging phase follows the perturbation phase and generates a sequence of motions that smoothly transitions from the perturbed pose to the required starting pose for the next task segment ("moving the cup") using standard kinematic interpolation techniques (linear interpolation for position and spherical linear interpolation for rotation). This bridging sequence of motions serves as a "correction" process, enabling the model to learn the ability to recover from anomalous poses to the normal task flow.
[0041] Furthermore, the bridging segment is combined with subsequent task segments to form a new training sequence, which is then added to the retry augmentation dataset. After generating the augmented trajectory, the system does not use uninstructed perturbation actions as training targets. Instead, it combines the bridging action with subsequent standard task actions to form new subsequences (e.g., bridging segment action + "move cup" action sequence). By training on such sequences, the vision-language-action model can explicitly learn how to correctly execute the next task action when the robot is in various different initial poses (i.e., the end state of the bridging segment), thus endowing it with an inherent "retry" capability.
[0042] This embodiment systematically decouples robot posture from environmental state through a "synthesis-perturbation-bridging" data augmentation pipeline, fundamentally solving the vulnerability of VLA models caused by monotonous training data trajectories. The injection of the perturbation-bridging segment allows the model to see examples during the learning process of "starting from a correct state (valid environmental state) but with an incorrect posture, and returning to the correct trajectory," thus learning to decouple task progress (reflected by the environmental state) from the robot's own configuration. This decoupling gives the model built-in retry robustness: when the robot's posture deviates (i.e., in-distribution error) due to minor perturbations or external collisions during execution, the model can autonomously start from the new posture and complete subsequent operations through bridging and task actions without external intervention. This strategy is not dependent on specific tasks or environments, is a general training paradigm, and its large-scale data generation process is highly automated, resulting in significantly higher sample efficiency compared to manually synthesizing fault data or reinforcement learning trial and error.
[0043] In some preferred embodiments, the disturbance parameters of the disturbance-bridging segment include a maximum rotation angle and a maximum translation distance, wherein the maximum rotation angle is 30° to 45° and the maximum translation distance is 0.5 m to 0.7 m.
[0044] In practice, the perturbation parameters of the perturbation-bridging segment include the maximum rotation angle and the maximum translation distance. These perturbation parameters need to be set when generating the retry augmentation dataset and the reset augmentation dataset. These parameters control the magnitude of the "perturbation" action: the rotation angle determines the extent of the robot arm's rotation away from its original posture, and the translation distance determines its movement distance in space.
[0045] According to experimental research, the maximum rotation angle is 30° to 45°, and the maximum translation distance is 0.5 meters to 0.7 meters. Experiments show that the model achieves optimal performance (i.e., the highest task success rate) when the rotation angle is set at 30° and the translation distance at 0.7 meters. Within this parameter range, the perturbation amplitude is sufficiently large to break the monotonicity of the original trajectory, providing effective training samples for the model to learn robustness. If the perturbation amplitude is too small, the generated samples will not differ much from the original data, failing to effectively improve robustness; if the perturbation amplitude is too large, although it can generate demonstration data with higher variance, it will lead to a decrease in the proportion of effective trajectories, with a large number of generated trajectories failing due to collisions and other reasons, thus reducing the efficiency of data generation.
[0046] Please see Figure 3 , Figure 3 The results of the sensitivity analysis of the perturbation-bridging hyperparameter on model performance and data generation success rate are presented.
[0047] Figure 3 (1) shows the impact of different rotation angles (0° to 75°) on the success rate of the model task and the success rate of data generation when the translation distance is fixed (0.5 meters); Figure 3 Figure (2) shows the effect of different translation distances (0.1 m to 1.1 m) on the above indicators when the rotation angle (45°) is fixed. The experimental results show that when the rotation angle is between 30° and 45° and the translation distance is between 0.5 m and 0.7 m, the model can achieve better performance and the data generation success rate remains at a relatively high level. This experimental conclusion provides a basis for determining the optimal range of the perturbation parameters, that is, under the premise of taking into account both model performance and data generation efficiency, the maximum rotation angle of the perturbation stage is set to 30° to 45° and the maximum translation distance is set to 0.5 m to 0.7 m.
[0048] S4. If the error type is an out-of-distribution error, then execute the reset skill learning process: extract the environmental state from the trajectory data based on the fault timestamp in the error metadata, collect a preset number of human demonstration data for the reset target, and apply the perturbation-bridging segment to the human demonstration data to enhance it, thereby generating a reset enhancement dataset.
[0049] In specific implementation, if the error type is an out-of-distribution error, a reset skill learning process is executed: Based on the fault timestamp in the error metadata, the environmental state is extracted from the trajectory data; a preset number of human demonstration data points for the reset target are collected; and the perturbation-bridging segment is applied to the human demonstration data to enhance it, generating a reset enhancement dataset. This process first accurately extracts the environmental physical state at the time of the fault from the recorded trajectory data based on the fault timestamp, such as the specific posture of a coffee powder container stuck on a bracket. Further, starting from this fault state, a small amount (e.g., 10-20) of "reset" demonstration data is provided by a human operator, demonstrating how to straighten or reset the reset target (e.g., the coffee powder container) to a valid state. This human demonstration data is then input into the perturbation-bridging segment processing flow for large-scale enhancement, generating a large number of reset trajectory samples under different robot starting postures and scene layouts, forming a robust reset skill dataset.
[0050] In some preferred embodiments, the step of extracting the environmental state from the trajectory data based on the fault timestamp in the error metadata, collecting a preset number of human demonstration data for the reset target, and applying the perturbation-bridging segment to the human demonstration data to enhance it, generating a reset enhancement dataset, includes: extracting the environmental state from the trajectory data based on the fault timestamp to obtain the error state of the reset target; collecting a preset number of human demonstration data with the error state as the starting state, each of the human demonstration data containing a recovery trajectory that restores the reset target from the error state to a valid state; based on the error group in the error metadata, preserving the relative posture within the error group when placing the reset target to maintain the semantic integrity of the fault; applying the perturbation-bridging segment to the recovery trajectory in the human demonstration data to generate reset trajectories under various robot starting postures and scene layouts; and adding the reset trajectories to the reset enhancement dataset.
[0051] In practice, firstly, the environmental state is extracted from the trajectory data based on the fault timestamp to obtain the error state of the reset target. During the offline fault analysis phase, the multimodal large language model has already output the precise timestamp of the fault occurrence. Using this timestamp, the system extracts the physical state (position, rotation angle, velocity, etc.) of all objects at the instant the fault occurred from the saved simulation or log data, thus obtaining a high-fidelity, reproducible error state. For example, if the fault is "the coffee powder cartridge is stuck on the stand," the extracted error state will accurately describe the rotation and position of the cartridge relative to the coffee machine stand.
[0052] Furthermore, starting from the error state, a predetermined number of human demonstration data points are collected. Each human demonstration data point contains a recovery trajectory that restores the reset target from the error state to a valid state. The system uses the extracted error state as the initial scenario for reset. Starting from this initial state, a human operator (such as a researcher) manually controls the robot to execute a trajectory that restores the faulty object (reset target) to a valid state (e.g., removing a stuck powder box and correctly placing it back). This step is crucial for "teaching" the robot how to perform the "reset" skill, and only a small number (e.g., 10-20) of high-quality demonstrations are needed.
[0053] Furthermore, based on the error groups in the error metadata, the relative poses within the error groups are preserved when placing the reset target to maintain the semantic integrity of the fault. The error metadata output by the multimodal large language model includes "error groups," which are multiple related objects constituting the fault (e.g., "stuck toner cartridge" and "stand"). To allow the model to learn the "essence" of the fault rather than its "specific location," when the system places the collected reset demonstration trajectory into a new random environment for augmentation, it extracts the relative pose relationships of each object within the error group (e.g., the rotation angle and contact point of the toner cartridge relative to the stand) and reproduces this relative relationship in the new scene. In this way, regardless of where the fault occurs on the desktop, the model can recognize the semantically consistent state that "the toner cartridge is stuck on the stand."
[0054] Furthermore, the perturbation-bridging segment is applied to the recovery trajectory in the human demonstration data to generate reset trajectories under various robot initial poses and scene layouts, and these reset trajectories are added to the reset enhancement dataset. Similar to the retry enhancement process, the system applies the perturbation-bridging segment enhancement strategy again to the reset trajectories from the human demonstration, generating a large number of reset trajectory samples with different robot initial poses and scene context layouts. This ensures that the finally trained reset skill not only possesses kinematic robustness (unaffected by the robot's initial pose) but also generalization ability in terms of environment layout (unaffected by the specific coordinates of the fault occurrence).
[0055] This embodiment successfully addresses the problem of model lack of reset skills caused by "successful sample bias" in demonstration data by using offline fault mining and guidance driven by a multimodal large language model. First, the offline multimodal large language model analyzer automatically transforms massive amounts of failure videos into accurate and reproducible fault states (error states), providing high-quality "teaching materials" for subsequent "instruction" and avoiding the tedious process of manually locating and reproducing faults from videos. Furthermore, by introducing the concept of "error groups," multiple spatially related fault objects are treated as a semantic whole, and their internal relative poses are preserved during data augmentation. This ensures that the model learns the semantic essence of the fault (such as the "stuck" state) rather than its absolute position in the scene, greatly improving the generalization ability of the reset skill. Further, based on a small number of human demonstrations, a perturbation-bridging augmentation strategy is used for large-scale expansion, enabling the model to master general reset capabilities for various robot poses and scene layouts with a few teaching samples, achieving efficient sample learning.
[0056] To further illustrate the application scenarios of resetting the skills learning process, please refer to [link / reference]. Figure 4 . Figure 4 Typical cases of distributed out-of-range error failures were presented in the "coffee making" and "three-piece assembly" tasks.
[0057] in, Figure 4 (a) shows the malfunction of the coffee powder cartridge tipping over during the "Make Coffee" task; Figure 4 (b) shows the malfunction of the coffee machine lid tipping over during the "make coffee" task; Figure 4 (c) shows the fault state of the T-block tipping over during the "three-piece assembly" task; Figure 4 Figure (d) illustrates the fault state of the U-shaped block tipping over in the "three-piece assembly" task. All of these fault states cause the environment to enter an out-of-distribution error state that cannot be recovered from using standard task strategies. For this type of fault, the aforementioned reset skill learning process needs to be executed: extracting the environmental state based on the fault timestamp, collecting human demonstration data starting from the error state, and generating a reset enhancement dataset using a perturbation-bridging enhancement strategy to train a dedicated reset skill adapter capable of performing actions such as "righting the coffee powder cartridge" and "resetting the coffee machine lid."
[0058] S5. Based on the retry enhancement dataset and the reset enhancement dataset, train an expert policy library containing a main task adapter and at least one reset skill adapter.
[0059] In practice, based on the retry augmentation dataset and the reset augmentation dataset, an expert policy library is trained, comprising a main task adapter and at least one reset skill adapter. The expert policy library adopts a modular design, containing a main task adapter for performing the original task (e.g., "making coffee"), and one or more reset skill adapters for performing specific reset tasks (e.g., "aligning the coffee powder hopper"). These adapters are all trained on a unified pre-trained VLA backbone model, but they operate independently and do not interfere with each other.
[0060] In some preferred embodiments, training an expert policy library comprising a main task adapter and at least one reset skill adapter based on the retry augmentation dataset and the reset augmentation dataset includes: training the main task adapter on a shared pre-trained visual-language-action backbone model using the retry augmentation dataset using low-rank adaptation techniques, wherein the main task adapter is invoked via a main task instruction; and training the reset skill adapter on the pre-trained visual-language-action backbone model using the reset augmentation dataset using the low-rank adaptation techniques, wherein each reset skill adapter corresponds to a unique reset skill cue word.
[0061] In this specific implementation, the embodiment achieves efficient and scalable modular training by introducing low-rank adaptation (LoRA) technology. The specific implementation method is as follows.
[0062] The main task adapter is trained on a shared pre-trained visual-language-action backbone model using the retry augmentation dataset, employing low-rank adaptation (LoRA). LoRA is a parameter-efficient fine-tuning method. It injects trainable low-rank decomposition matrices into the network layers while freezing the original weight matrices of the pre-trained model. Compared to fine-tuning all model parameters (full fine-tuning), LoRA requires training only a very small number of parameters to adapt to new tasks, significantly reducing training costs and memory usage. The system first acquires a VLA backbone model pre-trained on a large amount of general data and then freezes all its basic parameters. Next, targeting the retry augmentation dataset, a dedicated main task adapter is trained on this backbone model using LoRA. Specifically, this adapter consists of a set of low-rank matrices superimposed on specific layers of the backbone model (such as attention layers). After training, the main task adapter can be invoked via a main task command (such as "make coffee").
[0063] Furthermore, the low-rank adaptation technique is used to train the reset skill adapter on the pre-trained vision-language-action backbone model with the reset augmentation dataset. For each reset skill to be learned (e.g., "reset the cup"), the system repeats the LoRA training process described above. The system acquires the reset augmentation dataset corresponding to the reset skill and uses it as the training target. Since the training of LoRA adapters is independent of each other, this means that the system can train an independent, lightweight parameter package (i.e., LoRA weights) for each reset skill on the same shared backbone model. Each reset skill adapter corresponds to a unique reset skill cue word (e.g., "reset the cup"). This design allows the model to quickly switch capabilities without affecting the performance of other skills by dynamically loading or switching these lightweight LoRA adapters as needed during inference.
[0064] This embodiment introduces Low-Rank Adaptation (LoRA) technology to transform the training of the expert policy base into lightweight parameter fine-tuning of a shared backbone model, achieving efficient and scalable modular training. Using LoRA to train the main task and reset skill adapters reduces the number of training parameters by over 90% compared to traditional full-scale fine-tuning, significantly saving computational resources and storage space, making it possible to train multiple specialized policies under limited hardware conditions. Furthermore, each reset skill adapter is trained independently and bound to specific prompts, fundamentally avoiding gradient conflicts and negative transfer problems caused by mixing different tasks (such as "grab" and "correct") in a single model weight. This modular design gives the system excellent scalability. When a new reset skill needs to be added, only a small amount of demonstration data needs to be collected for the new skill, and a new adapter needs to be trained using LoRA, without retraining existing adapters or modifying the backbone model, greatly simplifying the system's iteration and upgrade process.
[0065] S6, during the inference phase, the execution process of the visual-language-action model is observed in real time by an online multimodal large language model monitor, and the adapter switching in the expert policy library is coordinated based on the observation results to execute the main task or reset the skill.
[0066] In practice, during the inference phase, an online multimodal large language model monitor observes the execution process of the vision-language-action model in real time and coordinates adapter switching in the expert policy library based on the observation results to execute the main task or reset the skill. The online multimodal large language model monitor acts as a high-level decision-maker, analyzing the video stream of the model's execution in real time. When the task is executed normally, the model loads the main task adapter; when an in-distribution error occurs, since the model has been trained with a retry augmentation dataset, it can autonomously adjust its posture and continue the task without switching; when an out-of-distribution error occurs, the online multimodal large language model monitor identifies the fault type (e.g., "the cup has been spilled") and, based on preset rules or inference results, determines the reset skill to be invoked (e.g., "reset the cup"), and instructs the system to switch the model from the main task adapter to the corresponding reset skill adapter. After the online multimodal large language model monitor confirms that the environment has returned to a valid state (e.g., "the cup has been righted"), it instructs the system to switch back to the main task adapter, allowing the task to continue execution.
[0067] This embodiment transforms a fragile visual-language-action model into a robust system with autonomous recovery capabilities by constructing a complete closed loop of "fault classification - data augmentation - modular training - online monitoring". First, based on an in-distribution / out-of-distribution error classification system, faults are systematically categorized into self-healing perturbation-type faults and catastrophic faults requiring intervention. Two recovery paths, retry and reset, are designed to comprehensively cover various execution errors. Furthermore, the retry path, enhanced through perturbation-bridging, fundamentally solves the "spurious association" problem caused by the monotonicity of training data trajectories in the VLA model, endowing the model with built-in retry robustness. The reset path, through fault analysis and guided few-shot learning using a multimodal large language model, addresses the problem of the model's inability to learn recovery behavior on "successful sample bias" datasets. Moreover, the offline multimodal large language model, acting as a fault analyzer, can automatically perform structured analysis on a large number of failed videos, producing accurate fault metadata, providing a quality foundation for subsequent targeted data augmentation and avoiding the expensive cost of manual annotation. Furthermore, the online multimodal large language model monitor enables real-time perception of task execution status during the inference phase, and can dynamically coordinate the switching between the main task and the reset skill. This allows the model to make autonomous decisions and recover autonomously when encountering faults during execution, thereby significantly improving the success rate and robustness of complex and long-duration robot operation tasks.
[0068] In some preferred embodiments, the step of having an online multimodal large language model monitor observe the execution process of the visual-language-action model in real time and coordinate adapter switching in the expert policy library based on the observation results to execute the main task or reset the skill includes: determining in real time whether an out-of-distribution error has occurred in the current execution process of the visual-language-action model through the online multimodal large language model monitor; if no out-of-distribution error has occurred, maintaining the loading of the main task adapter and continuing to execute the main task; if an out-of-distribution error has occurred, identifying the required reset skill through the online multimodal large language model monitor and instructing the visual-language-action model to switch to the corresponding reset skill adapter to execute the reset skill; when the online multimodal large language model monitor confirms that the environment has been restored to an effective state, instructing the visual-language-action model to reload the main task adapter and continue to execute the main task.
[0069] In practice, firstly, the online multimodal large language model monitor determines in real time whether out-of-distribution errors occur during the current execution of the vision-language-action model. During the inference phase, the online multimodal large language model monitor continuously analyzes the video stream generated by the model execution. Similar to offline multimodal large language models, it focuses on real-time fault identification. It determines the current state based on preset inference logic: whether irreversible faults such as overturning or freezing exist. If no out-of-distribution errors occur, the main task adapter is maintained, and the main task continues to execute. In this case, the model has already developed inherent robustness on the "retry augmentation dataset," sufficient to handle various pose perturbations (in-distribution errors), without the need for intervention from the online multimodal large language model, thus ensuring the smoothness and real-time performance of task execution.
[0070] Furthermore, if an out-of-distribution error occurs, the online multimodal large language model monitor identifies the required reset skill and instructs the visual-language-action model to switch to the corresponding reset skill adapter to execute the reset skill. When the online multimodal large language model monitor identifies an out-of-distribution error (e.g., "the coffee cup was knocked over"), it determines the specific reset skill (e.g., "reset the coffee cup") required to resolve the fault based on preset rules or through inference. Subsequently, the monitor issues a switching instruction to the VLA model, and the model dynamically unloads the current primary task adapter LoRA weights and loads the LoRA weights corresponding to the "reset the coffee cup" skill. The model then begins executing the reset skill, aiming to restore the environment to a stable state where the task can continue.
[0071] Furthermore, once the online multimodal large language model monitor confirms that the environment has been restored to a valid state, it instructs the visual-language-action model to reload the main task adapter and continue executing the main task. The online multimodal large language model monitor continuously observes the execution effect of the reset skill. Once it confirms that the reset target has been restored (e.g., the cup has been uprighted and is in a stable state), it issues a switching command again, causing the model to unload the reset adapter and reload the main task adapter. At this point, the task execution flow seamlessly transitions from fault recovery back to the subsequent steps of the main task.
[0072] This embodiment constructs a two-layer closed-loop inference system consisting of "high-level multimodal large language model coordination + low-level VLA adapter switching," achieving robust and adaptive autonomous operation of the robot. The high-level online multimodal large language model monitor acts as the system's "brain," capable of real-time environmental perception, accurate diagnosis of out-of-distribution errors, and correct selection and invocation of corresponding reset skills from the expert policy library. This decoupling design of high-level decision-making and low-level execution allows the system to take orderly recovery measures in the face of unforeseen catastrophic failures. Simultaneously, the design of switching adapters rather than retraining the model ensures high performance and low latency during normal task execution, as the main task adapter has undergone reinforcement training with specific task data. More importantly, this "retry-reset" closed-loop mechanism frees the robot from the binary outcome of "either success or failure," enabling it to achieve its goal through a "trial and error-correction" cycle, much like a human.
[0073] Please see Figure 2 , Figure 2 The overall process architecture of the FLARE framework is demonstrated. The FLARE framework mainly includes four core stages: fault collection, offline fault analysis, data augmentation, and closed-loop inference.
[0074] During the fault collection phase, the system executes tasks on a task library based on a pre-trained vision-language-action model, collecting observation data and operational trajectories. In the offline fault analysis phase, a multimodal large language model (MLLM) analyzes the collected fault videos, outputting structured error metadata based on an in-distribution / out-of-distribution error classification system. Depending on the type of the error metadata, the system proceeds to the data augmentation phase: if it's an in-distribution error, a perturbation and bridging augmentation process is executed to generate a retry augmentation dataset; if it's an out-of-distribution error, the system is guided to execute a reset skill data collection process to generate a reset augmentation dataset. Finally, in the closed-loop inference phase, an online multimodal large language model monitor observes the execution process in real time and coordinates adapter switching in the expert policy library based on the observation results.
[0075] Experimental results
[0076] To verify the effectiveness of the method described in this application (hereinafter referred to as "this method"), experimental tests were conducted on nine high-contact robot manipulation tasks on the RoboMimic simulation platform. The experiments compared this method with several existing baseline methods, including: the open-source vision-language-action model OpenVLA, task-conditional policy, subgoal-conditional policy, motion-conditional policy, SubgoalSelf-reflection, Phoenix (a motion-based self-reflection framework), Phoenix-Human (a variant with human correction as the upper bound for performance), and π0.5 (a high-performance vision-language-action backbone model).
[0077] The suffixes "D0" and "D1" after the task name indicate the range of object randomization during scene initialization, with "D1" representing a more difficult task configuration that includes a wider range of object position and pose variations. All experimental results are the average of 50 evaluation rounds for each task, as shown in Table 1.
[0078] Table 1. Comparison of experimental results for 9 operational tasks on the RoboMimic simulation platform.
[0079]
[0080] According to the experimental results in Table 1:
[0081] 1. This method achieved state-of-the-art performance in 8 out of 9 tasks. In the "Threading DO" task, the performance of this method was on par with the best method.
[0082] 2. In complex and challenging "D1" tasks, the performance improvement of this method compared to the backbone model π0.5 is more significant. For example, in the "Coffee Making D1" task, the success rate of this method is 84%, while that of π0.5 is 76%; in the "Three-Piece Assembly D1" task, the success rate of this method is 46%, while that of π0.5 is 32%.
[0083] 3. The average task success rate of this method significantly exceeds that of existing mainstream self-calibration methods (such as Phoenix's 57.8%), demonstrating that the "retry-reset" paradigm introduced in this method has more comprehensive fault recovery capabilities compared to traditional data augmentation and template-based self-reflection methods. Even compared to Phoenix-Human, which relies on manual instruction calibration (as a performance upper bound), this method remains competitive on multiple tasks, further validating the superiority of the described "online multimodal large language model monitor" coordination adapter switching mechanism.
[0084] Furthermore, to verify the effectiveness of the "Reset Skill Learning" module in this method, ablation experiments were conducted on the "Making Coffee" and "Three-Piece Assembly" tasks.
[0085] The experiment compared three configurations: First, "FLARE - No Reset," which only uses the perturbation-bridging enhancement strategy in this method to train the visual-language-action model, removing the skill learning reset process; second, "FLARE (Complete)," which is the complete solution including the retry enhancement process and the skill learning reset process in this method; and third, "FLARE - Prophet," which replaces the generation of skill reset instructions with human feedback as a performance upper bound reference. The experimental results are shown in Table 2.
[0086] Table 2. Ablation experiment results of this method on the resetting skills learning module.
[0087]
[0088] According to the experimental results in Table 2:
[0089] 1. In both the "Make Coffee" and "Assemble Three Pieces" tasks at difficulties "D0" and "D1," the success rate of "FLARE (Complete)" is higher than that of "FLARE - No Reset." For example, in the "Make Coffee D1" task, the success rate of "FLARE (Complete)" is 78%, higher than the 74% of "FLARE - No Reset"; in the "Assemble Three Pieces D1" task, the success rate of "FLARE (Complete)" is 58%, higher than the 54% of "FLARE - No Reset."
[0090] 2. The above results show that introducing the resetting skill learning process can provide the model with additional environmental resilience and significantly improve the task success rate, proving the effectiveness of the resetting skill module.
[0091] 3. FLARE-Prophet achieved the highest success rate across all tasks, indicating that the performance of the "Online Multimodal Large Language Model Monitor" in recognizing reset skills still has room for improvement. It is foreseeable that as the reasoning capabilities of multimodal large language models increase, the performance ceiling achievable by this method will approach the level of FLARE-Prophet.
[0092] Corresponding to the above-described VLA-oriented fault perception autonomous correction and recovery method, the present invention also provides a VLA-oriented fault perception autonomous correction and recovery device. This VLA-oriented fault perception autonomous correction and recovery device includes a unit for executing the above-described VLA-oriented fault perception autonomous correction and recovery method, and can be configured in a terminal or server. Specifically, the VLA-oriented fault perception autonomous correction and recovery device includes:
[0093] The acquisition unit is used to respond to the received trigger command for the pre-trained vision-language-action model, control the vision-language-action model to perform multiple rounds on a specified task, and acquire the execution video and trajectory data generated during the execution process;
[0094] The analysis unit is used to use a multimodal large language model as an offline fault analyzer to analyze the execution video. Based on a preset in-distribution / out-of-distribution error classification system, it outputs structured error metadata for each fault. The error metadata includes error type, reset target, and fault timestamp.
[0095] The retry enhancement unit is used to execute the retry enhancement process if the error type is an in-distribution error: based on the trajectory data, a disturbance-bridging segment is injected between task segments to generate a retry enhancement dataset. The disturbance-bridging segment includes a disturbance action to move the robot arm to an out-of-distribution posture and a bridging action to connect the robot from the disturbed posture to the effective starting posture of the next task segment.
[0096] The reset skill learning unit is used to execute the reset skill learning process if the error type is an out-of-distribution error: extract the environmental state from the trajectory data based on the fault timestamp in the error metadata, collect a preset number of human demonstration data for the reset target, and apply the perturbation-bridging segment to the human demonstration data to enhance it, thereby generating a reset enhancement dataset;
[0097] A training unit is used to train an expert policy library containing a main task adapter and at least one reset skill adapter based on the retry enhancement dataset and the reset enhancement dataset.
[0098] The inference unit is used to observe the execution process of the visual-language-action model in real time by an online multimodal large language model monitor during the inference phase, and coordinate the switching of adapters in the expert policy library based on the observation results to execute the main task or reset the skill.
[0099] In some preferred embodiments, training an expert policy base containing a main task adapter and at least one reset skill adapter based on the retry enhancement dataset and the reset enhancement dataset includes:
[0100] The main task adapter is trained on the shared pre-trained vision-language-action backbone model using a low-rank adaptation technique with the retry augmentation dataset, wherein the main task adapter is invoked via a main task instruction.
[0101] The low-rank adaptation technique is used to train the reset skill adapter on the pre-trained vision-language-action backbone model with the reset augmentation dataset, wherein each reset skill adapter corresponds to a unique reset skill cue word.
[0102] In some preferred embodiments, the step of injecting perturbation-bridging segments between task segments based on the trajectory data to generate a retry enhancement dataset includes:
[0103] Baseline trajectories were synthesized from a library of sub-task segments demonstrated by humans;
[0104] Between the task segments of the baseline trajectory, the perturbation-bridging segment is inserted, wherein the perturbation stage moves the robot arm to an out-of-distribution pose through random motion, and the bridging stage generates a motion trajectory from the perturbed pose to the subsequent task segment through kinematic interpolation.
[0105] The bridging segment is combined with the subsequent task segment to form a new training sequence, which is then added to the retry augmentation dataset.
[0106] In some preferred embodiments, the step of extracting environmental state from the trajectory data based on the fault timestamp in the error metadata, collecting a preset number of human demonstration data for the reset target, and applying the perturbation-bridging segment to the human demonstration data to generate a reset enhancement dataset includes:
[0107] Based on the fault timestamp, the environmental state is extracted from the trajectory data to obtain the error state of the reset target;
[0108] Starting from the error state, a preset number of human demonstration data are collected, and each human demonstration data contains a recovery trajectory that restores the reset target from the error state to the effective state;
[0109] Based on the error groups in the error metadata, the relative orientation within the error groups is preserved when placing the reset target in order to maintain the semantic integrity of the fault.
[0110] The perturbation-bridging segment is applied to the recovered trajectory in the human demonstration data to generate reset trajectories under various robot starting poses and scene layouts;
[0111] Add the reset trajectory to the reset enhancement dataset.
[0112] In some preferred embodiments, the step of having an online multimodal large language model monitor observe the execution process of the vision-language-action model in real time and coordinate adapter switching in the expert policy library based on the observation results to execute the main task or reset the skill includes:
[0113] The online multimodal large language model monitor determines in real time whether out-of-distribution errors occur during the current execution process of the vision-language-action model;
[0114] If no out-of-distribution error occurs, the main task adapter will remain loaded, and the main task will continue to be executed.
[0115] If an out-of-distribution error occurs, the required reset skill is identified by the online multimodal large language model monitor, and the vision-language-action model is instructed to switch to the corresponding reset skill adapter to execute the reset skill;
[0116] Once the online multimodal large language model monitor confirms that the environment has been restored to a valid state, it instructs the vision-language-action model to reload the main task adapter and continue executing the main task.
[0117] In some preferred embodiments, the structured error metadata is in JSON format, and the error type, reset target, error group, and fault timestamp contained in the structured error metadata are all stored in the form of structured fields.
[0118] In some preferred embodiments, the disturbance parameters of the disturbance-bridging segment include a maximum rotation angle and a maximum translation distance, wherein the maximum rotation angle is 30° to 45° and the maximum translation distance is 0.5 m to 0.7 m.
[0119] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned fault perception autonomous correction and recovery device and its various units for VLA can be referred to the corresponding descriptions in the foregoing method embodiments. For the sake of convenience and brevity, these will not be repeated here.
[0120] The aforementioned fault-aware autonomous correction and recovery device for VLA can be implemented as a computer program, which can, for example... Figure 5 It runs on the computer device shown.
[0121] Please see Figure 5 , Figure 5 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.
[0122] The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.
[0123] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it enables the processor 502 to perform a fault-aware autonomous correction and recovery method for VLA.
[0124] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.
[0125] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a fault-aware autonomous correction and recovery method for VLA.
[0126] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the above structure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.
[0127] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of a fault-aware autonomous correction and recovery method for VLA provided in any of the above embodiments.
[0128] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.
[0129] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.
[0130] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the steps of a fault-aware autonomous correction and recovery method for VLA provided in any of the above embodiments.
[0131] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.
[0132] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0133] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.
[0134] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0135] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.
[0136] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0137] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Since these modifications and variations fall within the scope of the claims and their equivalents, this invention also intends to include these modifications and variations.
[0138] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A method for fault-aware autonomous correction recovery for VLA, characterized in that, include: In response to receiving a trigger command for a pre-trained vision-language-action model, the system controls the vision-language-action model to perform multiple rounds on a specified task and collects execution video and trajectory data generated during the execution process. The multimodal large language model is used as an offline fault analyzer to analyze the execution video. Based on the preset in-distribution / out-of-distribution error classification system, structured error metadata is output for each fault. The error metadata includes error type, reset target and fault timestamp. If the error type is an in-distribution error, then the retry enhancement process is executed: based on the trajectory data, a disturbance-bridging segment is injected between task segments to generate a retry enhancement dataset. The disturbance-bridging segment includes a disturbance action that moves the robot arm to an out-of-distribution posture and a bridging action that connects the robot from the disturbed posture to the effective starting posture of the next task segment. If the error type is an out-of-distribution error, then the reset skill learning process is executed: based on the fault timestamp in the error metadata, the environmental state is extracted from the trajectory data, a preset number of human demonstration data for the reset target are collected, and the perturbation-bridging segment is applied to the human demonstration data to enhance it, generating a reset enhancement dataset; Based on the retry enhancement dataset and the reset enhancement dataset, train an expert policy library containing a main task adapter and at least one reset skill adapter; During the inference phase, the execution process of the vision-language-action model is observed in real time by an online multimodal large language model monitor, and the adapter switching in the expert policy library is coordinated based on the observation results to execute the main task or reset the skill.
2. The fault-aware autonomous correction and recovery method for VLA according to claim 1, characterized in that, The process of training an expert policy library containing a main task adapter and at least one reset skill adapter based on the retry enhancement dataset and the reset enhancement dataset includes: The main task adapter is trained on the shared pre-trained vision-language-action backbone model using a low-rank adaptation technique with the retry augmentation dataset, wherein the main task adapter is invoked via a main task instruction. The low-rank adaptation technique is used to train the reset skill adapter on the pre-trained vision-language-action backbone model with the reset augmentation dataset, wherein each reset skill adapter corresponds to a unique reset skill cue word.
3. The VLA oriented fault-aware self-correcting recovery method of claim 1, wherein, Based on the trajectory data, the process of injecting perturbation-bridging segments between task segments to generate a retry enhancement dataset includes: Baseline trajectories were synthesized from a library of sub-task segments demonstrated by humans; Between the task segments of the baseline trajectory, the perturbation-bridging segment is inserted, wherein the perturbation stage moves the robot arm to an out-of-distribution pose through random motion, and the bridging stage generates a motion trajectory from the perturbed pose to the subsequent task segment through kinematic interpolation. The bridging segment is combined with the subsequent task segment to form a new training sequence, which is then added to the retry augmentation dataset.
4. The VLA oriented fault-aware self-correcting recovery method of claim 1, wherein, The environmental state is extracted from the trajectory data based on the fault timestamp in the error metadata, a preset number of human demonstration data for the reset target are collected, and the perturbation-bridging segment is applied to the human demonstration data to enhance it, generating a reset enhancement dataset, including: Based on the fault timestamp, the environmental state is extracted from the trajectory data to obtain the error state of the reset target; Starting from the error state, a preset number of human demonstration data are collected, and each human demonstration data contains a recovery trajectory that restores the reset target from the error state to the effective state; Based on the error groups in the error metadata, the relative orientation within the error groups is preserved when placing the reset target in order to maintain the semantic integrity of the fault. The perturbation-bridging segment is applied to the recovered trajectory in the human demonstration data to generate reset trajectories under various robot starting poses and scene layouts; Add the reset trajectory to the reset enhancement dataset.
5. The VLA oriented, failure-aware, self-correcting recovery method of claim 1, wherein, The process of the visual-language-action model is observed in real time by an online multimodal large language model monitor, and the adapter switching in the expert policy library is coordinated based on the observation results to execute the main task or reset the skill, including: The online multimodal large language model monitor determines in real time whether out-of-distribution errors occur during the current execution process of the vision-language-action model; If no out-of-distribution error occurs, the main task adapter will remain loaded, and the main task will continue to be executed. If an out-of-distribution error occurs, the required reset skill is identified by the online multimodal large language model monitor, and the vision-language-action model is instructed to switch to the corresponding reset skill adapter to execute the reset skill; Once the online multimodal large language model monitor confirms that the environment has been restored to a valid state, it instructs the vision-language-action model to reload the main task adapter and continue executing the main task.
6. The VLA oriented, failure-aware, self-correcting recovery method of claim 1, wherein, The structured error metadata is in JSON format, and the error type, reset target, error group, and fault timestamp contained in the structured error metadata are all stored in the form of structured fields.
7. The VLA oriented, failure-aware, self-correcting recovery method of claim 1, wherein, The disturbance parameters of the disturbance-bridging section include the maximum rotation angle and the maximum translation distance, wherein the maximum rotation angle is 30° to 45° and the maximum translation distance is 0.5 m to 0.7 m.
8. A VLA-oriented fault-aware self-correcting recovery apparatus, characterized in that, Includes a unit for performing the method as described in any one of claims 1-7.
9. A computer device, comprising: The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-7.