Hierarchical freezing and control replay embodied task failure attribution method and device

Abstract

The application discloses a layered freezing and control playback embodied task failure attribution method and device. The method obtains embodied task failure execution data, determines failure anchor points based on task result data and abnormal event records, takes the state corresponding to the failure anchor points as the initial state of playback, calls a world model to construct a control playback track and a target dimension counterfactual track, applies directional disturbance in a single dimension of a perception dimension, a planning dimension or a control dimension, and performs layered freezing on downstream modules. Then, according to the track difference, a target dimension difference index is calculated, the main cause dimension, the secondary cause dimension and the causal attribution confidence are determined, and a structured failure attribution result is output. The application can improve the embodied task failure positioning efficiency, the attribution explainability and the repair closed loop efficiency.

Description

Technical Field

[0001] This invention relates to the fields of embodied intelligence, robot task execution monitoring, world model-assisted playback analysis, and fault diagnosis, and particularly to an embodied task failure attribution method and apparatus using layered freezing and comparative playback. Background Technology

[0002] With the development of embodied intelligence technology, robotic systems have expanded from executing single actions to performing long-chain tasks across perception, planning, and control. In scenarios such as warehousing, indoor delivery, industrial assembly, and service collaboration, embodied agents typically need to generate sub-target sequences and low-level control commands based on image observation, depth observation, pose estimation results, and task instructions, and continuously perform grasping, transporting, obstacle avoidance, and placement actions in dynamic environments. Because perception errors, planning violations, control deviations, and environmental disturbances propagate through the execution chain, a single task failure is often not a single point of failure, but rather a comprehensive result of multiple coupled modules.

[0003] Existing methods for embodied task failure analysis typically include log rule-based fault localization, simulation playback-based execution verification, coarse-grained capability label-based failure classification, and text-based counterfactual repair analysis. While these methods provide some auxiliary information, they still have the following shortcomings: First, it is difficult to separate the dominant failure sources in the perception, planning, and control dimensions from the coupling links; second, directly replaying or comparing the modified trajectory with the actual failure trajectory can easily misjudge downstream chain changes caused by upstream disturbances as failures of the downstream modules themselves; third, pure text-based attribution results are difficult to directly map to image observations, pose states, control feedback, and physical execution results in the embodied system, and are also difficult to directly integrate into the parameter correction, rule supplementation, sample re-collection, and regression verification closed loop. Therefore, an embodied task failure attribution method is needed that can perform single-dimensional directional perturbation, hierarchical freezing, comparative replay, and structured result output around the failure anchor point. Summary of the Invention

[0004] The purpose of this invention is to provide a method and apparatus for attributing embodied task failures by layering freezing and comparative replay, in order to solve the problems of difficulty in isolating the main cause dimension, difficulty in verifying attribution evidence, and difficulty in directly implementing the repair loop in the existing embodied task failure analysis process.

[0005] According to a first aspect of the embodiments of this application, a method for attributing embodied task failures through layered freezing and comparative replay is provided, comprising: Acquire embodied task failure execution data, which includes at least task instructions, environmental observation data, robot state data, planning sequence, control sequence, and task result data; The task completion criteria are determined according to the task instructions. The execution time of the environmental observation data, robot state data, planning sequence, control sequence and task result data is aligned and anomaly event records are generated. The failure anchor point is determined based on the task completion criteria, the task result data and the anomaly event records. Using the environmental observation data and robot state data corresponding to the failure anchor point as the initial state for playback, the world model is called to construct the replay trajectory and at least one counterfactual trajectory in the target dimension, and a directional perturbation is applied to the target dimension and a layered freeze is performed on the downstream module. The target dimension includes the perception dimension, the planning dimension and the control dimension. By comparing the differences between the target dimension counterfactual trajectory and the control replay trajectory in terms of state evolution, action execution results, and task completion results, the target dimension difference index and supporting difference evidence are obtained. Based on the target dimension difference index, the primary cause dimension, secondary cause dimension, and causal attribution confidence level of failure are determined. Structured failure attribution results are generated based on the primary failure dimension, the secondary failure dimension, the causal attribution confidence level, and supporting differential evidence.

[0006] According to a second aspect of the embodiments of this application, a layered freeze and comparative replay embodied task failure attribution apparatus is provided, comprising: The data acquisition module is used to acquire the failed execution data of the embodied task. The failed execution data of the embodied task includes at least the task instructions, environmental observation data, robot state data, planning sequence, control sequence and task result data. Anchor point determination module is used to determine task completion criteria based on the task instructions, align the execution time of the environmental observation data, robot state data, planning sequence, control sequence and task result data and generate abnormal event records, and determine failure anchor points based on the task completion criteria, the task result data and abnormal event records; The trajectory construction module is used to use the environmental observation data and robot state data corresponding to the failure anchor point as the initial state for playback, call the world model to construct the replay trajectory and at least one counterfactual trajectory in the target dimension, apply directional perturbation on the target dimension and perform hierarchical freezing on the downstream modules, wherein the target dimension includes the perception dimension, the planning dimension and the control dimension. The difference calculation module is used to compare the differences between the target dimension counterfactual trajectory and the control replay trajectory in terms of state evolution, action execution results and task completion results, and to obtain the target dimension difference index and supporting difference evidence. The attribution determination module is used to determine the primary cause dimension, secondary cause dimension, and causal attribution confidence level of failure based on the target dimension difference index. The result generation module is used to generate structured failure attribution results based on the primary failure dimension, the secondary failure dimension, the causal attribution confidence level, and supporting differential evidence.

[0007] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to perform the method as described in the first aspect.

[0008] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which computer instructions are stored, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0009] Compared with the prior art, the embodiments of the present invention have at least the following beneficial effects: (1) The main cause dimension of failure can be separated: By applying directional perturbation to a single target dimension around the failure anchor point and performing hierarchical freezing on downstream modules, the problem of difficulty in isolating the main cause dimension in multi-module coupled scenarios is overcome, thereby enabling more accurate differentiation between perception failure, planning failure and control failure.

[0010] (2) The replay results are closer to the actual execution chain: By using the state corresponding to the failure anchor point as the initial state of the replay and constructing a comparison replay trajectory, the problem of the superposition of timing noise and initial state noise caused by replaying directly from any time is overcome, thereby improving the consistency between the attribution analysis and the actual execution chain.

[0011] (3) Attribution evidence is easier to verify: By comparing the differences in state, action and result between the counterfactual trajectory of the target dimension and the control replay trajectory, the problem that the attribution process only outputs conclusions but lacks supporting evidence is overcome, thus providing engineers with a traceable differential evidence chain.

[0012] (4) Improved implementation of repair actions: By outputting structured failure attribution results that include primary cause dimension, secondary cause dimension, key evidence and repair suggestions, the problem that unstructured failure descriptions are difficult to directly translate into engineering repair actions is overcome, thereby directly supporting parameter correction, rule supplementation, sample re-collection and regression verification.

[0013] (5) Optimized closed loop is easier to sustain: By organizing failure anchors, difference indicators and attribution results into a unified structured result, the problem of failure analysis results being difficult to accumulate and reuse in the long term is overcome, so as to serve subsequent difficult case archiving, attribution case reuse and regression test updates. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0015] Figure 1 This is a schematic diagram of the overall process of an embodied task failure attribution method based on layered freezing and comparative replay, according to an exemplary embodiment. Figure 2 This is a structural block diagram illustrating an embodied task failure attribution device with layered freezing and contrast replay according to an exemplary embodiment; Figure 3 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0016] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0017] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0018] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0019] For ease of explanation, this embodiment uses the scenario of "a warehouse gripping robot picking up sample boxes from a shelf and placing them into a turnover box" as an example, but it is not limited to this. During execution, the warehouse gripping robot needs to read images from the camera directly in front of the shelf and the end-effector camera, identify the position of the sample box, plan the robotic arm's gripping path, control the gripper to close, and place the sample box into the turnover box. If failures occur, such as "sample box not picked up," "dropped after gripping," "sample box colliding with the shelf," or "sample box not placed in the turnover box," then a cause analysis of the failure is necessary.

[0020] Figure 1 This is a flowchart illustrating an embodied task failure attribution method based on layered freeze and contrastive replay, according to an exemplary embodiment. (See reference) Figure 1 The embodiment of the present invention provides a method for attributing embodied task failures through layered freezing and comparative replay, which may include the following steps: S1: Obtain data on failed execution of the embodied task; Specifically, first, obtain the execution data set corresponding to a failed execution process of the embodied task. This execution data set is denoted as:

[0021] in, Indicates time Environmental observation data, which includes at least RGB images, depth images, and target detection results; Indicates time The robot state data includes at least the chassis pose, robotic arm joint angles, end effector pose, and gripper opening / closing state. Indicates time The planning sequence includes at least sub-objective identifiers, action order, and constraints. Indicates time The control sequence includes at least velocity, acceleration, gripping force threshold, and trajectory tracking parameters; Indicates time The task result data includes at least the target grabbing result, target transport result, target placement result, and collision marker.

[0022] In a warehouse picking robot scenario, G can correspond to the task instruction of "picking up sample boxes from the shelf and placing them into a turnover box". It can correspond to images from both the camera directly in front of the shelf and the camera at the end of the shelf. It can correspond to the six joint angles of the robotic arm and the opening and closing angles of the gripper. This can correspond to the sub-target sequence of "approaching the shelf—aligning the sample box—closing the grippers—lifting the sample box—moving to the turnover box—releasing the sample box". It can correspond to the chassis linear velocity, the robotic arm end effector velocity, and the gripper force threshold. It can correspond to result states such as "successful capture", "failed capture", "collision occurred", and "placement offset". This setting can unify the task objective, observation chain, decision chain and action chain in a failed execution into the same data framework, which is convenient for subsequent causal tracing around the same scenario.

[0023] S2: Determine the task completion criteria based on the task instructions, align the execution time of environmental observation data, robot state data, planning sequence, control sequence, and task result data, and generate anomaly event records. Determine the failure anchor point based on the task completion criteria, task result data, and anomaly event records. This step includes the following sub-steps: S21: Align the execution time of the environmental observation data, robot state data, planning sequence, control sequence, and task result data to obtain an execution segment sequence sorted by time. Specifically, a unified clock alignment is performed on the image stream, depth stream, state stream, and control stream acquired at different sampling frequencies. This ensures that each execution moment corresponds to unique environmental observation data, robot state data, planning sequence, control sequence, and task result data, resulting in a sequence of execution segments ordered by time. This setup avoids timing misalignment judgments where "perception changes occur after planning changes" or "control deviations occur before perception deviations," providing a unified time reference for determining failure anchor points.

[0024] S22: For each execution segment in the execution segment sequence, calculate the anomaly score at each time step based on the perception deviation, planning violation, and control deviation, and generate an anomaly event record based on the anomaly score; Specifically, for each moment Calculate anomaly scores :

[0025] in, Indicates time The perceptual bias can be obtained by combining the change in target object category, the target object pose deviation, and the depth estimation deviation. Indicates time The number of planning violations can be obtained from the omission of sub-objectives, constraint violations, and incorrect step sequencing. Indicates time The control deviation can be obtained from the trajectory tracking error, gripper closure error, and end-effector attitude deviation. , as well as This represents the weighting coefficient.

[0026] Furthermore, execution segments whose anomaly scores reach the anomaly threshold are recorded as anomaly event records. Each anomaly event record includes at least the time of the anomaly occurrence, the anomaly type, and the corresponding execution segment. This configuration ensures that anomaly event records originate from a time-aligned sequence of execution segments, preventing a lack of data sources during subsequent failure anchor point determination.

[0027] S23: In the execution segment sequence, according to the abnormal event record, the execution segment corresponding to the moment when the abnormal score first reaches the abnormal threshold and the task result data is characterized as the task not being completed based on the task completion criterion is determined as the failure anchor point; Specifically, task completion criteria are determined based on task instructions. In a warehouse gripping robot scenario, the task instruction could be "grab a sample box from the shelf and place it in the turnover box." Corresponding task completion criteria could include successful sample box grabbing, no sample box falling during handling, sample box being placed in the designated area of ​​the turnover box, and no collision occurring during execution. This setup provides a clear basis for determining whether task results indicate task incompleteness. The system also considers the first occurrence of an anomaly threshold in the anomaly score. Furthermore, the task result data is based on the task completion criteria to characterize the moment when the task was not completed, thus determining the failure anchor point:

[0028] in, Indicates the moment of failure anchor point, The task result data indicates that the task was not completed based on the task completion criteria.

[0029] In warehouse retrieval robot scenarios, if the system is constantly... If a combined anomaly occurs for the first time, indicating "excessive target sample box position offset, unupdated planned grasping points, and failure to envelop the target after gripper closure," and the task results data shows that the sample box was not successfully grasped, then... The corresponding execution segment is identified as the failure anchor point. This setting allows the starting moment when "the task becomes irreversible" to be extracted from the entire failure trajectory, rather than simply using the task end moment, thereby narrowing the search scope for subsequent counterfactual analysis.

[0030] S3: Using the environmental state and robot state data corresponding to the failure anchor point as the initial state for replay, the world model is invoked to construct the replay trajectory and at least one counterfactual trajectory in the target dimension. Oriented perturbations are applied to the target dimension, and hierarchical freezing is performed on downstream modules. The target dimension includes the perception dimension, planning dimension, and control dimension. This step includes the following sub-steps: S31: Determine the environmental state and robot state at the moment before the failure anchor point as the initial state for playback; Specifically, the environmental state and robot state at the moment preceding the failure anchor point are used as the initial state for playback. This setting reduces the interference of state changes after failure on playback analysis, allowing the playback process to begin from the state where failure is about to occur but has not yet fully unfolded.

[0031] S32: Construct a replay trajectory by calling the world model from the initial replay state; Specifically, the environmental state and robot state corresponding to the moment before the failure anchor point are used as the initial state for replay, and the world model is invoked to perform a comparative replay. The comparative replay trajectory is denoted as:

[0032] in, This indicates the replay trajectory. Representing a world model, This indicates the execution state at the moment preceding the failure anchor. This represents the set of conditions in the actual failure trajectory. The set of conditions includes at least the actual perception output, the actual planning output, and the actual control output.

[0033] In warehouse gripping robot scenarios, the replay trajectory corresponds to "re-executing the action chain from aligning the sample box to closing the gripper, while keeping the state before failure unchanged and reusing the task context conditions of the real failed trajectory." This setting can provide a unified comparison baseline for all subsequent counterfactual trajectories, reducing the impact of initial posture differences and instantaneous environmental noise on the attribution results.

[0034] S33: Select a target dimension from the perception dimension, planning dimension, and control dimension, and apply a directional perturbation to the target dimension; Specifically, a target dimension is selected at the failure anchor point, and a directional perturbation is applied to that target dimension. When the target dimension is a perception dimension, the directional perturbation includes at least the target object category perturbation, the target object pose perturbation, the target object depth estimation perturbation, and key environmental attribute perturbation. In a warehouse gripping robot scenario, the sample box category can be corrected from "cardboard box" to "plastic box", the sample box center pose can be finely adjusted by 15 mm towards the gripper center, the depth estimation value can be corrected to the actual depth at the front edge of the shelf, and "no obstruction on the top of the sample box" can be corrected to "an obstruction strip exists on the top of the sample box".

[0035] When the target dimension is the planning dimension, the orientation perturbation includes at least the sub-target sequence perturbation, the action sequence relationship perturbation, the action precondition perturbation, and the obstacle avoidance constraint perturbation. In the warehouse gripping robot scenario, the "re-align sample box center" action can be inserted before "closing the gripper", the "lifting sample box" sequence can be adjusted to after "confirming the gripper envelope target", and the constraint of "maintaining a safe distance between the end effector and the shelf column" can be strengthened.

[0036] When the target dimension is the control dimension, the orientation perturbation includes at least the velocity parameter perturbation, acceleration parameter perturbation, gripping force threshold perturbation, gripper closure time perturbation, and path tracking gain perturbation. In the scenario of warehouse gripping robots, the end-effector approach speed can be reduced from 0.25 m / s to 0.18 m / s, the gripper closure time can be delayed by 80 milliseconds, and the gripping force threshold can be adjusted from 4 Newtons to 5.5 Newtons.

[0037] S34: In each counterfactual inference, select only one target dimension to perform perturbation, keep the input of the other dimensions unchanged, call the world model to construct the target dimension counterfactual trajectory corresponding to the target dimension, and perform hierarchical freezing on the downstream modules; Specifically, in each counterfactual inference, only one target dimension is perturbed, while keeping the inputs of the other dimensions unchanged. Under the layered freeze constraint, the world model is called to perform forward inference to obtain the counterfactual trajectory of the target dimension. The counterfactual trajectory of each target dimension is denoted as:

[0038] in, Indicates the first The counterfactual trajectory of each target dimension. Indicates the first The set of counterfactual conditions corresponding to each objective dimension. This setup allows for separate answers to "What would happen if only the perceived outcome were corrected?", "What would happen if only the planning steps were corrected?", and "What would happen if only the control parameters were corrected?", thus making the attribution evidence for different dimensions separable and comparable.

[0039] Furthermore, the layered freezing of downstream modules includes: if the current analysis target is the perception dimension, then the planning module and control module are considered downstream modules, and the planning output and control output in the actual failure trajectory are frozen; if the current analysis target is the planning dimension, then the control module is considered a downstream module, and the control output in the actual failure trajectory is frozen; if the current analysis target is the control dimension, then the perception output and planning output in the actual failure trajectory are reused. This setting can decouple the upstream dimension correction and the downstream dimension chain reaction as much as possible, avoiding misjudging "normal chain changes brought about by upstream correction" as "downstream module failure".

[0040] Furthermore, if upstream perturbations cause the downstream output in the actual failure trajectory to no longer satisfy the physical execution constraints, the world model is invoked to generate a compatible output, ensuring that the difference between the compatible output and the actual failure trajectory output does not exceed a compatibility threshold. The compatible output is denoted as:

[0041] in, Indicates compatible output. This represents the downstream output corresponding to the actual failure trajectory. This setting ensures that the downstream output remains "as close as possible to the actual failure trajectory" even when it must be corrected, preventing the compatible output from deviating too much and introducing new spurious causal factors.

[0042] S4: Compare the differences between the target dimension counterfactual trajectory and the control replay trajectory in terms of state evolution, action execution results, and task completion results to obtain the target dimension difference index and supporting difference evidence; this step includes the following sub-steps: S41: Compare the differences in state evolution between the target dimension counterfactual trajectory and the control replay trajectory to obtain the state difference term; Specifically, after constructing counterfactual trajectories for the perception, planning, and control dimensions, the state evolution of each counterfactual trajectory is compared with that of the control replay trajectory at the same time point. State difference terms are then extracted to characterize the degree of change between the environmental state and the robot's state. In a warehouse grasping robot scenario, the comparison can include changes in the sample box position and the robot's end-effector pose. This setup allows for the identification of the impact of perturbations in different target dimensions on intermediate state evolution, avoiding judgments based solely on the state at the end of the task.

[0043] S42: Compare the difference between the target dimension counterfactual trajectory and the control replay trajectory in terms of action execution results to obtain the action difference term; Specifically, the counterfactual trajectory of each target dimension is compared with the action execution results of the control replay trajectory at the same time, and action difference terms are extracted to characterize the degree of change in control and execution actions. In the scenario of warehouse gripping robots, the comparison can include the gripper closing time, the gripper closing action, and the gripper's envelope result on the sample box. This setup can determine whether the state changes caused by perturbations in different target dimensions are further transmitted to the action execution stage, so that the attribution analysis covers differences at the action level.

[0044] S43: Compare the differences between the counterfactual trajectory of the target dimension and the control replay trajectory in terms of task completion results to obtain the result difference term; Specifically, the counterfactual trajectory of each target dimension is compared with the control replay trajectory at the same time to extract result difference terms to characterize the success rate of grasping, collision status, and the degree of change in placement results. In the scenario of warehouse grasping robots, the comparison content can include whether the sample box was successfully grasped, whether a shelf collision occurred during the execution process, and whether the sample box was placed in the designated area of ​​the turnover box. This setting allows the result difference terms, along with the state difference terms and action difference terms, to cover the entire process of the task execution chain, avoiding attribution based solely on the single result of success or failure.

[0045] S44: Based on the state difference term, action difference term, and result difference term, and combined with the time weight from the failure anchor point to the trajectory termination time, calculate the target dimension difference index corresponding to each target dimension, and generate supporting difference evidence based on the state difference term, action difference term, and result difference term. Specifically, no. The target dimension difference index corresponding to each target dimension is calculated according to the following formula:

[0046] in, Indicates the first The target dimension difference index corresponding to each target dimension Indicates the moment of failure anchor point, Indicates the time when the trajectory ends. Indicates the weight of time. Indicates the first Each objective dimension at time State difference term, Indicates the first Each objective dimension at time Action difference term, Indicates the first Each objective dimension at time The result is the difference term.

[0047] Furthermore, key changes in the state difference term, action difference term, and result difference term that support the attribution conclusion are identified as supporting difference evidence. The supporting difference evidence includes at least key state changes, key action changes, and key result changes.

[0048] In a warehouse gripping robot scenario, if a perturbation in the perception dimension allows for successful gripping of a sample box without collision with the shelf, while perturbations in the planning and control dimensions do not significantly improve the result, then the target dimension difference index corresponding to the perception dimension is minimized. Supporting evidence for this difference could include "successful gripper envelope and shelf collision disappearance after correcting the sample box pose." This approach quantifies the "substantial impact of a correction in a certain dimension on the task execution chain" into a unified indicator, facilitating direct comparisons between different dimensions.

[0049] S5: Determine the primary cause dimension, secondary cause dimension, and causal attribution confidence level based on the difference indicators of each target dimension; this step includes the following sub-steps: S51: Normalize the target dimension difference indices corresponding to the perception dimension, planning dimension, and control dimension; Specifically, using the minimum and maximum values ​​of the target dimension difference indices corresponding to the perception, planning, and control dimensions as normalization boundaries, each target dimension difference index is mapped to the interval between 0 and 1, resulting in normalized target dimension difference indices. In the warehouse gripping robot scenario, the same normalization boundary is used for the target dimension difference indices corresponding to correcting sample box pose, adjusting gripping planning, and adjusting gripper control. When the target dimension difference indices are the same, the normalized target dimension difference indices are set to the same value. This setting can eliminate the impact of different magnitudes of different target dimension difference indices on subsequent confidence calculations.

[0050] S52: Calculate the causal attribution confidence of each dimension based on the normalized target dimension difference index; Specifically, the causal attribution confidence scores for each dimension are calculated based on the normalized target dimension difference index:

[0051] in, Indicates the first The causal attribution confidence scores for each objective dimension. Indicates the first The normalized target dimension difference index corresponding to each target dimension Indicates the target dimension index. The values ​​are 1, 2, and 3; the causal attribution confidence scores for each dimension are between 0 and 1, and the sum of the three is 1. In the scenario of warehouse grasping robots, the smaller the normalized target dimension difference index, the greater the corresponding causal attribution confidence score. This setting can transform the difference indices of different target dimensions into sortable and comparable quantitative results.

[0052] S53: The target dimension with the highest confidence in causal attribution among all dimensions is determined as the primary cause dimension of failure, and the target dimension with the second highest confidence in causal attribution is determined as the secondary cause dimension. Specifically, the perception, planning, and control dimensions are ranked from highest to lowest causal attribution confidence. The target dimension with the highest causal attribution confidence is identified as the primary cause of failure, and the target dimension with the second highest causal attribution confidence is identified as the secondary cause. If the difference in causal attribution confidence between the primary and secondary cause dimensions is less than a conflict threshold, the improvement magnitude of the counterfactual trajectories of the two target dimensions at key result nodes is further compared. Key result nodes include at least the target grasping completion time, the target handling stabilization time, and the target placement completion time. The target dimension with the larger improvement magnitude is identified as the primary cause of failure. In warehouse grasping robot scenarios, the improvement degree of different target dimension perturbations on the sample box grasping, handling, and placement results can be compared. This setup avoids misjudging accidental sensitive dimensions as primary cause of failure dimensions, makes the primary cause ranking results more stable, and facilitates engineers in identifying priority repair targets.

[0053] S6: Generate structured failure attribution results based on the primary cause dimension, secondary cause dimension, causal attribution confidence level, and supporting differential evidence; this step includes the following sub-steps: S61: Generate evidence summaries based on supporting differential evidence; Specifically, key difference segments from the replay trajectory and the counterfactual trajectories of each target dimension are extracted as evidence summaries. These evidence summaries include at least the target dimension identifier, perturbation content, key state changes, key action changes, and key result changes. The supporting difference evidence originates from state difference terms, action difference terms, and result difference terms, and is used to support the determination of the primary and secondary failure dimensions.

[0054] S62: Based on the evidence summary, the primary cause dimension of failure, the secondary cause dimension, and the causal attribution confidence, construct the primary cause dimension hypothesis, the secondary cause dimension hypothesis, and the exclusion dimension hypothesis, and perform consistency verification to obtain the hypothesis verification results. Specifically, hypotheses are constructed around the primary, secondary, and exclusionary dimensions of failure. These hypotheses are then validated using difference indicators for the target dimension, causal attribution confidence levels, and supporting difference evidence to obtain the hypothesis verification results. This approach avoids structured failure attribution results that only output conclusions without verifiable supporting evidence.

[0055] S63: Based on the hypothesis testing results, the primary cause dimension of failure, the secondary cause dimension, the causal attribution confidence level, and the supporting differential evidence, output the structured failure attribution results; Specifically, a structured failure attribution report is output based on the hypothesis testing results. The structured failure attribution report includes at least the failure task identifier, failure anchor point time, primary cause dimension, secondary cause dimension, causal attribution confidence level, key differential evidence, remediation objects, remediation actions, and validation recommendations.

[0056] In a warehouse gripping robot scenario, if the primary cause dimension is determined to be the perception dimension, the structured failure attribution result can be written as: "Failure Task Identifier: Task 20260628-017; Failure Anchor Time: 37; Primary Cause Dimension: Perception Dimension; Secondary Cause Dimension: Control Dimension; Primary Cause Confidence: 0.62; Key Differential Evidence: After correcting the sample box pose, the gripper envelope was successful and the shelf collision disappeared; Repair Object: Target object pose estimation model; Repair Action: Supplement shelf edge occlusion scene samples and adjust pose regression loss weights; Validation Suggestion: Replay 50 sets of edge occlusion samples and compare the gripping success rate." This setting allows the causal attribution conclusions to be directly converted into a result format that can be consumed by the engineering team.

[0057] Furthermore, when the primary cause of failure is the perception dimension, the repair objects should at least include the object detection model, pose estimation model, and occlusion recognition rules; when the primary cause of failure is the planning dimension, the repair objects should at least include sub-object decomposition rules, step sequencer, and constraint checker; when the primary cause of failure is the control dimension, the repair objects should at least include velocity parameters, grasping force threshold, and path tracking gain. After generating repair actions based on the primary cause of failure, verification suggestions are then generated based on similar failure scenarios. This setup can further clarify "where the error is" into "what needs to be changed and how to verify it," allowing the failure attribution results to directly enter the parameter correction, rule supplementation, and regression verification closed loop.

[0058] Corresponding to the aforementioned embodiment of the embodied task failure attribution method of layered freezing and comparative replay, this application also provides an embodiment of the embodied task failure attribution device of layered freezing and comparative replay.

[0059] Figure 2 This is a structural block diagram illustrating an embodied task failure attribution device with layered freezing and contrast replay according to an exemplary embodiment. (Reference) Figure 2 The device may include: Data acquisition module 1 is used to acquire embodied task failure execution data, which includes at least task instructions, environmental observation data, robot state data, planning sequence, control sequence and task result data; Anchor point determination module 2 is used to determine the task completion criteria according to the task instructions, align the execution time of the environmental observation data, robot state data, planning sequence, control sequence and task result data and generate abnormal event records, and determine the failure anchor point based on the task completion criteria, the task result data and the abnormal event records; The trajectory construction module 3 is used to use the environmental state and robot state data corresponding to the failure anchor point as the initial state for playback, call the world model to construct the replay trajectory and at least one counterfactual trajectory in the target dimension, apply directional perturbation to the target dimension and perform hierarchical freezing on the downstream modules. The target dimension includes the perception dimension, planning dimension and control dimension. The difference calculation module 4 is used to compare the differences between the target dimension counterfactual trajectory and the control replay trajectory in terms of state evolution, action execution results and task completion results, and to obtain the target dimension difference index and supporting difference evidence. Attribution determination module 5 is used to determine the primary cause dimension, secondary cause dimension, and causal attribution confidence level based on the target dimension difference index; The result generation module 6 is used to generate structured failure attribution results based on the primary failure dimension, the secondary failure dimension, the causal attribution confidence level, and supporting differential evidence.

[0060] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0061] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0062] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the embodied task failure attribution method of hierarchical freeze and contrast replay as described above. Figure 3 The diagram shown is a hardware structure diagram of any data processing-capable device where a layered freezing and comparative replay-based task failure attribution device provided in an embodiment of the present invention is located. (Except for...) Figure 3 In addition to the processor, network interface, memory, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0063] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the embodied task failure attribution method of layered freezing and comparative replay as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0064] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only.

[0065] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for attributing embodied task failures through layered freezing and comparative replay, characterized in that, include: Acquire embodied task failure execution data, which includes at least task instructions, environmental observation data, robot state data, planning sequence, control sequence, and task result data; The task completion criteria are determined according to the task instructions. The execution time of the environmental observation data, robot state data, planning sequence, control sequence and task result data is aligned and anomaly event records are generated. The failure anchor point is determined based on the task completion criteria, the task result data and the anomaly event records. Using the environmental observation data and robot state data corresponding to the failure anchor point as the initial state for playback, the world model is called to construct the replay trajectory and at least one counterfactual trajectory in the target dimension, and a directional perturbation is applied to the target dimension and a layered freeze is performed on the downstream module. The target dimension includes the perception dimension, the planning dimension and the control dimension. By comparing the differences between the target dimension counterfactual trajectory and the control replay trajectory in terms of state evolution, action execution results, and task completion results, the target dimension difference index and supporting difference evidence are obtained. Based on the target dimension difference index, the primary cause dimension, secondary cause dimension, and causal attribution confidence level of failure are determined. Structured failure attribution results are generated based on the primary failure dimension, the secondary failure dimension, the causal attribution confidence level, and supporting differential evidence.

2. The method according to claim 1, characterized in that, Based on the task instructions, a task completion criterion is determined. The environmental observation data, robot state data, planning sequence, control sequence, and task result data are time-aligned, and anomaly event records are generated. Based on the task completion criterion, the task result data, and the anomaly event records, failure anchor points are determined, including: The environmental observation data, robot state data, planning sequence, control sequence, and task result data are time-aligned to obtain a sequence of execution segments ordered by time. For each execution segment in the execution segment sequence, an anomaly score is calculated at each time step based on the perception deviation, planning violation, and control deviation, and an anomaly event record is generated based on the anomaly score. In the sequence of execution segments, based on the abnormal event records, the execution segment corresponding to the moment when the abnormal score first reaches the abnormal threshold and the task result data is characterized as not being completed based on the task completion criterion is determined as the failure anchor point.

3. The method according to claim 1, characterized in that, Using the environmental observation data and robot state data corresponding to the failed anchor point as the initial state for playback, the world model is invoked to construct a replay trajectory and at least one counterfactual trajectory in the target dimension. A directional perturbation is applied to the target dimension, and a layered freeze is performed on the downstream modules, including: The environmental observation data and robot state data at the moment before the failed anchor point are determined as the initial state for playback. The replay initial state is used to construct a comparative replay trajectory by calling the world model; Choose a target dimension from the perception dimension, planning dimension, and control dimension, and apply directional perturbations to the target dimension; wherein, when the target dimension is the perception dimension, perturbations are applied to the target object category, target object pose, target object depth estimate, and key environmental attributes; when the target dimension is the planning dimension, perturbations are applied to the sub-target sequence, step sequence relationship, action preconditions, and obstacle avoidance constraints; when the target dimension is the control dimension, perturbations are applied to the velocity parameters, acceleration parameters, gripping force threshold, gripper closure time, and path tracking gain. In each counterfactual simulation, only one target dimension is selected for perturbation, while keeping the inputs of the other dimensions unchanged. The world model is then invoked to construct the counterfactual trajectory of the target dimension, and a hierarchical freeze is performed on the downstream modules.

4. The method according to claim 3, characterized in that, The layered freezing of downstream modules includes: When the target dimension is the perception dimension, freeze the output of the planning module and the control module in the actual failure trajectory; When the target dimension is the planning dimension, freeze the output of the control module in the actual failure trajectory; When the target dimension is the control dimension, the perception output and planning output in the real failure trajectory are reused; When the frozen output cannot be directly executed due to upstream disturbances, the world model is invoked to generate a compatible output that satisfies the minimum change principle. The minimum change principle is used to constrain the difference between the compatible output and the output corresponding to the actual failure trajectory to not exceed a preset compatibility threshold.

5. The method according to claim 1, characterized in that, By comparing the differences between the target dimension counterfactual trajectory and the control replay trajectory in terms of state evolution, action execution results, and task completion results, the target dimension difference index and supporting difference evidence are obtained, including: By comparing the differences in state evolution between the target dimension counterfactual trajectory and the control replay trajectory, a state difference term is obtained; By comparing the difference in action execution results between the target dimension counterfactual trajectory and the control replay trajectory, action difference terms are obtained; By comparing the difference between the target dimension counterfactual trajectory and the control replay trajectory in terms of task completion results, the result difference term is obtained; Based on the state difference term, the action difference term, and the result difference term, and combined with the time weight from the failure anchor point to the trajectory termination time, the target dimension difference index corresponding to each target dimension is calculated, and supporting difference evidence is generated according to the state difference term, the action difference term, and the result difference term.

6. The method according to claim 1, characterized in that, Based on the target dimension difference index, the primary cause dimension, secondary cause dimension, and causal attribution confidence level of failure are determined, including: Normalize the target dimension difference indicators corresponding to the perception dimension, planning dimension, and control dimension; Calculate the causal attribution confidence of each dimension based on the normalized target dimension difference index; The target dimension with the highest confidence in causal attribution among all dimensions is identified as the primary cause dimension of failure, and the target dimension with the second highest confidence in causal attribution is identified as the secondary cause dimension.

7. The method according to claim 1, characterized in that, Based on the primary failure dimension, the secondary failure dimension, the causal attribution confidence level, and supporting differential evidence, a structured failure attribution result is generated, including: A summary of evidence is generated based on the supporting differential evidence; Based on the evidence summary, the primary cause dimension of failure, the secondary cause dimension, and the causal attribution confidence, the primary cause dimension hypothesis, the secondary cause dimension hypothesis, and the exclusion dimension hypothesis are constructed, and a consistency check is performed to obtain the hypothesis verification results. Based on the hypothesis verification results, the primary cause dimension of failure, the secondary cause dimension, the causal attribution confidence level, and the supporting differential evidence, a structured failure attribution result is output. The structured failure attribution result includes at least the failure task identifier, failure anchor time, primary cause dimension, secondary cause dimension, causal attribution confidence level, key differential evidence, repair object, repair action, and verification suggestions.

8. A device for attributing embodied task failures through layered freezing and comparative replay, characterized in that, include: The data acquisition module is used to acquire the failed execution data of the embodied task. The failed execution data of the embodied task includes at least the task instructions, environmental observation data, robot state data, planning sequence, control sequence and task result data. Anchor point determination module is used to determine task completion criteria based on the task instructions, align the execution time of the environmental observation data, robot state data, planning sequence, control sequence and task result data and generate abnormal event records, and determine failure anchor points based on the task completion criteria, the task result data and abnormal event records; The trajectory construction module is used to use the environmental observation data and robot state data corresponding to the failure anchor point as the initial state for playback, call the world model to construct the replay trajectory and at least one counterfactual trajectory in the target dimension, apply directional perturbation on the target dimension and perform hierarchical freezing on the downstream modules, wherein the target dimension includes the perception dimension, the planning dimension and the control dimension. The difference calculation module is used to compare the differences between the target dimension counterfactual trajectory and the control replay trajectory in terms of state evolution, action execution results and task completion results, and to obtain the target dimension difference index and supporting difference evidence. The attribution determination module is used to determine the primary cause dimension, secondary cause dimension, and causal attribution confidence level of failure based on the target dimension difference index. The result generation module is used to generate structured failure attribution results based on the primary failure dimension, the secondary failure dimension, the causal attribution confidence level, and supporting differential evidence.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 7.

Images