Methods, apparatus, equipment, media and program products for evaluating embodied robots
By collecting data related to embodied robot tasks, determining multi-dimensional scoring vectors and failure reason labels, the problems of single evaluation dimensions and cumbersome evaluation of generalization ability in existing technologies are solved, realizing a simplified evaluation of multi-dimensional assessment and generalization ability of embodied robots.
Patent Information
- Application Number
- CN202611423433.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-14
- Publication Date
- 2026-11-10
AI Technical Summary
Existing technologies for evaluating embodied intelligent robots rely on a limited set of evaluation dimensions, making it difficult to distinguish between specific reasons for failure and types of insufficient transfer. Furthermore, the evaluation process for generalization ability is cumbersome and cannot provide reproducible comparisons of generalization across different robots, model versions, and test environments.
By collecting task-related data, a multi-dimensional scoring vector for the embodied robot is determined, including task success, time efficiency, trajectory efficiency, grasping stability, collision safety, risk avoidance, and failure recovery. Based on these scoring vectors and preset training conditions, the generalization ability index value is determined, and the failure reason label is determined through the constraint relationship of task failure data.
It enriches the evaluation dimensions, simplifies the generalization ability evaluation process, can distinguish specific reasons for failure and types of insufficient transfer in a single test, and provides interpretable multidimensional evaluation and reproducible generalization ability indicators.
Smart Images

Figure CN123060697A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, equipment, medium and program product for evaluating embodied robots. Background Technology
[0002] In recent years, with the rapid development of embodied intelligence technology, embodied intelligent robots have gradually evolved from executing preset actions to understanding tasks, perceiving the environment, and completing complex operational tasks in real or simulated environments. The evaluation objects of embodied intelligent robots have also expanded from single action execution or algorithm modules to complete task execution capabilities jointly determined by perception, semantic understanding, task planning, motion control, and safety constraints.
[0003] Currently, embodied intelligent robots are mainly evaluated by constructing preset task sets, standard datasets, or simulation test environments. A typical evaluation process includes: first, configuring test conditions and recording the execution process for tasks such as navigation, grasping, transporting, or interaction; then, calculating the task success rate, completion time, or number of collisions, and outputting a comprehensive score; finally, evaluating generalization ability by retesting on new tasks, new objects, or new scenarios, and generating a test report or model ranking after the evaluation is completed.
[0004] However, the above method relies primarily on task success rate when evaluating embodied intelligent robots, failing to distinguish between specific reasons for failure and types of insufficient transferability; furthermore, multiple tests are required to obtain a generalization ability assessment. This results in a relatively singular evaluation dimension for embodied intelligent robots and a cumbersome process for obtaining a generalization ability assessment. Summary of the Invention
[0005] This application provides a method, apparatus, device, medium, and program product for evaluating embodied robots, which can enrich the evaluation dimensions of embodied intelligent robots and simplify the process of obtaining generalization ability evaluation.
[0006] In a first aspect, embodiments of this application provide a method for evaluating an embodied robot. The method includes: collecting task-related data during the execution of a first task by the embodied robot; determining a multi-dimensional scoring vector for the embodied robot based on the task-related data and preset task success conditions, preset safety constraints, and preset scoring indicators corresponding to the first task; wherein the multi-dimensional scoring vector includes a task success scoring vector, a time efficiency scoring vector, a trajectory efficiency scoring vector, a grasping stability scoring vector, a homing accuracy scoring vector, a collision safety scoring vector, a risk avoidance scoring vector, a failure recovery scoring vector, and a human-robot interaction scoring vector; determining a generalization ability index value for the embodied robot based on the multi-dimensional scoring vector and preset training conditions and preset evaluation conditions corresponding to the first task; and determining a failure reason label based on the constraint relationship between the task failure data and the task failure data, when the task-related data includes task failure data.
[0007] The technical solution provided in this application offers at least the following beneficial effects: When evaluating embodied robots, a multi-dimensional scoring vector can be determined based on task-related data collected during task execution; then, based on the multi-dimensional scoring vector and the preset training and evaluation conditions corresponding to the task, the generalization ability index value of the embodied robot can be determined; and failure reason labels can be determined based on the constraints between task failure data and the data. Thus, specific failure reasons and transfer insufficiency types can be distinguished through the multi-dimensional scoring vector and failure reason labels; and the generalization ability index value of the embodied robot can be obtained with only one test. This enriches the evaluation dimensions of embodied intelligent robots and simplifies the process of obtaining generalization ability evaluation.
[0008] One possible implementation is that the aforementioned task-related data includes: visual observation data, movement trajectory data, joint state data, end-effector pose data, contact force, collision events, task events, voice interaction data, and anomaly recovery logs.
[0009] Another possible implementation involves determining the generalization ability index value of the embodied robot based on the multi-dimensional scoring vector and the preset training conditions and preset evaluation conditions corresponding to the first task. This includes: determining the task success retention rate, task quality score retention rate, safety constraint performance index value, and failure recovery performance index value of the embodied robot based on the multi-dimensional scoring vector, preset training conditions, and preset evaluation conditions; and determining the generalization ability index value based on the task success retention rate, task quality score retention rate, safety constraint performance index value, and failure recovery performance index value.
[0010] Another possible implementation is that, after determining the failure reason label based on the constraint relationship between the task failure data and the task failure data, the above-mentioned embodied robot evaluation method further includes: determining at least one of the following tasks based on the above-mentioned failure reason label and task failure data: data replenishment task, simulation disturbance task, annotation and verification task, and model optimization task.
[0011] Another possible implementation is that, after determining the multi-dimensional scoring vector of the embodied robot based on task-related data and the preset task success conditions, preset safety constraints, and preset scoring indicators corresponding to the first task, the above-mentioned embodied robot evaluation method further includes: determining the comprehensive score of the embodied robot based on the above-mentioned multi-dimensional scoring vector and the weights corresponding to each scoring vector in the multi-dimensional scoring vector; and storing the multi-dimensional scoring vector and the comprehensive score in the evaluation task record corresponding to the embodied robot.
[0012] Secondly, embodiments of this application provide an embodied robot evaluation device, comprising: a data acquisition module and a determination module; the data acquisition module is used to acquire task-related data during the execution of a first task by the embodied robot; the determination module is used to determine a multi-dimensional scoring vector of the embodied robot based on the task-related data and preset task success conditions, preset safety constraints, and preset scoring indicators corresponding to the first task; wherein, the multi-dimensional scoring vector includes a task success scoring vector, a time efficiency scoring vector, a trajectory efficiency scoring vector, a grasping stability scoring vector, a homing accuracy scoring vector, a collision safety scoring vector, a risk avoidance scoring vector, a failure recovery scoring vector, and a human-robot interaction scoring vector; the determination module is also used to determine the generalization ability index value of the embodied robot based on the multi-dimensional scoring vector and preset training conditions and preset evaluation conditions corresponding to the first task; the determination module is also used to determine a failure reason label based on the constraint relationship between the task failure data and the task failure data when the task-related data includes task failure data.
[0013] One possible implementation is that the aforementioned task-related data includes: visual observation data, movement trajectory data, joint state data, end-effector pose data, contact force, collision events, task events, voice interaction data, and anomaly recovery logs.
[0014] Another possible implementation, the aforementioned determining module, is specifically used to: determine the task success retention rate, task quality score retention rate, safety constraint performance index value, and failure recovery performance index value of the embodied robot based on the multi-dimensional scoring vector, preset training conditions, and preset evaluation conditions; and determine the generalization ability index value based on the task success retention rate, task quality score retention rate, safety constraint performance index value, and failure recovery performance index value.
[0015] In another possible implementation, the aforementioned determining module is further used to determine, after determining the failure reason label based on the constraint relationship between the task failure data and the task failure data, at least one of the following tasks based on the failure reason label and the task failure data: data re-collection task, simulation disturbance task, annotation and verification task, and model optimization task.
[0016] In another possible implementation, the aforementioned embodied robot evaluation device further includes a storage module; a determination module, which is further used to determine the comprehensive score of the embodied robot based on the multi-dimensional scoring vector of the embodied robot after determining the multi-dimensional scoring vector and the weights corresponding to each scoring vector in the multi-dimensional scoring vector, based on the task-related data and the preset task success conditions, preset safety constraints, and preset scoring indicators corresponding to the first task; and a storage module, which is used to store the multi-dimensional scoring vector and the comprehensive score in the evaluation task record corresponding to the embodied robot.
[0017] Thirdly, this application provides an electronic device comprising: a processor and a memory; the memory stores a program or instructions executable on the processor, wherein the program or instructions, when executed by the processor, implement the method of the first aspect described above.
[0018] Fourthly, this application provides a readable storage medium on which a program or instructions are stored, which, when executed by a computer, implement the method of the first aspect described above.
[0019] Fifthly, this application provides a computer program product stored in a storage medium, which, when executed by a computer, implements the method described in the first aspect.
[0020] In a sixth aspect, embodiments of this application provide a chip including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the method described in the first aspect.
[0021] The beneficial effects of the second to sixth aspects mentioned above are described in the corresponding description of the first aspect and will not be repeated here. Attached Figure Description
[0022] Figure 1 A schematic diagram of the network architecture for an application of an embodied robot evaluation method provided in this application embodiment;
[0023] Figure 2 A flowchart illustrating an embodiment of this application provides a method for evaluating an embodied robot.
[0024] Figure 3 A flowchart illustrating another embodied robot evaluation method provided in this application embodiment;
[0025] Figure 4 A flowchart illustrating another method for evaluating embodied robots provided in this application embodiment;
[0026] Figure 5 A flowchart illustrating another embodied robot evaluation method provided in this application embodiment;
[0027] Figure 6 This is a schematic diagram of the structure of the evaluation task record in another embodied robot evaluation method provided in the embodiments of this application;
[0028] Figure 7 A flowchart illustrating the implementation process of an embodied robot evaluation method provided in this application embodiment;
[0029] Figure 8 This is a schematic diagram of the structure of an embodied robot evaluation system provided in an embodiment of this application;
[0030] Figure 9 This is a schematic diagram of the structure of an embodied robot evaluation device provided in an embodiment of this application;
[0031] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0032] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.
[0033] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0034] The terms "at least one," "at least one of," etc., used in the specification and claims of this application refer to any one, any two, or a combination of two or more of the included items. For example, at least one of a, b, and c can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more items, and its meaning is similar to that of "at least one."
[0035] In the description of this application, unless otherwise stated, "a plurality of" means two or more.
[0036] The embodiments of this application provide a method, apparatus, device, medium, and program product for evaluating embodied robots, which can be applied to evaluation scenarios of embodied robots.
[0037] With the development of embodied intelligence, basic robot models, vision-language-action models, and multimodal large-scale models, robots are gradually shifting from executing pre-set actions to understanding tasks, perceiving objects, planning actions, and completing complex tasks such as grasping, carrying, returning, delivering, obstacle avoidance, and guided interaction in real-world, simulated, digital twin, or hybrid test environments. The objects of robot evaluation have also expanded from single actions or single algorithm modules to complete task execution capabilities jointly determined by perception, semantic understanding, task planning, motion control, safety constraints, and anomaly recovery.
[0038] Currently, relevant technologies mainly evaluate robots or embodied intelligent agents in the following ways: First, by constructing a pre-set set of tasks, standard datasets, or benchmarks, and statistically analyzing success rates or creating leaderboards for tasks such as navigation, grasping, transporting, opening and closing operations, visual question answering, and multi-turn interactions; second, by configuring test tasks in simulation environments, real test fields, or digital twin environments, and recording results such as task completion time, trajectory, and collisions; and third, by setting scoring indicators and weights for several capability dimensions, and outputting individual scores or comprehensive scores. While these methods can reflect the robot's performance under specific tasks and test conditions, the evaluation results typically stop at success or failure, comprehensive scores, test reports, or model rankings.
[0039] However, for real-world embodied tasks, simply adding scoring dimensions or using weighted summation cannot fully address the capability assessment problem. Different robots may achieve the same or similar overall scores, but the reasons for the lost points may stem from perceptual misdetection, semantic understanding errors, unstable grasping control, path planning failure, safety constraint triggering, or insufficient recovery strategies. Furthermore, a severe deficiency in one safety dimension may be offset by other high-scoring dimensions during weighted calculation. Therefore, the overall score itself struggles to retain structural information about task execution quality and cannot answer at which stage, for which object, and under what conditions the robot exhibits capability deficiencies.
[0040] The evaluation of generalization ability in related solutions often involves retesting on new tasks, new items, or new scenarios and calculating the success rate. However, the differences between training conditions, benchmark evaluation conditions, and current evaluation conditions are usually not structured and hierarchically controlled. Item category, size, material, grasping attributes, task combinations, scene layout, occlusion, lighting, human interference, path obstruction, dynamic obstacles, and robot platform changes can all occur simultaneously, leading to a decline in evaluation results. It is difficult to determine whether this decline corresponds to insufficient transfer across items, tasks, scenarios, disturbances, or platforms, and it is also difficult to conduct reproducible generalization comparisons between different robots, model versions, and test environments.
[0041] Furthermore, visual observations, recognition confidence levels, trajectories, joint states, end-effector poses, contact forces, collision events, safety constraint trigger records, voice interactions, and anomaly recovery logs during robot execution are typically stored as scattered logs, lacking a stable correspondence with task conditions, scoring items, generalization results, and failure reasons. Failure reasons often rely on manual review, making it difficult for evaluation systems to differentiate between perception failures, semantic understanding failures, grasping and control failures, path planning failures, risk constraint failures, interaction failures, or recovery failures based on process evidence. Benchmarks or testing platforms usually use scores, reports, or rankings as evaluation endpoints. Even if a robot's poor performance under certain object, scene, or disturbance conditions is identified, the evaluation results are difficult to directly translate into data acquisition, simulation disturbance generation, annotation verification, object semantic correction, or model optimization tasks. Therefore, a breakpoint still exists between evaluation and training data governance, simulation generation, and model iteration.
[0042] The main shortcomings of the related technologies can be summarized as follows:
[0043] (a) The lack of standardized templates for evaluation tasks makes it difficult to ensure that the task conditions are consistent across different test sites, different robots, and different batches.
[0044] (ii) The evaluation objects and scene disturbances lack structured descriptions, and factors such as item categories, grasping difficulty, risk level, scene layout, lighting occlusion and human-machine interference are difficult to control in a unified manner.
[0045] (iii) The scoring dimensions are too singular. Existing evaluations often focus on whether the task is successful, lacking a comprehensive evaluation of process safety, trajectory efficiency, risk avoidance, failure recovery and generalization transfer.
[0046] (iv) Cross-scenario generalization is difficult to quantify. There is a lack of unified indicators for the transfer of robot capabilities between training fields, simulation fields, real exhibition halls, home or warehouse scenarios.
[0047] (v) The evaluation results are difficult to drive training in reverse. Failed samples, weak capabilities and high-risk tasks cannot automatically form training data reinforcement or strategy optimization tasks.
[0048] Therefore, the technical problem that needs to be solved is not simply to add evaluation indicators or design a new weighted scoring formula, but rather how to conduct interpretable multidimensional evaluation of task execution quality based on the conditions and process evidence of a single task execution, conduct hierarchical generalization evaluation of ability transfer under different changing conditions, form failure attribution based on process evidence, and transform the evaluation conclusions into executable training reinforcement tasks, thereby forming a reproducible, comparable, attributable, and revertible embodied intelligent robot evaluation closed loop.
[0049] To address the aforementioned technical problems, this application provides a method, apparatus, device, medium, and program product for evaluating embodied robots. When evaluating an embodied robot, a multi-dimensional scoring vector can be determined based on task-related data collected during task execution. Then, based on the multi-dimensional scoring vector and the preset training and evaluation conditions corresponding to the task, the generalization ability index value of the embodied robot is determined. Furthermore, failure reason labels can be determined based on the constraints between task failure data and the data. Thus, specific failure reasons and transfer insufficiency types can be distinguished through the multi-dimensional scoring vector and failure reason labels; and the generalization ability index value of the embodied robot can be obtained with only one test. This enriches the evaluation dimensions of embodied intelligent robots and simplifies the process of obtaining generalization ability evaluation.
[0050] The following description, in conjunction with the accompanying drawings, details the embodied robot evaluation method, apparatus, equipment, medium, and program products provided in the embodiments of this application.
[0051] Figure 1 This illustration shows a network architecture for an application of a semantic tagging method provided in an embodiment of this application. For example... Figure 1 As shown, the network architecture includes an embodied robot evaluation device 101 and an embodied robot 102. The embodied robot evaluation device 101 and the embodied robot 102 are interconnected.
[0052] In some embodiments, the embodied robot evaluation device 101 may be a server, a computer, or a processor or processing unit within a server or computer. The server may be a single server or a server cluster comprising multiple servers. It should be noted that the embodiments of this application do not limit the specific device form of the embodied robot evaluation device 101. Figure 1 The example shown is a single server, embody robot evaluation device 101.
[0053] In some embodiments, during the execution of the first task by the embodied robot 102, the embodied robot evaluation device 101 collects task-related data; based on the task-related data and the preset task success conditions, preset safety constraints, and preset scoring indicators corresponding to the first task, the embodied robot evaluation device 101 determines a multi-dimensional scoring vector for the embodied robot 102; wherein, the multi-dimensional scoring vector includes a task success scoring vector, a time efficiency scoring vector, a trajectory efficiency scoring vector, a grasping stability scoring vector, a homing accuracy scoring vector, a collision safety scoring vector, a risk avoidance scoring vector, a failure recovery scoring vector, and a human-robot interaction scoring vector; based on the multi-dimensional scoring vector and the preset training conditions and preset evaluation conditions corresponding to the first task, the embodied robot evaluation device 101 determines the generalization ability index value of the embodied robot 102; when the task-related data includes task failure data, the embodied robot evaluation device 101 determines the failure reason label based on the constraint relationship between the task failure data and the task failure data.
[0054] It should be noted that the network architecture described in the embodiments of this application is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided in the embodiments of this application. As network architectures evolve, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.
[0055] See Figure 2 This is a flowchart illustrating a method for evaluating an embodied robot provided in an embodiment of this application. Figure 2 As shown, the embodied robot evaluation method provided in this application embodiment can be implemented by the above-mentioned embodied robot evaluation device, specifically including the following steps 201 to 204.
[0056] Step 201: During the execution of the first task by the embodied robot, the embodied robot evaluation device collects task-related data.
[0057] In some embodiments of this application, the aforementioned embodied robot may be a robot to be tested or a simulated robot.
[0058] In some embodiments of this application, the embodied robot evaluation device can access the embodied robot, record the robot identifier (robot_id), model version (model_version), control interface, and operating environment, and issue a structured task instruction corresponding to the first task to the robot. The embodied robot can then execute the first task based on this structured task instruction.
[0059] In some embodiments of this application, the aforementioned task-related data includes: visual observation data, movement trajectory data, joint state data, end-effector pose data, contact force, collision events, task events, voice interaction data, and anomaly recovery logs.
[0060] In some embodiments of this application, visual observation data refers to environmental perception information collected by the robot through visual sensors such as cameras, depth cameras, and LiDAR, including red-green-blue (RGB) images, depth maps, point clouds, and semantic segmentation results. This type of data records the scene content "seen" by the robot and is the core basis for environmental understanding, object detection, pose estimation, and navigation and obstacle avoidance.
[0061] In some embodiments of this application, motion trajectory data refers to the record of the robot's motion path during task execution, typically stored in time-series format as changes in the robot's pose (position and orientation) on a map or world coordinate system. For wheeled / legged robots, this includes chassis displacement, heading angle, and velocity; for drones, it also includes altitude and flight attitude. This data is used for trajectory analysis, playback reproduction, and navigation algorithm evaluation.
[0062] In some embodiments of this application, joint state data refers to the real-time operating state of each joint of the robot, including physical quantities such as joint angle (position), angular velocity, torque / current, etc. For multi-degree-of-freedom systems such as robotic arms and humanoid robots, joint state data reflects the robot's internal kinematic configuration and serves as the fundamental input for motion planning, control, and fault diagnosis.
[0063] In some embodiments of this application, end-effector pose data refers to the six-dimensional pose of the robot's end effector (such as a robotic arm gripper or tool center point) in the task space, namely, its three-dimensional spatial position (x, y, z) and three-dimensional rotational orientation (usually represented by quaternions or Euler angles). This data directly describes the state of the robot's "hand" in space and is a key evaluation indicator for the success of the operation task.
[0064] In some embodiments of this application, contact force refers to the force and torque information between the robot's end effector or tactile sensor and environmental objects, typically including normal force and tangential force components. Contact force data reflects the intensity of physical interactions such as gripping force and assembly pressure, and is an important feedback signal for achieving compliant control, collision avoidance, and fine manipulation.
[0065] In some embodiments of this application, a collision event refers to an unexpected collision record that occurs during robot task execution, typically triggered by a collision detection algorithm or a sudden force change. Event data includes a timestamp of the collision, the location of the collision (such as a joint or link), and an estimated collision force. Collision events are used for safety monitoring, anomaly analysis, and task failure attribution.
[0066] In some embodiments of this application, task events refer to key state node records during task execution, such as task start, step switching, subtask completion, task success / failure, etc. Each event typically includes an event type, a timestamp, and additional descriptive information. Task events constitute the logical skeleton of task execution, used for process tracing, performance evaluation, and post-mortem analysis.
[0067] In some embodiments of this application, voice interaction data refers to the audio signals generated during human-computer voice interaction, along with corresponding text transcription and intent recognition results. This includes user voice commands, robot voice broadcast content, and dialogue context. This data records the complete process of natural language interaction and serves as a direct basis for evaluating the performance of the voice interaction system and the user experience.
[0068] In some embodiments of this application, the anomaly recovery log refers to a detailed record of the robot's execution of recovery strategies after encountering anomalies (such as lost positioning, failed grasping, communication interruption, etc.). The log content includes the anomaly type, trigger time, recovery action sequence, and recovery result (success / failure). The anomaly recovery log is a key data source for system robustness analysis and fault tolerance mechanism optimization.
[0069] Thus, the aforementioned task-related data can include: visual observation data, movement trajectory data, joint state data, end-effector pose data, contact forces, collision events, task events, voice interaction data, and anomaly recovery logs. This allows for the acquisition of comprehensive, multi-dimensional task-related data, leading to more accurate evaluations of embodied robots.
[0070] Step 202: The embodied robot evaluation device determines the multi-dimensional scoring vector of the embodied robot based on task-related data and the preset task success conditions, preset safety constraints, and preset scoring indicators corresponding to the first task.
[0071] The aforementioned multi-dimensional scoring vectors include task success scoring vector, time efficiency scoring vector, trajectory efficiency scoring vector, grasping stability scoring vector, homing accuracy scoring vector, collision safety scoring vector, risk avoidance scoring vector, failure recovery scoring vector, and human-machine interaction scoring vector.
[0072] In some embodiments of this application, before the embodied robot performs the first task, the embodied robot evaluation device can first select a task template based on the ability to be tested, such as recognition, grasping, carrying, returning, delivering, obstacle avoidance, guided interaction, opening and closing operations, or combined tasks. The task template presets task objectives, initial conditions, success conditions, safety constraints, time thresholds, and scoring indicators. Then, the test scenario is configured according to the task template, a set of target items is selected, semantic tags of the items are referenced, and item position and posture, occlusion level, illumination level, human interference, path obstruction, dynamic obstacles, and risk constraints are set. The embodied robot evaluation device can obtain the preset task success conditions, preset safety constraints, and preset scoring indicators corresponding to the first task from the task template.
[0073] In some embodiments of this application, preset task success conditions refer to quantitative or logical standards that are clearly defined before task execution and used to determine whether the task has achieved its objective. These typically include whether key operations are completed (e.g., whether an object is grasped and placed at the target location), whether the final state meets expectations (e.g., the end-effector pose error is within a threshold), and whether the task is completed within a specified time. Preset success conditions provide an objective "passing grade" for task evaluation and serve as the basis for automated evaluation and comparative experiments.
[0074] In some embodiments of this application, preset safety constraints refer to safety boundaries and restrictions that must be strictly followed during task design and execution, typically defined in the form of physical quantity thresholds or behavioral no-go zones. Examples include: joint torque must not exceed a safety limit, movement speed is limited to a safe range, the minimum distance between the end effector and the human body must not be lower than a set value, and entry into specific areas is prohibited. Safety constraints are a prerequisite for ensuring the reliable operation of the robot system in a real environment; once triggered, a protection mechanism (such as emergency stop or deceleration) should be activated.
[0075] In some embodiments of this application, preset scoring metrics refer to a multi-dimensional measurement system set before task execution to quantitatively evaluate task completion quality and system performance. Unlike binary success / failure determination, scoring metrics provide a more granular performance profile, such as task completion time, path smoothness, energy consumption, positioning accuracy, capture success rate, collision count, and recovery time. By weighting or combining multiple scoring metrics, different algorithms, strategies, or systems can be ranked and compared horizontally, guiding optimization directions.
[0076] In some embodiments of this application, the embodied robot evaluation device can calculate the following sub-items, such as task success, time efficiency, trajectory quality, grasping stability, homing accuracy, collision safety, risk avoidance, failure recovery, and human-computer interaction, based on the success conditions, scoring indicators, and safety constraints in the task template, to form the aforementioned multi-dimensional scoring vector score_vector.
[0077] For example, the above-mentioned multi-dimensional scoring vector calculation method includes: a task success score, a time efficiency score, a trajectory quality score, a grasping stability score, a homing accuracy score, a collision safety score, a risk avoidance score, a failure recovery score, and a human-computer interaction quality score. Based on the weight configuration in the task template, the scores of each dimension are weighted to obtain a comprehensive task score; if a safety constraint is triggered, the comprehensive task score is capped or deducted according to the constraint level. The scoring result also retains the original scores of each dimension, facilitating the identification of the robot's capability shortcomings. The multi-dimensional scoring module can employ rule-based scoring, statistical scoring, learning-based scoring, or a hybrid scoring method.
[0078] An implementable scoring method is shown in the following formula (1):
[0079] score_vector = [S_success, S_time, S_path, S_grasp, S_place, S_safety, S_risk, S_recovery, S_interaction]; (1)
[0080] Among them, S_success represents the task success score, S_time represents the time efficiency score, S_path represents the trajectory quality or path efficiency score, S_grasp represents the grasping stability score, S_place represents the placement or repositioning accuracy score, S_safety represents the collision safety score, S_risk represents the risk avoidance score, S_recovery represents the failure recovery score, and S_interaction represents the human-computer interaction quality score. Each sub-item can use a score of 0 to 100, a normalized score of 0 to 1, or an equivalent scoring range preset in the task template.
[0081] Step 203: The embodied robot evaluation device determines the generalization ability index value of the embodied robot based on the multi-dimensional scoring vector and the preset training conditions and preset evaluation conditions corresponding to the first task.
[0082] In some embodiments of this application, the above-mentioned generalization ability index value is used to indicate the generalization ability of the embodied robot across objects, tasks, scenes, disturbances and platforms; if the robot can still complete the task under the conditions of objects, scenes or disturbances that do not appear in the training set, the corresponding generalization score is improved.
[0083] It is understandable that the larger the value of the above generalization ability index, the stronger the generalization ability; conversely, the smaller the value of the above generalization ability index, the weaker the generalization ability.
[0084] In some embodiments of this application, the embodied robot evaluation device determines the generalization ability index value based on the multi-dimensional scoring vector and the difference between the preset training conditions and the preset evaluation conditions.
[0085] In some embodiments of this application, combined with Figure 2 ,like Figure 3 As shown, step 203 can be implemented through steps 203a and 203b below.
[0086] Step 203a: The embodied robot evaluation device determines the embodied robot's task success retention rate, task quality score retention rate, safety constraint performance index value, and failure recovery performance index value based on multi-dimensional scoring vectors, preset training conditions, and preset evaluation conditions.
[0087] In some embodiments of this application, the task success retention rate refers to the percentage of times an embodied robot successfully completes the same task multiple times over time or during training iterations. It reflects not only the success rate of a single task but also, more importantly, whether the system can consistently and stably meet the preset task success conditions after long-term operation, environmental changes, or policy updates. A high task success retention rate signifies that the strategy possesses good robustness and generalization ability, and will not experience significant performance fluctuations due to accumulated errors, environmental disturbances, or model degradation. It is a core indicator for measuring system reliability.
[0088] In some embodiments of this application, the task quality score retention rate refers to the degree to which the mean or distribution of the task quality score (derived from a preset scoring index system) of the embodied robot remains consistent over time across multiple task executions. Unlike the task success retention rate, which focuses on "whether it is completed," this indicator focuses more on "how well it is completed," such as the accuracy of the grasping posture, the smoothness of the movement path, and the stability of the operation time. A high retention rate indicates that the system can not only complete the task stably, but also that the quality level of each completion fluctuates little, reflecting the continuous consistency of the strategy at the fine operation level.
[0089] In some embodiments of this application, the safety constraint performance index refers to a quantitative measure of the degree to which the embodied robot adheres to preset safety constraints during task execution. This index is typically presented in the form of the number of safety violations, the magnitude of constraint violations, and the frequency of approaching safety boundaries. Examples include the number of times joint torque exceeds limits, the percentage of time the minimum human-robot distance is below a threshold, and the frequency of collision events. A lower index value indicates a stronger ability for the system to ensure its own and its surrounding environment's safety in complex interactive environments, and is a key basis for assessing whether the embodied system is suitable for practical deployment.
[0090] In some embodiments of this application, the failure recovery performance index refers to the quantified value of the embodied robot's ability to return to normal execution through a preset recovery strategy after encountering an anomaly or failure in task execution. This index typically considers dimensions such as recovery success rate (whether it can successfully recover after an anomaly), recovery time (the time from the occurrence of the anomaly to the return to normal), and task continuity quality after recovery (whether the original goal can still be achieved after recovery). A higher failure recovery performance index means that the system has good fault tolerance and resilience, and can autonomously resolve unexpected interference in real-world scenarios rather than being directly paralyzed.
[0091] Step 203b: The embodied robot evaluation device determines the generalization ability index value based on the task success retention rate, task quality score retention rate, safety constraint performance index value, and failure recovery performance index value.
[0092] In some embodiments of this application, the embodied robot evaluation device can calculate cross-item generalization index, cross-task generalization index, cross-scene generalization index, cross-disturbance generalization index, and cross-platform generalization index through the generalization ability evaluation module. Each index is determined based on the difference between training and evaluation conditions, task success rate, score decay rate, safety constraint trigger rate, and failure recovery rate. For example, when evaluation items belong to the same transfer group but their geometry, weight, or material changes, the system calculates the cross-item generalization index; when the levels of lighting, occlusion, human interference, or path obstruction increase, the system calculates the cross-disturbance generalization index. The generalization ability evaluation module calculates the generalization index value (generalization_index) based on the difference between training and evaluation conditions. This index may include the following sub-items:
[0093] G_object: Cross-item generalization score, used to evaluate the robot's performance when faced with new categories, sizes, materials, or grasping attributes of items.
[0094] G_task: Cross-task generalization score, used to evaluate the robot's performance when faced with insufficiently trained combined tasks, long-term tasks, or tasks with changing order.
[0095] G_scene: Cross-scene generalization score, used to evaluate the robot's performance when it moves from the training field to exhibition halls, homes, warehouses, or commercial spaces.
[0096] G_disturbance: Cross-disturbance generalization score, used to evaluate the robot's performance when faced with occlusion, changes in lighting, human interference, path blockage, or dynamic obstacles.
[0097] G_platform: Cross-platform generalization score, used to evaluate the adaptability of the same task to different robot bodies, manipulators, grippers or sensor configurations.
[0098] The generalization_index can be calculated using the following formula (2):
[0099] generalization_index=α1×G_object+α2×G_task+α3×G_scene+α4×G_disturbance+α5×G_platform; (2)
[0100] Here, G_object, G_task, G_scene, G_disturbance, and G_platform represent the generalization scores across objects, tasks, scenes, disturbances, and platforms, respectively; α1 to α5 represent the weight coefficients of the above five generalization scores, used to reflect the degree of attention the evaluation target pays to different generalization dimensions. α1 to α5 can be determined by the evaluation task template, industry testing specifications, or the operator's evaluation strategy; typically, each α value can be constrained to be greater than or equal to 0, and α1+α2+α3+α4+α5=1. This index not only records whether the task is completed, but also reflects the robot's ability transfer level under new objects, new tasks, new scenes, new disturbances, and new platforms.
[0101] In some embodiments of this application, for the variable dimension d, the system obtains the task performance under baseline and variable conditions from the corresponding Evaluation Task Record (ETR), and calculates the task success retention rate R_success,d, the task quality score retention rate R_score,d, the safety constraint performance R_safety,d, and the failure recovery performance R_recovery,d respectively, and forms the generalized component G_d of the variable dimension. Specifically, it can be determined by the following formula (3):
[0102] G_d=β1×R_success,d+β2×R_score,d+β3×R_safety,d+β4×R_recovery,d; (3)
[0103] Where d represents object, task, scene, disturbance, or platform; each R value is normalized to 0 to 1; β1 to β4 are the weight coefficients of the corresponding sub-items, where β1, β2, β3, and β4 are all greater than or equal to 0 and their sum is 1. R_safety,d is determined based on the increment of the safety constraint trigger rate under changing conditions relative to the baseline condition, and this item decreases when the number of safety events increases. The system further obtains G_object, G_task, G_scene, G_disturbance, and G_platform, and forms a comprehensive generalization_index according to the existing weights α1 to α5.
[0104] In this way, the embodied robot evaluation device can first determine the task success retention rate, task quality score retention rate, safety constraint performance index value, and failure recovery performance index value, and then generalization ability index value based on this information. Thus, by using parameters from multiple dimensions, the determined generalization ability index value can be made more accurate.
[0105] Step 204: When the task-related data includes task failure data, the embodied robot evaluation device determines the failure reason label based on the constraint relationship between the task failure data and the task failure data.
[0106] In some embodiments of this application, the constraint relationships between the above-mentioned task failure data and task failure data include: failure stage, failure object, failure action, perceived confidence level, risk constraint triggering situation, and execution log.
[0107] In some embodiments of this application, the embodied robot evaluation device generates failure reason labels based on the failure stage, failure object, failure action, perception confidence, risk constraint triggering status, and execution log, and feeds back the failure reason, scoring vector, and sample reference to the training data acquisition system, simulation task generation system, or model optimization system.
[0108] In some embodiments of this application, the embodied robot evaluation device associates the failure stage, failure object ref, and failure action with perception confidence, object semantic reference, safety constraint trigger record, contact force, collision event, trajectory deviation, and anomaly recovery log. For example: when the target object is not identified and the perception confidence is below a threshold, a perception failure label is generated; when the object is correctly identified but the selected action violates the risk type, handling constraint, or allowed action in the object semantics, a semantic understanding failure or risk constraint failure label is generated; when the contact force exceeds the grasping force range, the object slips, or the grasping posture exceeds the allowed range, a grasping control failure label is generated; when a collision occurs, the target area is not reached, or the path deviation exceeds a threshold, a path planning failure label is generated; when the task fails and cannot be replanned and completed within a limited time, a recovery failure label is generated. The failure reason also stores the triggering rule and evidence ref for easy review.
[0109] The embodied robot evaluation method provided in this application allows for the determination of a multi-dimensional scoring vector for the embodied robot based on task-related data collected during task execution. Then, based on the multi-dimensional scoring vector and the preset training and evaluation conditions corresponding to the task, the generalization ability index value of the embodied robot is determined. Furthermore, failure reason labels can be determined based on the constraints between task failure data and the data. This allows for the differentiation between specific failure reasons and transfer insufficiency types through the multi-dimensional scoring vector and failure reason labels; and the generalization ability index value of the embodied robot can be obtained with only one test. This enriches the evaluation dimensions of embodied intelligent robots and simplifies the process of obtaining generalization ability evaluations.
[0110] In some embodiments of this application, combined with Figure 2 ,like Figure 4 As shown, after step 204 above, the embodied robot evaluation method provided in this application embodiment may further include step 205 below.
[0111] Step 205: Based on the failure reason labels and task failure data, the embodied robot evaluation device determines at least one of the following tasks: data acquisition task, simulation disturbance task, annotation and verification task, and model optimization task.
[0112] In some embodiments of this application, when the embodied robot evaluation device identifies the failure cause as a false detection of perception, incorrect selection of gripping points, insufficient gripping force, failure of path planning to avoid obstacles, triggering of risk rules, or misunderstanding of human-computer interaction, it generates reflow tasks such as supplementing visual samples, supplementing gripping annotations, adding force control training samples, generating simulated disturbance scenarios, updating safety rules, or supplementing language interaction samples, and associates them with the corresponding data acquisition, simulation generation, annotation verification, or model training system. The failure attribution module can classify failure causes into types such as perception failure, semantic understanding failure, gripping failure, path planning failure, risk constraint failure, interaction failure, execution timeout, and recovery failure. The system can generate failure cause labels according to the following rules:
[0113] 1. If the target item is not identified or the identification confidence level is below the threshold, it is marked as a perception failure.
[0114] 2. If the item is correctly identified but an incorrect task strategy is selected, such as treating an untouchable exhibit as a graspable object, it is marked as a semantic understanding failure or a risk constraint failure.
[0115] 3. If the gripping point is selected correctly but the object slips, the gripping force is abnormal, or the contact force exceeds the threshold, it will be marked as a gripping failure.
[0116] 4. If the robot encounters a collision, fails to detour, or fails to reach the target area, it is marked as a path planning failure.
[0117] 5. If the robot can replan and complete the task after a failure, record the failure recovery as successful; otherwise, record the recovery failure as unsuccessful.
[0118] The training feedback module writes `ailure_reason`, `score_vector`, `generalization_index`, and process data references into the training feedback reference (`training_feedback_ref`), and generates data re-collection tasks, simulation perturbation tasks, annotation verification tasks, or model optimization tasks. For example, when the system finds that the robot has a high failure rate in "identifying medicine bottles under occlusion conditions", it can automatically generate a re-collection task; when the system finds that the robot exceeds the contact force limit in "grasping fragile items", it can generate simulation perturbation tasks and grasping strategy optimization tasks.
[0119] In some embodiments of this application, the embodied robot evaluation device accumulates ETR records under the same comparison group identifier (comparison_group_id). For general failures, when the occurrence frequency of a certain failure_reason reaches the minimum sample size N_min and the occurrence ratio exceeds the threshold θ_f, or when a certain generalization component G_d is lower than the threshold θ_g, a training feedback action is generated according to a preset mapping relationship. For serious failures such as major safety incidents, violations of key safety constraints, or failures specified in the task template, a single ETR can immediately trigger a training_feedback_action. Perception failure corresponds to data supplementation or annotation verification; semantic understanding failure or risk constraint failure corresponds to item semantic correction or safety rule supplementation; grasping control failure corresponds to stress control sample supplementation, simulation disturbance, or grasping strategy retraining; path planning failure or G_disturbance being lower than the threshold corresponds to obstacle, occlusion, lighting, or personnel interference scene generation; when G_object, G_task, G_scene, or G_platform is lower than the threshold, cross-item, cross-task, cross-scene, or cross-platform supplementary training tasks are generated accordingly. The system writes the generated specific actions into training_feedback_action and associates them with the corresponding training reinforcement task records through training_feedback_ref. N_min, θ_f, and θ_g are configured by task templates, evaluation specifications, or operational strategies and written to ETR or its associated configuration version.
[0120] Thus, the embodied robot evaluation device, based on failure reason labels and task failure data, determines at least one of the following tasks: data acquisition, simulation perturbation, annotation and verification, and model optimization. This allows for training reflow based on the failure reasons, triggering different types of training reinforcement tasks, thereby continuously improving the capabilities of the embodied robot.
[0121] In some embodiments of this application, combined with Figure 2 ,like Figure 5 As shown, after step 202 above, the embodied robot evaluation method provided in this application embodiment may further include steps 206 and 207 as described below.
[0122] Step 206: The embodied robot evaluation device determines the comprehensive score of the embodied robot based on the multi-dimensional scoring vector and the weights corresponding to each scoring vector in the multi-dimensional scoring vector.
[0123] In some embodiments of this application, the above-mentioned comprehensive score total_score can be calculated by the following formula (4):
[0124] total_score = Σ wi × Si; (4)
[0125] Where Si represents the i-th scoring item in score_vector, and wi represents the weight coefficient corresponding to the i-th scoring item, used to reflect the importance that different evaluation tasks place on indicators such as success rate, security, efficiency, and interaction quality. wi can be pre-configured by the task template or automatically assigned by the evaluation platform according to the task type; typically, wi can be constrained to be greater than or equal to 0, and the sum of all weights should be 1. If the security constraint is triggered, the system can also cap, deduct, or directly determine failure for total_score.
[0126] Step 207: The embodied robot evaluation device stores the multi-dimensional scoring vector and comprehensive score in the evaluation task record corresponding to the embodied robot.
[0127] In some embodiments of this application, the total score can be obtained by weighting each sub-item according to its weight, but ETR also retains the original value of each sub-item and its evidence reference. For security constraint triggering events of different levels, the system performs security gating processing such as deduction, score capping, or direct failure to prevent serious security flaws from being offset by other high scores. Therefore, the focus of task quality evaluation in this application is the correspondence between sub-item results, evidence, and security constraints, rather than the weighting formula itself.
[0128] In some embodiments of this application, the embodied robot evaluation device can first establish the aforementioned evaluation task record (ETR). For multiple evaluations of the same robot or model version and the same task template, the system forms an ETR evaluation record group according to comparison_group_id and associates it with the baseline evaluation record through baseline_evaluation_ref. Each ETR record to be compared has changed_dimension and condition_delta, where changed_dimension indicates that the main changes in this evaluation relative to the baseline record are in the item, task, scene, disturbance, or robot platform, and condition_delta records the specific changes and their levels. The system prioritizes comparing ETRs with the same conditions or those falling within a preset tolerance range, except for the target change dimension; when multiple dimensions change simultaneously, each change dimension is recorded separately, and the confidence level of this record for single-dimensional generalization evaluation is reduced or it is included in the combined generalization evaluation. This avoids directly mixing and statistically analyzing success rates under different conditions.
[0129] In some embodiments of this application, the ETR is used to describe the task conditions, execution process, scoring results, generalization results, and feedback information of a robot evaluation task. The ETR can correspond to a complete evaluation task or a subtask within an evaluation task. Specifically, the evaluation task record ETR is not a regular log or a result table merely used to store scores, but rather a structured control record that runs through the aforementioned continuous relationships.
[0130] like Figure 6 As shown, an ETR includes at least the following fields: evaluation_id, robot_id, robot_model, control_policy_version, model_version, task_template_id, task_goal, initial_condition, success_condition, safety_constraint, object_semantic_ref, scene_id, scene_type, disturbance_profile, difficulty_level, start_time, end_time, process_data_ref, trajectory_digest, contact_event_log, collision_event_log, human_interaction_log, score_vector, generalization_vector, failure_stage, failure_object_ref, failure_action, failure_reason_label, training_feedback_action, and dataset_or_simulation_update_ref. An ETR can correspond to a complete task or sub-tasks such as grasping, moving, placing, and obstacle avoidance.
[0131] Wherein, object_semantic_ref represents a reference to an item's semantic identifier or semantic record; process_data_ref represents a reference to execution process data; generalization_vector represents a hierarchical generalization evaluation vector composed of G_object, G_task, G_scene, G_disturbance, and G_platform; generalization_index represents a comprehensive generalization index obtained by summarizing the hierarchical generalization evaluation vector; failure_reason represents a failure reason generated based on process evidence and constraint relationships; training_feedback_action represents a specific training reinforcement action generated by the system; and training_feedback_ref represents a reference identifier for the corresponding training reinforcement task or task record.
[0132] It can be summarized into the following field group:
[0133] 1. Basic identification fields: evaluation_id is used to uniquely identify an evaluation; robot_id is used to identify the robot under test; model_version is used to identify the version of the model or control strategy under test; task_template_id is used to associate the evaluation task template; dataset_ref is used to associate the training dataset or evaluation dataset version.
[0134] 2. Task Scenario Fields: scene_id is used to identify the test scenario; object_set_id is used to identify the set of items participating in the evaluation; semantic_tag_ref is used to reference the semantic tag of the item; initial_pose is used to record the initial pose of the robot and the target item; storage_location is used to record the location of the return or target placement.
[0135] 3. Disturbance Difficulty Fields: difficulty_level records the task difficulty level; disturbance_profile records the disturbance combination; occlusion_level records the occlusion level; lighting_level records the lighting level; human_interference records human interference conditions; path_block_level records the path blockage degree.
[0136] 4. Task constraint fields: initial_condition records the initial conditions of the task; success_condition records the success conditions of the task; safety_constraint records safety constraints; time_limit records the time limit; allowed_action records allowed or prohibited actions; risk_policy records the rules for handling high-risk objects.
[0137] 5. Process logging fields: execution_log_ref is used to associate execution logs; trajectory_ref is used to associate movement trajectories or robotic arm trajectories; event_log_ref is used to associate critical events; collision_record is used to record collision information; contact_force_ref is used to record contact force or torque; perception_confidence is used to record changes in perception confidence.
[0138] 6. Scoring result fields: metric_config is used to record the scoring metrics and weights; score_vector is used to record the multi-dimensional scoring vector; success_flag is used to record whether the task was successful; risk_avoidance_score is used to record the risk avoidance score; recovery_score is used to record the failure recovery score; human_interaction_score is used to record the human-computer interaction quality score.
[0139] 7. Generalization Feedback Fields: generalization_index records the generalization ability index; failure_reason records the failure reason label; ability_profile generates a robot capability profile; training_feedback_ref is used to associate supplementary training, simulation disturbance generation, or data supplementation tasks.
[0140] An ETR simultaneously binds task_template_id, object_semantic_ref, disturbance_profile, process_data_ref, score_vector, generalization_vector, generalization_index, failure_reason, training_feedback_action, and training_feedback_ref, ensuring a traceable correspondence between task conditions, execution process, evaluation conclusion, failure reason, and feedback action.
[0141] In this way, the embodied robot evaluation device stores multi-dimensional scoring vectors and comprehensive scores in the evaluation task record corresponding to the embodied robot. This allows the evaluation task conditions, robot execution process, and evaluation results to be bound in the same record, making the evaluation results of different robots, different model versions, different test fields, and different disturbance conditions comparable and traceable.
[0142] The following specific embodiments illustrate the embodied robot evaluation method of this application.
[0143] like Figure 7 As shown, the implementation process of the embodied robot evaluation method provided in this application embodiment includes the following S1 to S7:
[0144] S1. Select an evaluation task template. The system selects a task template based on the ability being tested, such as recognition, grasping, carrying, returning, delivery, obstacle avoidance, interactive navigation, opening and closing operations, or combined tasks. The task template includes preset task objectives, initial conditions, success conditions, safety constraints, time thresholds, and scoring metrics.
[0145] S2, Configure Scene, Items, and Disturbances. The system configures the test scene according to the task template, selects the target item set, references item semantic tags, and sets item position and pose, occlusion level, lighting level, personnel interference, path blocking, dynamic obstacles, and risk constraints.
[0146] S3, Connect to the robot and issue tasks. The system connects to the robot under test or the simulated robot, records the robot_id, model_version, control interface and operating environment, and issues structured task instructions to the robot.
[0147] S4, Collect execution process data. During the robot's task execution, the system collects visual observations, trajectory, joint status, end-effector pose, contact force, collision events, task events, voice interaction, and anomaly recovery logs.
[0148] S5, Calculate the multi-dimensional score. The system calculates the score_vector based on the task success conditions, safety constraints, and scoring indicators. The score_vector can include the task success score, time efficiency score, trajectory efficiency score, capture stability score, homing accuracy score, collision safety score, risk avoidance score, failure recovery score, and human-computer interaction score.
[0149] S6, Calculate the generalization ability index. The system calculates the generalization ability across items, tasks, scenes, disturbances, and platforms based on the differences between training and evaluation conditions. If the robot can still complete the task under conditions of items, scenes, or disturbances not present in the training set, the corresponding generalization score is improved.
[0150] S7, Failure Attribution and Backflow Training. The system generates failure reason labels based on the failure stage, failure object, failure action, perceived confidence, risk constraint triggering status, and execution log, and backflows the failure reasons, scoring vectors, and sample references to the training data acquisition system, simulation task generation system, or model optimization system.
[0151] Thus, by establishing a comparison relationship between evaluation records using `comparison_group_id`, `baseline_evaluation_ref`, `changed_dimension`, and `condition_delta`, the generalization evaluation can clearly correspond to specific condition changes, improving the reproducibility and comparability of evaluation results across different robots, model versions, and test environments. By simultaneously saving `score_vector`, process evidence references, and safety gating results, we avoid using comprehensive scores alone to mask specific capability shortcomings or safety risks, enabling evaluation results to pinpoint task stages, operational objects, action types, and triggering events. By associating process events, item semantics, and safety constraints and saving the triggering rules and `evidence_ref` for failure reasons, we reduce reliance on manual review, improving the consistency and verifiability of failure attribution. Through the mapping between `failure_reason`, `G_d`, and `training_feedback_action`, and the threshold triggering mechanism, evaluation conclusions are transformed into data supplementation, simulation perturbation, annotation verification, semantic correction, or model retraining tasks, shortening the processing link from problem discovery in evaluation to training enhancement.
[0152] It should be noted that the descriptions of each step S1 to S7 in this embodiment can be found in the descriptions in the above embodiments, and will not be repeated here.
[0153] The following examples illustrate the embodied robot evaluation method provided in this application, using several specific scenarios as examples.
[0154] For example, in a cultural and tourism exhibition hall scenario, three evaluation tasks are configured: identifying and explaining non-touchable exhibits, grabbing and demonstrating interactive props before returning them to their original positions, and delivering cultural and creative products to visitors. The system generates an ETR (Extended Time Record) for each task, referencing the semantic tags of the exhibits, interactive props, and cultural and creative products, and sets occlusion levels, visitor interference locations, lighting levels, and path obstruction conditions. When the robot performs the task of "identifying and explaining non-touchable exhibits," the system collects its visual observations, navigation trajectory, distance to the exhibit, explanation trigger time, and collision events. If the robot does not touch the exhibit, the path distance meets the constraints, and the explanation content is triggered in the designated area, the task is successful. Risk avoidance and collision safety scores have higher weights in the scoring. If the robot mistakenly identifies a non-touchable exhibit as an interactive prop and extends its robotic arm, the system marks the failure as "semantic understanding failure + risk constraint not triggered" and feeds this evaluation record back to the training data acquisition system to generate more comparative samples of non-touchable exhibits and touchable interactive props.
[0155] Taking the task of "grabbing interactive props, completing the demonstration, and returning them to their place" in a cultural and tourism exhibition hall as an example, the system first forms a baseline ETR under conditions of no obstruction, standard lighting, and fixed props. Then, by changing the prop size or material, obstruction level, visitor interference location, and robot gripper platform, a comparative ETR with the same task_template_id and comparison_group_id is formed. The system retains the score_vector and safety constraint trigger evidence from each evaluation, and calculates G_object, G_disturbance, and G_platform respectively.
[0156] If the robot succeeds in its task under standard conditions, but its target recognition confidence level repeatedly falls below a threshold under occlusion conditions, the system generates a perception failure label based on process data. When the failure rate exceeds θ_f or G_disturbance falls below θ_g, the system automatically generates visual sample supplementation and simulation perturbation tasks under occlusion conditions. If the robot can recognize the prop but its contact force exceeds the grasp_force_range in the object's semantic label, a grasping control failure label is generated, triggering a force control sample supplementation or grasping strategy retraining task. This implementation process demonstrates a complete technical chain from comparative evaluation, evaluation calculation, failure attribution to training reinforcement.
[0157] For example, in a home service scenario, the system is configured with an evaluation task of "returning desktop items to a designated area," with items including cups, medicine bottles, remote controls, books, and fruit bowls. The system sets cups and books, which appeared in the training set, as basic items, and new medicine bottles and remote controls, not present in the training set, as cross-item generalization test objects. When the robot performs the task, the system records the object recognition result, grasping point, trajectory, placement error, contact force, and task completion time. If the robot can correctly keep the new medicine bottle upright and return it to the medicine box, the cross-item generalization score improves; if the robot puts the remote control into the medicine box, the task success score and semantic understanding score decrease. Based on the ETR (Extended Time Reference), the system generates a capability profile, indicating that the robot "has good ability to return common items, but insufficient ability to recognize and return sensitive items for children and small black objects," and generates a supplementary task.
[0158] For example, in a warehouse scenario, the system is configured with an evaluation task of "moving boxes from shelves to a designated area". In the basic evaluation, the boxes are unobstructed and the path is unobstructed; in the generalized evaluation, the system adds disturbances such as insufficient lighting, partial occlusion, temporary path blockage, and dynamic personnel passage. The robot completes the task under the basic conditions with a high total_score; under the path blockage condition, the robot fails to detour and times out. The system marks the failure as "path planning failure + insufficient disturbance adaptation" and calculates a low generalization score G_disturbance across disturbances. The training reflow module generates a simulated disturbance task based on this, requiring the addition of training samples with different blockage locations, different personnel movement speeds, and different shelf spacings in the simulation environment.
[0159] Thus, the embodied robot evaluation method provided in this application embodiment can achieve the following effects:
[0160] 1. Improve the comparability of evaluation results. Different robots and different test environments can be compared based on a unified task template, a unified scoring vector, and a unified generalization metric.
[0161] 2. Improve the interpretability of evaluation results. The system not only outputs success or failure, but also the reasons for failure, weaknesses, and specific scoring dimensions.
[0162] 3. Improve robot deployment safety. By using risk avoidance scores, collision safety scores, and safety constraint trigger records, high-risk behaviors during actual deployment can be identified in advance.
[0163] 4. Improve the efficiency of training data reinforcement. Failed evaluation results can be directly fed back to the data acquisition, simulation generation, or model optimization system, forming a closed loop of training and evaluation.
[0164] 5. Supports standardized evaluation and patent cluster layout. This solution can synergize with item semantic tagging patents and data collection and annotation closed-loop patents, and can be subsequently used for the development of embodied intelligence testing specifications, dataset standards, and evaluation standards.
[0165] The embodied robot evaluation method provided in this application focuses on the evaluation processing and evaluation result feedback based on task execution process data. Task template selection, scene and perturbation configuration, item semantic reference, robot access, and process data collection are used to form unified and reproducible evaluation conditions and process evidence. The key protection lies in how the system utilizes these conditions and process evidence to generate multi-dimensional task quality evaluation, hierarchical generalization ability evaluation, and failure reasons, and further generate training reinforcement tasks. The overall score is merely a summary display of the evaluation results and is not the core innovation of this application. Instead of simply piecing together task configuration, data collection, weighted scoring, generalization index, and training feedback, the structured results generated in the previous stage serve as the calculation or triggering basis for the next stage: task templates, item semantics, and perturbation configuration determine evaluation conditions; process data forms evaluation evidence; evaluation conditions and process evidence jointly generate a scoring vector and generalization sub-items; process events and semantic constraints jointly generate failure reasons; and failure reasons and generalization shortcomings trigger corresponding types of training reinforcement tasks.
[0166] The key points include:
[0167] (i) Propose a standardized task evaluation template for embodied intelligent robots, which uniformly describes the task objectives, initial conditions, success conditions, safety constraints, disturbance parameters and scoring indicators, so that different test fields, different robots and different model versions can be evaluated under the same task semantics.
[0168] (ii) A joint configuration mechanism for scene disturbance and object semantics is proposed, which uses object semantic tags, object position and pose, occlusion level, illumination level, personnel interference, path blockage and risk level as evaluation conditions to avoid evaluating robot capabilities solely based on static task results.
[0169] (iii) Propose a multi-dimensional scoring vector to comprehensively evaluate task success, time efficiency, trajectory quality, grasping stability, homing accuracy, collision safety, risk avoidance, failure recovery and human-computer interaction quality, so that the evaluation results are interpretable.
[0170] (iv) Propose a generalization capability evaluation mechanism to quantify robot capability transfer through cross-item, cross-task, cross-scenario, cross-disturbance and cross-platform indicators, rather than just recording whether a single task is successful.
[0171] (v) Propose a failure attribution and training reflow mechanism, which transforms failure stage, failure object, failure action, perceived confidence and environmental disturbance into failure reason labels, and further generates data supplementation, simulation disturbance, annotation verification or model optimization tasks.
[0172] The purpose of the embodied robot evaluation method provided in this application is to improve the comparability, interpretability, and reusability of robot capability evaluation by using standardized task templates, scene perturbation configurations, semantic evaluation objects, multidimensional scoring, generalization index, and failure backflow mechanism.
[0173] The embodied robot evaluation method provided in this application addresses the task execution process of embodied intelligent robots in real-world scenarios, simulated scenarios, digital twin scenarios, or hybrid test environments. It establishes a unified evaluation closed loop around "task templates, scenario perturbations, object semantics, process data, multi-dimensional scoring, generalization evaluation, failure attribution, and training retraining." This solution does not merely involve manual scoring of robot task results; instead, through structured task records and process data collection, it enables the evaluation results to be quantified, reproduced, compared, and used for retraining.
[0174] It should be noted that before formal deployment, batch evaluations of the robot can be performed in a digital twin scenario, followed by retesting on a selected portion of the tasks using a real machine. The system generates both the simulation ETR and the real machine ETR, and associates them using task_template_id, scene_id, object_set_id, and disturbance_profile. If the robot has a high success rate in grasping during simulation, but slips due to unstable contact force control in the real machine, the system marks the difference as "simulation-to-reality migration deviation" and feeds this record back to the simulation parameter calibration and real grasping data re-collection process.
[0175] As can be seen from the above implementation methods, the solution can perform structured evaluation of the task execution process of embodied intelligent robots. It can output not only whether the task is successful, but also multi-dimensional scores, generalization ability index, failure reasons and reflow tasks, thereby supporting robot capability classification, training data reinforcement, simulation perturbation generation and continuous model optimization.
[0176] In practical implementation, fields such as `semantic_tag_ref`, `risk_type`, `grasp_type`, `task_type`, and `success_condition` from the item semantic tagging proposal can be used to configure evaluation tasks and safety constraints. The EDSR sample records, `dataset_version`, and `training_feedback_ref` output from the data collection and annotation closed-loop proposal can be called to locate training data gaps and generate supplementary training tasks. The task execution process, scoring results, failure reasons, and capability profiles can be provided to the generated content trust tagging proposal to record the task context and dispute resolution evidence of the robot during cultural and tourism services. This proposal does not duplicate the protection of item tag generation or training data collection processes; instead, it protects the evaluation task template, multi-dimensional scoring, generalization index, and failure feedback mechanism.
[0177] ETR evaluation task logs are not ordinary evaluation logs, but rather structured control logs used to drive scoring, generalization evaluation, failure attribution, and training refeedback. An ETR is simultaneously bound to task_template_id, semantic_tag_ref, disturbance_profile, execution_log_ref, score_vector, generalization_index, failure_reason, and training_feedback_ref, enabling the system to automatically generate capability evaluations and refeedback tasks based on task conditions, item semantics, scene perturbations, and the execution process.
[0178] Failure attribution can be implemented using the following automatic judgment rules: If the target item is not recognized and perception_confidence is below a preset threshold, failure_reason is marked as perception failure; if the target item is correctly recognized but the robot's selected action violates risk_type, handling_constraint, or allowed_action in semantic_tag_ref, it is marked as risk constraint failure; if the contact force exceeds the grasp_force_range in the item's semantic tag or the object slips, it is marked as grasping control failure; if the trajectory deviates from the target path, collides, or fails to reach the target area, it is marked as path planning failure; if the robot fails to replan and complete the task within a limited time after the task fails, it is marked as recovery failure.
[0179] The evaluation results feedback can generate different types of follow-up tasks: when failure_reason is perception failure, a data acquisition task or annotation review task is generated; when failure_reason is risk constraint failure, an item semantic label correction task or risk rule supplementation task is generated; when failure_reason is grasping control failure, a simulation perturbation task and grasping strategy retraining task are generated; when generalization_index is below a threshold, cross-item, cross-task, cross-scene, or cross-perturbation retraining tasks are generated. This mechanism distinguishes this proposal from existing evaluation schemes that only output evaluation scores or benchmark rankings.
[0180] It should be noted that the above-described method embodiments, or the various possible implementations of the method embodiments, can be executed individually, or, provided there is no conflict, they can be combined with each other. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.
[0181] Figure 8 This is a schematic diagram of the structure of an embodied robot evaluation system provided in an embodiment of this application. Figure 8 As shown, the embodied robot evaluation system 800 may include:
[0182] 1. Input and Data Layer 801. This layer includes the evaluation task template, the robot to be evaluated, the model version, the real or simulated test scenario, the items to be manipulated, the item semantic labels, the scene perturbation parameters, and the safety constraint parameters. The evaluation task template is used to constrain the task objectives, initial conditions, success conditions, time thresholds, action restrictions, and scoring metrics; the item semantic labels are used to provide item category, geometry, weight, grasping capability, risk, scene, and task attributes; the scene perturbation parameters are used to control test conditions such as lighting, occlusion, item posture, human interference, path obstruction, noise, and dynamic obstacles.
[0183] 2. Semantic Processing Layer 802. This layer includes a task template management module, a scene and item configuration module, a robot access and execution monitoring module, and a process data acquisition module. The task template management module calls task templates such as grasping, handling, returning, sorting, delivery, obstacle avoidance, opening and closing, and guided interaction according to the evaluation objectives. The scene and item configuration module generates test scenes based on the task templates, configuring item positions, postures, graspable areas, non-touchable areas, risk levels, and disturbance intensity. The robot access and execution monitoring module connects to different robot platforms through robot control interfaces, simulation interfaces, or edge gateways, recording the robot's perception, planning, control, execution, and anomaly recovery processes. The process data acquisition module simultaneously collects visual observations, task events, trajectories, joint states, end-effector poses, contact forces, collision events, voice interactions, manual confirmation results, and system logs.
[0184] 3. Apply closed-loop layer 803. This layer includes a multi-dimensional scoring module, a generalization ability evaluation module, and a failure attribution and training reflow module. The multi-dimensional scoring module generates a scoring vector based on task success conditions, safety constraints, and scoring configuration; the generalization ability evaluation module calculates the generalization index based on differences in items, tasks, scenes, disturbances, and robot platforms between the training and evaluation scenarios; the failure attribution and training reflow module generates failure reason labels based on failure stage, failure object, failure action, perception confidence, collision records, and safety constraint triggering conditions, and reflows the evaluation results to the data acquisition, simulation generation, annotation correction, or model training system.
[0185] It should be noted that for a detailed explanation of the steps performed by each module and their beneficial effects, please refer to the description in the above embodiments, which will not be repeated here.
[0186] As can be seen, the above mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the embodiments of this application provide corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0187] This application embodiment can divide the embodied robot evaluation device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. Optionally, the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.
[0188] In some embodiments, this application also provides a holographic robot evaluation apparatus. This holographic robot evaluation apparatus may include one or more functional modules for implementing the holographic robot evaluation method of the above method embodiments.
[0189] For example, Figure 9 This is a schematic diagram of the structure of a body robot evaluation device provided in an embodiment of this application. Figure 9 As shown, the embodied robot evaluation device 900 includes: a data acquisition module 901 and a determination module 902.
[0190] The acquisition module 901 is used to collect task-related data during the execution of the first task by the embodied robot. The determination module 902 is used to determine a multi-dimensional scoring vector for the embodied robot based on the task-related data and the preset task success conditions, preset safety constraints, and preset scoring indicators corresponding to the first task. The multi-dimensional scoring vector includes a task success scoring vector, a time efficiency scoring vector, a trajectory efficiency scoring vector, a grasping stability scoring vector, a homing accuracy scoring vector, a collision safety scoring vector, a risk avoidance scoring vector, a failure recovery scoring vector, and a human-robot interaction scoring vector. The determination module 902 is also used to determine the generalization ability index value of the embodied robot based on the multi-dimensional scoring vector and the preset training conditions and preset evaluation conditions corresponding to the first task. The determination module 902 is also used to determine the failure reason label based on the constraint relationship between the task failure data and the task failure data when the task-related data includes task failure data.
[0191] The embodied robot evaluation device provided in this application, when evaluating an embodied robot, can determine a multi-dimensional scoring vector for the embodied robot based on task-related data collected during task execution; then, based on the multi-dimensional scoring vector and the preset training and evaluation conditions corresponding to the task, it determines the generalization ability index value of the embodied robot; and it can determine the failure reason label based on the constraint relationship between task failure data and the failure reason label. Thus, by using the multi-dimensional scoring vector and failure reason label, specific failure reasons and types of insufficient transfer can be distinguished; and the generalization ability index value of the embodied robot can be obtained with only one test. This enriches the evaluation dimensions of embodied intelligent robots and simplifies the process of obtaining generalization ability evaluation.
[0192] One possible implementation is that the aforementioned task-related data includes: visual observation data, movement trajectory data, joint state data, end-effector pose data, contact force, collision events, task events, voice interaction data, and anomaly recovery logs.
[0193] Another possible implementation, the aforementioned determining module 902, is specifically used to: determine the task success retention rate, task quality score retention rate, safety constraint performance index value, and failure recovery performance index value of the embodied robot based on the multi-dimensional scoring vector, preset training conditions, and preset evaluation conditions; and determine the generalization ability index value based on the task success retention rate, task quality score retention rate, safety constraint performance index value, and failure recovery performance index value.
[0194] In another possible implementation, the aforementioned determining module 902 is further used to determine, after determining the failure reason label based on the constraint relationship between the task failure data and the task failure data, at least one of the following tasks based on the failure reason label and the task failure data: data re-collection task, simulation disturbance task, annotation verification task, and model optimization task.
[0195] In another possible implementation, the aforementioned embodied robot evaluation device further includes a storage module; the determining module 902 is further used to determine the comprehensive score of the embodied robot based on the multi-dimensional scoring vector of the embodied robot after determining the multi-dimensional scoring vector and the weights corresponding to each scoring vector in the multi-dimensional scoring vector, based on the task-related data and the preset task success conditions, preset safety constraints, and preset scoring indicators corresponding to the first task; the storage module is used to store the multi-dimensional scoring vector and the comprehensive score in the evaluation task record corresponding to the embodied robot.
[0196] It should be noted that the embodied robot evaluation device can realize all the processes implemented in the above method embodiments and achieve the same beneficial effects. To avoid repetition, it will not be described again here.
[0197] In the case where the functions of the integrated modules described above are implemented in hardware, this application provides a possible structural schematic diagram of the electronic device involved in the above embodiments. For example... Figure 10 As shown, the electronic device 90 includes: a processor 92, a communication interface 93, and a bus 94. Optionally, the electronic device 90 may also include a memory 91.
[0198] Processor 92 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 92 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 92 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.
[0199] Communication interface 93 is used to connect with other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.
[0200] The memory 91 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.
[0201] As one possible implementation, the memory 91 can exist independently of the processor 92. The memory 91 can be connected to the processor 92 via a bus 94 and is used to store instructions or program code. When the processor 92 calls and executes the instructions or program code stored in the memory 91, it can implement the embodied robot evaluation method provided in the embodiments of this application.
[0202] In another possible implementation, memory 91 can also be integrated with processor 92.
[0203] Bus 94 can be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 94 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0204] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the service calling device can be divided into different functional modules to complete all or part of the functions described above.
[0205] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described embodiments of the embodied robot evaluation method and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0206] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.
[0207] This application also provides a readable storage medium storing a program or instructions that, when executed by a computer, implement the embodied robot evaluation method provided in the above embodiments. It is understood that all or part of the processes in the above method embodiments can be executed by computer instructions instructing related hardware; the readable storage medium can be any of the foregoing embodiments or memory; the readable storage medium can also be an external storage device of the service invocation device, such as a pluggable hard drive, SmartMedia Card (SMC), Secure Digital (SD) card, flash card, etc., equipped on the service invocation device. Further, the readable storage medium can include both internal storage units of the service invocation device and external storage devices. The readable storage medium is used to store the computer program and other programs and data required by the service invocation device. The readable storage medium can also be used to temporarily store data that has been output or will be output.
[0208] This application also provides a computer program product, which is stored in a storage medium and, when executed by a computer, implements the embodied robot evaluation method provided in the above embodiments.
[0209] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0210] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0211] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. A method for evaluating embodied robots, characterized in that, include: During the process of the embodied robot performing its first task, task-related data is collected; Based on the task-related data and the preset task success conditions, preset safety constraints, and preset scoring indicators corresponding to the first task, a multi-dimensional scoring vector for the embodied robot is determined; wherein, the multi-dimensional scoring vector includes a task success scoring vector, a time efficiency scoring vector, a trajectory efficiency scoring vector, a grasping stability scoring vector, a homing accuracy scoring vector, a collision safety scoring vector, a risk avoidance scoring vector, a failure recovery scoring vector, and a human-robot interaction scoring vector. Based on the multi-dimensional scoring vector and the preset training conditions and preset evaluation conditions corresponding to the first task, the generalization ability index value of the embodied robot is determined. If the task-related data includes task failure data, a failure reason label is determined based on the constraints between the task failure data and the task failure data.
2. The method for evaluating embodied robots according to claim 1, characterized in that, The task-related data includes: visual observation data, movement trajectory data, joint status data, end-effector pose data, contact force, collision events, task events, voice interaction data, and anomaly recovery logs.
3. The method for evaluating embodied robots according to claim 1, characterized in that, The step of determining the generalization ability index value of the embodied robot based on the multi-dimensional scoring vector and the preset training conditions and preset evaluation conditions corresponding to the first task includes: Based on the multi-dimensional scoring vector, the preset training conditions, and the preset evaluation conditions, the task success retention rate, task quality score retention rate, safety constraint performance index value, and failure recovery performance index value of the embodied robot are determined. The generalization capability index value is determined based on the task success retention rate, the task quality score retention rate, the security constraint performance index value, and the failure recovery performance index value.
4. The method for evaluating an embodied robot according to any one of claims 1 to 3, characterized in that, After determining the failure reason label based on the constraint relationship between the task failure data and the task failure data, the embodied robot evaluation method further includes: Based on the failure reason label and the task failure data, at least one of the following tasks is determined: data re-collection task, simulation disturbance task, annotation and verification task, and model optimization task.
5. The method for evaluating an embodied robot according to any one of claims 1 to 3, characterized in that, After determining the multi-dimensional scoring vector of the embodied robot based on the task-related data and the preset task success conditions, preset safety constraints, and preset scoring indicators corresponding to the first task, the embodied robot evaluation method further includes: The comprehensive score of the embodied robot is determined based on the multi-dimensional scoring vector and the weights corresponding to each scoring vector in the multi-dimensional scoring vector. The multi-dimensional scoring vector and the comprehensive score are stored in the evaluation task record corresponding to the embodied robot.
6. A device for evaluating embodied robots, characterized in that, include: The data acquisition module and the determination module; The data acquisition module is used to collect task-related data during the process of the embodied robot performing its first task; The determining module is used to determine a multi-dimensional scoring vector for the embodied robot based on the task-related data and the preset task success conditions, preset safety constraints, and preset scoring indicators corresponding to the first task; wherein, the multi-dimensional scoring vector includes a task success scoring vector, a time efficiency scoring vector, a trajectory efficiency scoring vector, a grasping stability scoring vector, a homing accuracy scoring vector, a collision safety scoring vector, a risk avoidance scoring vector, a failure recovery scoring vector, and a human-robot interaction scoring vector. The determining module is further configured to determine the generalization ability index value of the embodied robot based on the multi-dimensional scoring vector and the preset training conditions and preset evaluation conditions corresponding to the first task. The determining module is further configured to, when the task-related data includes task failure data, determine the failure reason label based on the constraint relationship between the task failure data and the task failure data.
7. The embodied robot evaluation device according to claim 6, characterized in that, The task-related data includes: visual observation data, movement trajectory data, joint status data, end-effector pose data, contact force, collision events, task events, voice interaction data, and anomaly recovery logs.
8. The embodied robot evaluation device according to claim 6, characterized in that, The determining module is specifically used for: Based on the multi-dimensional scoring vector, the preset training conditions, and the preset evaluation conditions, the task success retention rate, task quality score retention rate, safety constraint performance index value, and failure recovery performance index value of the embodied robot are determined. The generalization capability index value is determined based on the task success retention rate, the task quality score retention rate, the security constraint performance index value, and the failure recovery performance index value.
9. The embodied robot evaluation apparatus according to any one of claims 6 to 8, characterized in that, The determining module is further configured to, after determining the failure reason label based on the constraint relationship between the task failure data and the task failure data, determine at least one of the following tasks based on the failure reason label and the task failure data: data acquisition task, simulation disturbance task, annotation and verification task, and model optimization task.
10. The embodied robot evaluation apparatus according to any one of claims 6 to 8, characterized in that, The embodied robot evaluation device also includes a storage module; The determining module is further configured to determine the comprehensive score of the embodied robot based on the multi-dimensional scoring vector of the embodied robot after determining the multi-dimensional scoring vector of the embodied robot based on the task-related data and the preset task success conditions, preset safety constraints and preset scoring indicators corresponding to the first task; The storage module is used to store the multi-dimensional scoring vector and the comprehensive score in the evaluation task record corresponding to the embodied robot.
11. An electronic device, characterized in that, It includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the embodied robot evaluation method as described in any one of claims 1-5.
12. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a computer, implement the embodied robot evaluation method as described in any one of claims 1-5.
13. A computer program product, characterized in that, The computer program product is stored in a storage medium, and when executed by a computer, the computer program product implements the embodied robot evaluation method as described in any one of claims 1-5.
