Evaluation method, system and device for body model, medium and program product

Abstract

The invention provides an evaluation method, system and device for a body model, a medium and a program product. The evaluation method comprises the steps of obtaining task description information; the task description information is input into a first large language model, a task instruction set used for controlling an execution mechanism is generated by the first large language model, and the task instruction set comprises at least one task instruction and associated information thereof; the task instruction set and the simulation configuration parameters are input into the second large language model, and an evaluation configuration file matched with the simulation configuration parameters is generated; and inputting the evaluation configuration file into the simulator, instructing the to-be-tested body model to control the robot to execute the task instruction set in the simulator, and generating an evaluation result of the to-be-tested body model. And full-process automatic closed-loop, large-scale and high-precision evaluation from task description information to evaluation configuration and evaluation execution is realized through a large language model.

Description

Technical Field

[0001] This disclosure relates to methods for evaluating models, and more particularly to a method, system, device, medium, and program product for evaluating embodied models. Background Technology

[0002] In the field of embodied intelligence, performance evaluation of Visual-Language-Action (VLA) models is one of the core research directions, with a particular focus on assessing the generalization ability and robustness of such models in complex interactive scenarios. To achieve efficient and secure evaluation, researchers generally use simulation environments to replace real-world scenarios. By generating diverse task scenarios for the agent, they simulate real interaction processes and quantitatively evaluate the agent's behavioral performance. Existing simulation evaluation tools include Habitat, iGibson, Isaac Gym, and PyBullet, which have been widely used in embodied intelligence evaluation.

[0003] However, existing simulation and evaluation schemes for embodied intelligent VLA models still have many technical shortcomings, making it difficult to meet the requirements of large-scale, high-precision, and automated evaluation. Specifically, these shortcomings are reflected in the following two aspects:

[0004] On the one hand, the design and execution of assessment tasks heavily rely on manual intervention, resulting in high assessment costs and poor scalability. For example, in the current assessment process, core aspects such as the construction of task sets, manual annotation of task success conditions, repeated execution of test cases on simulation platforms or real devices, and success rate statistics all require full manual involvement. This not only consumes a large amount of manpower and time, but also results in manually designed task scenarios that lack diversity and are highly subjective, making it difficult to cover various variations of complex interaction scenarios and failing to fully reflect the model's generalization ability and robustness, thus restricting the expansion of assessment scale and the objectivity of assessment results.

[0005] On the other hand, existing simulation evaluation tools have functional limitations, high learning and usage costs, and lack automated evaluation capabilities. In particular, most existing tools are static benchmarks or rely on manually written scripts, which can only support evaluation of fixed scenarios and single task types. They cannot achieve automated generation of diverse natural language task descriptions, adaptive construction of scenario variations, or dynamic adjustment and adaptive execution of the evaluation process. Summary of the Invention

[0006] The technical problem to be solved by this disclosure is to overcome the shortcomings of existing technologies in meeting the requirements of large-scale, high-precision, and automated testing, and to provide a testing method, system, device, medium, and program product for embodied models.

[0007] This disclosure solves the above-mentioned technical problems through the following technical solution:

[0008] This disclosure provides a method for evaluating embodied models, wherein the embodied model under test is used to control the actuator of a robot, and the evaluation method includes:

[0009] Get task description information;

[0010] The task description information is input into the first large language model, and the first large language model generates a task instruction set for controlling the actuator. The task instruction set includes at least one task instruction and its associated information.

[0011] Input the task instruction set and simulation configuration parameters into the second language model to generate an evaluation configuration file that matches the simulation configuration parameters;

[0012] The evaluation configuration file is input into the simulator, which instructs the embodied model under test to control the robot in the simulator to execute the task instruction set and generate the evaluation results of the embodied model under test.

[0013] Optionally, obtaining the task description information includes:

[0014] Obtain a natural language task description;

[0015] The natural language task description is input into the task parsing model, and the task parsing model parses the semantics of the natural language task description to obtain task scenario information and task constraint information.

[0016] The task parsing model calls the scene asset library based on the task scene information to generate the evaluation environment generation file;

[0017] A simulation testing environment instance is constructed based on the test environment generation file.

[0018] The simulation evaluation environment instance and the task constraint information are used as the task description information.

[0019] Optionally, the step of generating a set of task instructions for controlling the actuator includes:

[0020] The first large language model generates a first task instruction based on the task description information;

[0021] The first task instruction is input into the first large language model, and the first large language model generates the second task instruction based on the expansion rules.

[0022] The first task instruction and the second task instruction have the same task type but different description granularity;

[0023] The first task instruction and the second task instruction are used as the task instruction set.

[0024] Optionally, the task instruction set is generated in JSON (JavaScript Object Notation); and / or,

[0025] The evaluation configuration file is generated in JSON or DSL (Domain-Specific Language).

[0026] Optionally, the simulator includes a rule-based evaluation system, and the step of obtaining the evaluation result of the embodied model under test based on the evaluation rules includes:

[0027] The simulator acquires the execution status information of the embodied model under test;

[0028] The execution status information is verified based on the initial check rules of the rule evaluation system.

[0029] If the execution status information exceeds the coverage of the initial check rule, the rule evaluation system calls the extended rule resource to generate an extended check rule that matches the execution status information.

[0030] The rule evaluation system combines the initial check rule and the extended check rule to check the execution status information and generate the evaluation result.

[0031] Optionally, after generating extended detection rules for matching the evaluation configuration file, the rule evaluation system further includes:

[0032] If the execution status information exceeds the coverage of the extended inspection rules, an image of the robot's execution process is obtained.

[0033] The execution process image and the corresponding task instruction set are input into the visual language model to generate the evaluation result.

[0034] Optionally, the evaluation method further includes:

[0035] The test embodied model is controlled to execute the task instruction set in the simulator to generate structured simulation log information;

[0036] The simulation log information includes environmental errors of the simulation environment and execution errors of the robot.

[0037] In response to the environmental error exceeding the environmental error threshold and / or the execution error exceeding the execution error threshold, the simulation log information is sent to the second language model, which then updates the evaluation configuration file based on the simulation log information.

[0038] Optionally, the evaluation method further includes:

[0039] A visual assessment presentation file is generated based on the assessment results.

[0040] This disclosure also provides an evaluation system for embodied models, wherein the embodied model under test is used to control the actuator of a robot, and the evaluation system includes an information acquisition module, an instruction set generation module, an evaluation configuration generation module, and an evaluation execution module;

[0041] The information acquisition module is used to acquire task description information;

[0042] The instruction set generation module is used to input the task description information into a first language model, and the first language model generates a task instruction set for controlling the execution mechanism. The task instruction set includes at least one instruction and its associated information.

[0043] The evaluation configuration generation module is used to input the task instruction set and simulation configuration parameters into the second language model to generate an evaluation configuration file that matches the simulation configuration parameters.

[0044] The evaluation execution module is used to input the evaluation configuration file into the simulator, instruct the embodied model under test to control the robot in the simulator to execute the task instruction set and generate the evaluation result of the embodied model under test.

[0045] This disclosure also provides an electronic device including a memory, a processor, and a computer program stored in the memory and for running on the processor, wherein the processor executes the computer program to implement the evaluation method for embodied models as described above.

[0046] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described evaluation method for embodied models.

[0047] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the evaluation method for embodied models as described above.

[0048] Based on common knowledge in the field, the above-mentioned preferred conditions can be combined arbitrarily to obtain various preferred embodiments of this disclosure.

[0049] The positive and progressive effects of this disclosure are as follows: by using a large language model, a closed-loop automation of the entire process from task description information to assessment configuration and assessment execution is achieved. The large language model generates natural, rich, and diverse first and second task instructions based on natural language instructions, which take into account both logical and common sense expressions, to obtain reasonable and highly generalizable prompts. This replaces manual design of task sets with automation, improves the assessment efficiency of the embodied model to be tested, and enables diversified, large-scale, and high-precision assessments. Attached Figure Description

[0050] Figure 1 A flowchart illustrating an exemplary embodiment of this disclosure for evaluating an embodied model;

[0051] Figure 2 A flowchart of step S101 in an evaluation method for an embodied model provided as an exemplary embodiment of this disclosure;

[0052] Figure 3 A flowchart of step S102 in an evaluation method for an embodied model provided as an exemplary embodiment of this disclosure;

[0053] Figure 4 A flowchart of step S104 in an evaluation method for an embodied model provided as an exemplary embodiment of this disclosure;

[0054] Figure 5 A schematic diagram of a module for evaluating an embodied model, provided as an exemplary embodiment of this disclosure.

[0055] Figure 6 This is a schematic diagram of the structure of a computer device provided for an exemplary embodiment of the present disclosure. Detailed Implementation

[0056] The present disclosure is further illustrated below by way of embodiments, but the present disclosure is not limited to the scope of the embodiments described herein.

[0057] The prefixes such as "first" and "second" used in this disclosure are merely for distinguishing different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in this disclosure does not constitute a limitation on the described objects. The description of the described objects is given in the claims or the context of the embodiments, and should not be construed as an unnecessary limitation. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0058] In this embodiment of the disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information comply with relevant laws and regulations and do not violate public order and good morals.

[0059] Example 1

[0060] Figure 1 A flowchart of a method provided as an exemplary embodiment of this disclosure. This embodiment provides an evaluation method for embodied models, such as... Figure 1 As shown, the embodied model under test is used to control the actuators of the robot, and the evaluation methods include:

[0061] S101. Obtain task description information;

[0062] The task description information includes task scenario association information and task execution association information;

[0063] Specifically, the task description information can be in the form of the user's natural language instructions, and usually includes at least one task scenario-related information, such as instruction information defining the task type.

[0064] S102. Input the task description information into the first language model, and the first language model generates a task instruction set for controlling the actuator. The task instruction set includes at least one instruction and its associated information.

[0065] Specifically, the first and second large language models can be the same or different models, such as the API (Application Programming Interface) of public large language models like DeepSeek, Qianwen, or Seed.

[0066] In one embodiment, based on the input task description information and associated information, and based on the task execution associated information in the task description information, such as controlling the right robotic arm to grab the largest red fruit on the table and place it on the empty tray on the right side of the table, the large language model can decompose the task description information into task generation instructions, such as grabbing type instructions, scene parameter instructions, target selection instructions, and execution step instructions, as well as evaluation rules that match the task generation instructions, including grabbing type matching degree, target selection accuracy, grabbing success rate, and placement position deviation. Therefore, based on the prompts corresponding to the decomposed task instructions and evaluation rules, the prompts related to the task generation instructions are used as the task instruction set.

[0067] S103. Input the task instruction set and simulation configuration parameters into the second language model to generate an evaluation configuration file that matches the simulation configuration parameters.

[0068] Specifically, the robot parameters controlled by the embodied model under test are obtained from the database. Based on the task instructions, associated information, simulation configuration parameters and robot parameters in the task instruction set, a prompt including instruction information, robot parameters, evaluation rules and generation constraints is constructed. The simulation configuration parameters can be based on the requirements of the embodied model under test and the relevant parameters of the robot.

[0069] S104. Input the evaluation configuration file into the simulator, instruct the embodied model under test to control the robot in the simulator to execute the task instruction set and generate the evaluation results of the embodied model under test.

[0070] In this solution, a large language model is used to achieve a fully automated closed loop from user natural language commands to test configuration and test execution. The large language model generates natural, rich, and diverse first and second task commands based on natural language commands, which take into account both logic and common sense. This results in reasonable and highly generalizable prompts, which replace manual design of task sets with automation, improve the test efficiency of the embodied model under test, and achieve diversified, large-scale, and high-precision tests.

[0071] As a feasible approach, such as Figure 2 As shown, step S101 includes:

[0072] S1011. Obtain the natural language task description;

[0073] S1012. Input the natural language task description into the task parsing model, and the task parsing model parses the semantics of the natural language task description to obtain task scenario information and task constraint information.

[0074] S1013. The task parsing model calls the scene asset library based on the task scene information to generate the evaluation environment generation file;

[0075] Specifically, the task parsing model can adopt a large language model or a visual language model. Based on the task scene information, it retrieves and matches the corresponding scene and the objects bound to the scene in the scene asset library, and generates a simulation evaluation environment instance based on the retrieval and matching results.

[0076] In one embodiment, if the task description is "grabbing a red object on a table with the right hand," the task scenario association information includes the table, the red object, and the right hand. Based on the task scenario association information, the interface of the scenario asset library is called to obtain the corresponding table model, the red object model, and the robot's right hand model. The large language model then generates a simulation evaluation environment instance based on the corresponding assets retrieved from the scenario asset library, which is used to call the simulator interface to build the evaluation environment.

[0077] S1014. Example of constructing a simulation testing environment based on the test environment generation file;

[0078] Specifically, the simulator interface is called to parse the evaluation environment file. This parses the scene parameters of the desktop model within the evaluation environment file, such as desktop coordinates, the friction coefficient of the desktop, and the coordinates of the area on the desktop corresponding to the robot's right hand; and the parameters of the robot's right hand model, such as the robot's joint range, initial pose, and object parameters. Static scene elements, the robot model, the desktop model, the apple model bound to the desktop model, and interference objects are added based on the evaluation requirements to initialize the physical rules of the evaluation environment and generate a simulation evaluation environment instance that matches the evaluation environment file.

[0079] S1015. Use simulation test environment instances and task constraint information as task description information.

[0080] As a feasible approach, such as Figure 3 As shown, step S102, which generates a set of task instructions for controlling the actuator, includes:

[0081] S1021. The first task instruction is generated by the first language model based on the task description information;

[0082] Specifically, the first language model parses the semantics of the task description information to generate the first task instruction. The first task instruction includes information related to the task type, the range of which is smaller than the types of simulation tasks that the simulator can implement.

[0083] S1022. Input the first task instruction into the first large language model, and the first large language model generates the second task instruction based on the extended rules.

[0084] Among them, the first task instruction and the second task instruction have the same task type, but different description granularity;

[0085] Specifically, in fields such as computer science, artificial intelligence, and evaluation technology, granularity refers to the level of detail or abstraction of data, information, descriptions, or systems. Descriptive granularity refers to the level of detail, scope, and abstraction of a description of an object or task.

[0086] In one embodiment, the expansion rules include at least one of semantic variant generation, parameterized randomization, and semantic-level transformation. For example, semantic variant generation expands "grab an apple on the table" to "pick up a red apple on the table" or "grab an apple on the table to a specified position," thus providing different linguistic descriptions of the task. For example, parameterized randomization expands "grab an apple with a diameter of 5cm" to "grab an apple with a diameter of 4-8cm" or "grab an apple whose position is randomly distributed on the table x∈[0.2-0.5] m," thus incorporating object parameters, positions, and environmental conditions in the scene. For example, semantic-level transformation expands "grab an apple on the table" to "grab a red round fruit on the table" or "grab an edible round object on the table," thus broadening the distribution of target categories.

[0087] 1023. The first task instruction and the second task instruction are used as the task instruction set.

[0088] In this scheme, the second task instruction is obtained by extending the rules, so as to improve the ability of the embodied model under test to understand different natural language expressions of the same task, adapt to the fluctuations of object parameters, positions and environmental conditions in the scene, and transfer from specific target distribution to generalized target distribution.

[0089] As one possible approach, the task instruction set is generated in JSON format; and / or,

[0090] The structured format of the assessment configuration file is JSON or DSL.

[0091] Specifically, the first task instructions and associated information are generated synchronously and structurally by the first major language model, rather than being designed separately and manually matched. This ensures that the core metrics of the evaluation rules, such as the success rate of grasping, the accuracy of target recognition, and the collision threshold, accurately anchor the core semantics of the first task instructions, such as the various types of instructions in "grab the red apple in a fixed position." This avoids the problem of rules and instructions becoming disconnected from each other at the source, such as instructions focusing on grasping accuracy while rules overemphasize irrelevant movement speed metrics. This ensures that the final evaluation results truly reflect the model's performance in executing the corresponding task.

[0092] Additionally, the structured first task instructions are input into the large language model, which then generates the second task instructions in a structured manner.

[0093] Specifically, by inputting structured first task instructions, the expansion direction of second task instructions can avoid biases introduced by manual translation, ensuring that the differences between the second and first task instructions are clear and the comparison dimensions are well-defined. This enables the expanded second task instructions to accurately support the evaluation objectives of natural language diversity, scene perturbation, and distributional migration, avoiding meaningless instruction variations.

[0094] Additionally, the assessment configuration file is generated in a structured manner from the second largest language model.

[0095] Specifically, by generating assessment configuration files in a structured manner through a large language model, the consistency of semantics and parameter quantification throughout the input and output process is ensured. This replaces the traditional mode of manually writing cumbersome configuration scripts, significantly improves the automation level of assessment task design, reduces manual costs and tool learning barriers, and the generated structured configuration files can be directly connected to the simulator interface.

[0096] In this solution, the large language model generates task instruction sets and initial evaluation configurations in a structured format. The output content features unified fields, quantified parameters, and clear logic, allowing for direct parsing and execution by the simulator. This avoids parameter bias and semantic ambiguity issues introduced during manual translation of instructions or rules. Simultaneously, the structured data facilitates archiving and traceability, providing a standardized basis for subsequent comparative analysis of evaluation results. The use of the universal structured format JSON effectively adapts to the parsing and generation logic of the large language model; the adoption of the structured format DSL allows for precise definition of specific rules in areas such as environmental physical parameters and robot motion constraints, meeting the customization needs of embodied models in simulation evaluation scenarios. This improves the efficiency of automated integration from instruction input to configuration generation.

[0097] As one possible approach, simulators include rule evaluation systems, such as... Figure 4 As shown, step S104, the step of obtaining the evaluation result of the embodied model to be tested based on the evaluation rules, includes:

[0098] S1041. The simulator acquires the execution status information of the embodied model under test;

[0099] Specifically, the execution status information adopts a structured data format, including mechanical structural parameters such as joint rotation angle, joint angular velocity, end effector pose and clamping force, dynamic parameters such as joint load torque, driving torque, system energy consumption, motion timing parameters such as action execution time and dual-arm coordination time difference, environmental interaction parameters such as workpiece contact pressure and workspace coordinates, and status indicators such as normal operation or abnormal alarm.

[0100] S1042. The execution status information is verified based on the initial check rules of the rule evaluation system;

[0101] Specifically, the rule evaluation system has a built-in structured initial check rule library. The rule engine traverses the initial check rule library and matches the preprocessed execution status information with each rule condition field by field. For numerical parameters, it verifies whether they are within the threshold range; for logical parameters, it verifies whether they meet the constraint relationship. After the matching is completed, the initial verification result is generated.

[0102] S1043. In response to the execution status information exceeding the coverage of the initial check rule, the rule evaluation system calls the extended rule resource to generate extended check rules that match the execution status information.

[0103] Specifically, the rule evaluation system quantifies the initial verification results. If the percentage of execution status information that does not match the corresponding initial check rule, or if undefined status features are detected, it is determined that the results exceed the coverage of the initial check rule.

[0104] The system calls the extended rule template library and automatically populates the dynamic parameters in the templates based on the execution status information characteristics that exceed the scope. For special scenarios where no corresponding template exists, the system can prompt manual configuration of the interface and output a status information characteristic report for manual input of extended check rule parameters. Simultaneously, the generated extended check rules undergo syntax and conflict checks to ensure that there are no conflicts between the extended check rules and the initial check rules.

[0105] S1044. The evaluation system, in conjunction with the initial and extended inspection rules, checks the execution status information and generates evaluation results.

[0106] Specifically, the execution status information is fully traversed and checked using the initial check rules and the extended check rules: for parameters covered by the initial rules, the initial verification results are used; for parameters that exceed the initial range, the extended rules are used for verification, and the application of the extended rules is marked.

[0107] After verification, the rule evaluation system integrates the verification data and generates structured evaluation results, including evaluation timestamps, identifiers of the embodied model to be tested, initial rule verification results, extended rule verification results, and overall compliance scores. The evaluation results are stored in a specified structured format and synchronized to the simulator's result output interface for subsequent analysis.

[0108] In this solution, the initial inspection rules in the rule evaluation system are efficiently and accurately verified. For scenarios where the initial rules are insufficiently covered, the extended template is called to automatically adapt and generate extended inspection rules. The joint verification of the two types of rules is performed, and the structured evaluation results containing detailed verification information and compliance rating are output. This ensures that the rule evaluation system fully covers the complex execution states of the embodied model, while taking into account the automation efficiency of rule extension and the adaptability and flexibility of special scenarios, thereby improving the adaptability and reliability of embodied model simulation evaluation.

[0109] As one possible approach, step S1043 may be followed by:

[0110] If the execution status information exceeds the coverage of the extended inspection rules, an image of the robot's execution process is acquired.

[0111] Specifically, the rule evaluation system performs secondary verification on the application results of the extended inspection rules. When the extended inspection rules cannot match the execution status information, the rule evaluation system sends an image acquisition command to the simulator. The simulator then activates its built-in virtual camera module to acquire images of the robot's head, left and right hands, and third-person perspective of the execution process.

[0112] The execution process image and the corresponding task instruction set are input into the visual language model to generate the evaluation result.

[0113] Specifically, the visual language model can be deployed locally in the simulator, or it can be configured with a calling interface for the visual language model. When the extended check rules cannot match the execution status information, the rule evaluation system calls the visual language model interface to generate the evaluation results.

[0114] The visual language model understands the task objective by parsing the semantics of the task instruction set. The task objective includes metrics such as task success rate and completion time. By parsing the semantics of the execution process images, the actual execution status of the robot is obtained, and corresponding evaluation results are generated based on the task objective and the actual execution status.

[0115] In this solution, when all the check rules of the rule evaluation system fail, the execution result image of the embodied model under test is used as an alternative means. By utilizing the cross-modal understanding capability of the VLM model, the task completion status can be accurately determined, thereby improving the scenario adaptability, judgment comprehensiveness and result reliability of the embodied model simulation evaluation.

[0116] As an feasible approach, assessment methods also include:

[0117] Control the embodied model under test to execute a set of task instructions in the simulator and generate structured simulation log information;

[0118] The simulation log information includes environmental errors of the simulation environment and execution errors of the robot.

[0119] Specifically, the first or second task instruction is sent to the embodied model under test through the interface, driving the embodied model under test to execute the corresponding action sequence in the simulation evaluation environment instance, recording environmental errors and execution errors in real time, and generating formatted simulation log information.

[0120] Environmental errors include deviations between the actual coordinates of scene elements and the configuration file, object penetration, object suspension, and fluctuations in lighting intensity. Execution errors include the number of robot collisions, the number of joint overruns, the time taken to complete the task, and the target misgrabbing rate.

[0121] If the environmental error exceeds the environmental error threshold and / or the execution error exceeds the execution error threshold, the simulation log information is sent to the second language model, which then updates the evaluation configuration file based on the simulation log information.

[0122] In this solution, environmental errors and execution errors are quantitatively distinguished through structured simulation logs, achieving accurate correspondence between calibration objects and error types, avoiding blind calibration, improving the stability and reliability of the evaluation process, realizing automated monitoring of model instruction execution, significantly reducing the cost of manual intervention, and forming a complete error record and calibration trajectory.

[0123] As an feasible approach, assessment methods also include:

[0124] A visual assessment presentation file is generated based on the assessment results.

[0125] Specifically, the format and display type of the testable files can be configured according to the user's actual needs. For example, multi-dimensional dashboards and bar charts comparing the success rates of two commands can be generated to intuitively present the execution differences between the first and second task commands and core indicators such as the execution compliance rate. Alternatively, 3D trajectory diagrams of robot movements, scatter plots of mapping deviations, error time-series change diagrams, and video files can be generated to present the robot's execution process in chronological order.

[0126] In this solution, structured data association and coding rules ensure that the visualization results are deeply bound to the evaluation instructions and configuration parameters, which not only improves the efficiency of evaluation closed-loop optimization, but also provides an intuitive and reliable presentation carrier for the evaluation results of the embodied model to be tested.

[0127] The evaluation method for embodied models provided in this embodiment achieves a fully automated closed loop from user natural language commands to evaluation configuration and execution through a large language model. It utilizes the large language model to generate natural, rich, and diverse first and second task commands based on natural language commands, taking into account both logic and common sense, to obtain reasonable and highly generalizable prompts. This automated approach replaces manual task design, improves the evaluation efficiency of the embodied models under test, and enables diversified, large-scale, and high-precision evaluation.

[0128] Example 2

[0129] Corresponding to the foregoing embodiments of the evaluation method for embodied models, this disclosure also provides embodiments of the evaluation system for embodied models.

[0130] Figure 5 This is a schematic diagram of a module for evaluating an embodied model, provided as an exemplary embodiment of the present disclosure. The embodied model under test is used to control the actuators of a robot, such as... Figure 5As shown, the evaluation system 100 includes an information acquisition module 101, an instruction set generation module 102, an evaluation configuration generation module 103, and an evaluation execution module 104;

[0131] Information acquisition module 101 is used to acquire task description information;

[0132] The initial instruction generation module 102 is used to input task description information into the first language model, and the first language model generates a task instruction set for controlling the execution mechanism. The task instruction set includes at least one instruction and its associated information.

[0133] The evaluation configuration generation module 103 is used to input the task instruction set and simulation configuration parameters into the second language model and generate an evaluation configuration file that matches the simulation configuration parameters.

[0134] The evaluation execution module 104 is used to input the evaluation configuration file into the simulator, instruct the embodied model under test to control the robot in the simulator to execute the task instruction set and generate the evaluation results of the embodied model under test.

[0135] As one possible implementation, the information acquisition module 101 includes a description information acquisition unit, a semantic analysis unit, a resource allocation unit, a simulation environment generation unit, and a task information generation unit:

[0136] The description information acquisition unit is used to acquire the description of the natural language task.

[0137] The semantic analysis unit is used to input the natural language task description into the task parsing model, and the task parsing model parses the semantics of the natural language task description to obtain task scenario information and task constraint information.

[0138] The resource allocation unit is used by the task parsing model to call the scene asset library based on the task scene information and generate the evaluation environment generation file.

[0139] The simulation environment generation unit is used to construct simulation evaluation environment instances based on the evaluation environment generation file.

[0140] The task information generation unit is used to generate task description information using simulation evaluation environment instances and task constraint information.

[0141] As one possible implementation, the initial instruction generation module 102 includes a first instruction generation unit, a second instruction generation unit, and an instruction set generation unit:

[0142] The first instruction generation unit is used to generate the first task instruction from the first large language model based on the task description information.

[0143] The second instruction generation unit is used to generate a corresponding second task instruction from the first large language model in response to the extended instruction of the first task instruction.

[0144] Among them, the first task instruction and the second task instruction have the same task type, but different description granularity;

[0145] The instruction set generation unit is used to use the first task instruction and the second task instruction as the task instruction set.

[0146] As one possible approach, the task instruction set is generated in JSON format; and / or,

[0147] The evaluation configuration file is generated in JSON or DSL format.

[0148] As one possible implementation, the simulator includes a rule evaluation system, and the evaluation execution module 104 includes a status information acquisition unit, an initial verification unit, an extended rule invocation unit, and an evaluation result generation unit.

[0149] The status information acquisition unit is used by the simulator to acquire the execution status information of the physical model under test;

[0150] The initial verification unit is used to verify the execution status information based on the initial check rules of the rule evaluation system;

[0151] The extended rule invocation unit is used to respond to situations where the execution status information exceeds the coverage of the initial check rule. In this case, the rule evaluation system invokes the extended rule resource to generate extended check rules that match the execution status information.

[0152] The evaluation result generation unit is used by the evaluation system to combine the initial check rules and extended check rules, check the execution status information, and generate evaluation results.

[0153] As one possible approach, the evaluation execution module 104 also includes an image acquisition unit:

[0154] An execution image acquisition unit is configured to acquire an image of the robot's execution process in response to the execution status information exceeding the coverage of the extended inspection rules.

[0155] The evaluation result generation unit is also used to input the execution process image and the corresponding task instruction set into the visual language model to generate evaluation results.

[0156] As one feasible approach, the evaluation system also includes a log generation module and an error calibration module:

[0157] It includes a log generation module, which controls the embodied model under test to execute a set of task instructions in the simulator and generate structured simulation log information;

[0158] The simulation log information includes environmental errors of the simulation environment and execution errors of the robot.

[0159] The error calibration module is used to send simulation log information to the second language model in response to environmental errors exceeding the environmental error threshold and / or execution errors exceeding the execution error threshold. The second language model then updates the evaluation configuration file based on the simulation log information.

[0160] As one feasible approach, the assessment system also includes a visualization module:

[0161] The visualization module is used to generate visual evaluation display files based on the evaluation results.

[0162] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs.

[0163] Example 3

[0164] Figure 6 This is a schematic diagram of the structure of an electronic device according to an example embodiment of the present disclosure. The electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor. When the processor executes the computer program, it implements the evaluation method for embodied models of any of the above embodiments. Figure 6 The electronic device 90 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0165] like Figure 6 As shown, the electronic device 90 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 90 may include, but are not limited to: at least one processor 91, at least one memory 92, and a bus 93 connecting different system components (including memory 92 and processor 91).

[0166] Bus 93 includes a data bus, an address bus, and a control bus.

[0167] The memory 92 may include volatile memory, such as random access memory (RAM) 921 and / or cache memory 922, and may further include read-only memory (ROM) 923.

[0168] The memory 92 may also include a program tool 925 (or utility) having a set (at least one) program module 924, such program module 924 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0169] The processor 91 executes various functional applications and data processing by running computer programs stored in the memory 92, such as the evaluation method for embodied models provided in any of the above embodiments.

[0170] Electronic device 90 can also communicate with one or more external devices 94 (e.g., keyboard, pointing device, etc.). This communication can be performed through input / output (I / O) interface 95. Furthermore, electronic device 90 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 96. As shown, network adapter 96 communicates with other modules of electronic device 90 via bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 90, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0171] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0172] Example 4

[0173] This disclosure also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the evaluation method for embodied models provided in any of the above embodiments.

[0174] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0175] Example 5

[0176] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the evaluation method for embodied models described above.

[0177] The program code for executing the computer program product disclosed herein can be written in any combination of one or more programming languages. The program code can be executed entirely on a user device, partially on a user device, as a stand-alone software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0178] While specific embodiments of this disclosure have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this disclosure is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this disclosure, but all such changes and modifications fall within the scope of protection of this disclosure.

Claims

1. A method for evaluating embodied models, characterized in that, The embodied model under test is used to control the actuators of the robot, and the evaluation method includes: Get task description information; The task description information is input into the first language model, and the first language model generates a task instruction set for controlling the actuator. The task instruction set includes at least one task instruction and its associated information. Input the task instruction set and simulation configuration parameters into the second language model to generate an evaluation configuration file that matches the simulation configuration parameters; The evaluation configuration file is input into the simulator, which instructs the embodied model under test to control the robot in the simulator to execute the task instruction set and generate the evaluation results of the embodied model under test.

2. The evaluation method according to claim 1, characterized in that, The acquisition of task description information includes: Obtain a natural language task description; The natural language task description is input into the task parsing model, which then parses the semantics of the natural language task description to obtain task scenario information and task constraint information. The task parsing model calls the scene asset library based on the task scene information to generate the evaluation environment generation file; A simulation testing environment instance is constructed based on the test environment generation file. The simulation evaluation environment instance and the task constraint information are used as the task description information.

3. The evaluation method according to claim 1, characterized in that, The step of generating a task instruction set for controlling the actuator includes: The first large language model generates a first task instruction based on the task description information; In response to the extended instructions of the first task instruction, the first large language model generates the corresponding second task instruction; The first task instruction and the second task instruction have the same task type but different description granularity; The first task instruction and the second task instruction are used as the task instruction set.

4. The evaluation method according to claim 1, characterized in that, The task instruction set is generated in JSON format; and / or, The evaluation configuration file is generated in JSON or DSL format.

5. The evaluation method according to claim 1, characterized in that, The simulator includes a rule-based evaluation system, and the step of generating the evaluation results for the embodied model under test includes: The simulator acquires the execution status information of the embodied model under test; The execution status information is verified based on the initial check rules of the rule evaluation system. If the execution status information exceeds the coverage of the initial check rule, the rule evaluation system calls the extended rule resource to generate an extended check rule that matches the execution status information. The rule evaluation system combines the initial check rule and the extended check rule to check the execution status information and generate the evaluation result.

6. The evaluation method according to claim 5, characterized in that, After generating extended detection rules for matching the evaluation configuration file, the rule evaluation system further includes: If the execution status information exceeds the coverage of the extended inspection rules, an image of the robot's execution process is obtained. The execution process image and the corresponding task instruction set are input into the visual language model to generate the evaluation result.

7. The evaluation method according to claim 1, characterized in that, The assessment method also includes: The test embodied model is controlled to execute the task instruction set in the simulator to generate structured simulation log information; The simulation log information includes environmental errors of the simulation environment and execution errors of the robot. In response to the environmental error exceeding the environmental error threshold and / or the execution error exceeding the execution error threshold, the simulation log information is sent to the second language model, which then updates the evaluation configuration file based on the simulation log information.

8. The evaluation method according to claim 1, characterized in that, The assessment method also includes: A visual assessment presentation file is generated based on the assessment results.

9. An evaluation system for embodied models, characterized in that, The embodied model under test is used to control the actuators of the robot. The evaluation system includes an information acquisition module, an instruction set generation module, an evaluation configuration generation module, and an evaluation execution module. The information acquisition module is used to acquire task description information; The instruction set generation module is used to input the task description information into a first language model, and the first language model generates a task instruction set for controlling the execution mechanism. The task instruction set includes at least one instruction and its associated information. The evaluation configuration generation module is used to input the task instruction set and simulation configuration parameters into the second language model to generate an evaluation configuration file that matches the simulation configuration parameters. The evaluation execution module is used to input the evaluation configuration file into the simulator, instruct the embodied model under test to control the robot in the simulator to execute the task instruction set and generate the evaluation result of the embodied model under test.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and for running on the processor, characterized in that, When the processor executes the computer program, it implements the evaluation method for embodied models as described in any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the evaluation method for embodied models as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the evaluation method for embodied models as described in any one of claims 1 to 8.

Images