Embodied intelligence model-oriented evaluation method and device, equipment and storage medium

Abstract

The application discloses a kind of evaluation methods, devices and equipment for embodied intelligent model, relating to artificial intelligence evaluation technical field, comprising: receiving evaluation task request, evaluation task request includes target benchmark information and constraint condition;Through the uniform handle protocol of sandbox plane, resource acquisition instruction carrying with simulator type identifier is sent to sandbox plane, and the simulator handle returned by sandbox plane is received;Through simulator handle and simulator back-end instance interact, to execute the evaluation task corresponding to evaluation task request, and obtain the evaluation trajectory data generated in execution process;Index data is obtained by index calculation to evaluation trajectory data, and evaluation result of to-be-tested model under the target evaluation benchmark indicated by target benchmark information is generated based on index data, by introducing sandbox plane and uniform handle protocol, the decoupling of evaluation upper layer logic and bottom simulator is realized, and efficient and reproducible embodied intelligent evaluation is realized.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence evaluation technology, and in particular to evaluation methods, apparatus, devices and storage media for embodied intelligence models. Background Technology

[0002] Currently, in the research and iteration of embodied AI technology, evaluating the visual-motor strategy layer and the whole-body stability and control layer is a crucial step in verifying algorithm performance. Existing evaluation schemes typically construct independent evaluation scripts and runtime environments for specific simulators or benchmarks. For example, navigation tasks rely on Habitat-Sim, operation tasks are scattered across different simulators such as MuJoCo and SAPIEN, and whole-body control evaluation may use other platforms such as Isaac Sim. This fragmented evaluation architecture leads to a lack of unified interaction protocols and data standards between different simulators, making it difficult to compare model performance across simulators and benchmarks in the same coordinate system. Integrating new benchmarks often requires reconstructing the entire evaluation system, resulting in high maintenance costs. Summary of the Invention

[0003] The main objective of this application is to provide an evaluation method, apparatus, device, and storage medium for embodied intelligence models, aiming to solve the technical problems in current solutions where simulator fragmentation leads to difficulties in cross-platform comparison and the inability to measure the difference between simulation and real machine.

[0004] To achieve the above objectives, this application proposes an evaluation method for embodied intelligence models, which includes: Receive an evaluation task request, the evaluation task request including target benchmark information and constraints, the constraints including at least a simulator type identifier; Through the unified handle protocol of the sandbox plane, a resource acquisition command carrying the simulator type identifier is sent to the sandbox plane, and the simulator handle returned by the sandbox plane is received. The simulator handle is used to uniquely identify the simulator backend instance called by the sandbox plane according to the simulator type identifier. The simulator handle interacts with the simulator backend instance to execute the evaluation task corresponding to the evaluation task request and obtain the evaluation trajectory data generated during the execution process. The evaluation trajectory data is used to calculate the index data, and the evaluation results of the model under test are generated based on the index data under the target evaluation benchmark indicated by the target benchmark information.

[0005] In one embodiment, the step of interacting with the simulator backend instance through the simulator handle to execute the evaluation task corresponding to the evaluation task request and obtaining the evaluation trajectory data generated during the execution includes: At the current time step of the evaluation round, obtain the unified action command output by the model under test; The unified action instructions are converted into native action instructions adapted to the simulator backend instance. The simulator handle sends the native action command to the simulator backend instance for interaction, so that the simulator backend instance executes the evaluation task corresponding to the evaluation task request based on the native action command, and obtains the evaluation trajectory data generated during the execution process.

[0006] In one embodiment, the step of converting the unified action command into a native action command adapted to the emulator backend instance includes: Call the preset interface mapping layer to obtain the motion space definition corresponding to the simulator backend instance, wherein the motion space definition includes the number of joint degrees of freedom and the range of motion values; Based on the defined action space, the unified action instructions are subjected to dimensional matching and numerical normalization to generate native action instructions that can be recognized by the simulator backend instance.

[0007] In one embodiment, the step of sending the native action command to the simulator backend instance via the simulator handle for interaction to execute the evaluation task corresponding to the evaluation task request, and obtaining the evaluation trajectory data generated during the execution process includes: The native action command is sent to the simulator backend instance through the simulator handle to drive the virtual object in the simulator backend instance to execute the evaluation task corresponding to the evaluation task request; Receive the raw observation data returned by the simulator backend instance after executing the evaluation task, and convert the raw observation data into unified observation data; The unified action instructions and the unified observation data are appended to the trajectory record in time step order until the round termination condition is met, thereby generating evaluation trajectory data.

[0008] In one embodiment, the step of sending a resource acquisition command carrying the simulator type identifier to the sandbox plane via the unified handle protocol of the sandbox plane, and receiving the simulator handle returned by the sandbox plane, includes: Parse the parallelism parameter in the resource acquisition instruction, wherein the parallelism parameter is used to specify the required number of parallel simulation processes; Through the unified handle protocol of the sandbox plane, a resource acquisition command carrying the parallelism parameter is sent to the sandbox plane, so that the sandbox plane starts the corresponding number of parallel simulation processes according to the simulator type identifier and returns the process group identifier. Receive the process group identifier returned by the sandbox plane, and encapsulate the process group identifier into an emulator handle.

[0009] In one embodiment, the step of calculating index data from the evaluation trajectory data and generating evaluation results of the model under test under the target evaluation benchmark indicated by the target benchmark information based on the index data includes: Based on the evaluation trajectory data, the first evaluation trajectory data of the simulation backend and the second evaluation trajectory data of the physical machine backend are obtained; Based on the first evaluation trajectory data and the second evaluation trajectory data, respectively, calculate the first indicator value of the same evaluation index in the simulation environment and the second indicator value in the real machine environment; Calculate the numerical deviation between the first indicator value and the second indicator value, and calculate the ranking consistency coefficient between the first indicator value and the second indicator value; The evaluation results of the model under test under the target evaluation benchmark indicated by the target benchmark information are obtained based on the numerical deviation and the ranking consistency coefficient.

[0010] In one embodiment, the step of calculating the ranking consistency coefficient between the first indicator value and the second indicator value includes: The first index value and the second index value of multiple models under test are ranked in the simulation environment to obtain a first ranking set and a second ranking set; Pair multiple models to be tested together to determine whether the ranking order of each model to be tested in the first ranking set is consistent with the ranking order in the second ranking set, and obtain the consistent logarithm and inconsistent logarithm. Calculate the Kendall rank correlation coefficient based on the consistent logarithm and the inconsistent logarithm; The ranking consistency coefficient between the first indicator value and the second indicator value is obtained based on the Kendall rank correlation coefficient.

[0011] In one embodiment, the step of calculating the ranking consistency coefficient between the first indicator value and the second indicator value includes: The first index value and the second index value of multiple models under test are ranked in the simulation environment to obtain a first ranking set and a second ranking set; The Spearman rank correlation coefficient is calculated based on the ranking difference of the same model under test in the first ranking set and the second ranking set. The ranking consistency coefficient between the first indicator value and the second indicator value is obtained based on the Spearman rank correlation coefficient.

[0012] In one embodiment, the step of calculating index data from the evaluation trajectory data and generating evaluation results of the model under test under the target evaluation benchmark indicated by the target benchmark information based on the index data includes: Based on the evaluation trajectory data corresponding to each model to be tested, calculate each evaluation index separately; The system associates and displays the identifiers of each model under test with their corresponding evaluation metrics, generating a cross-model evaluation leaderboard. Obtain the benchmark indicator data of each model under test in historical evaluations, and use the aforementioned evaluation indicators as reproducible indicator data; In the cross-model evaluation leaderboard, for the same model under test, the benchmark index data and the reproduction index data are displayed in parallel, and the evaluation results of each model under test under the target evaluation benchmark indicated by the target benchmark information are obtained based on the benchmark index data and the reproduction index data.

[0013] Furthermore, to achieve the above objectives, this application also proposes an evaluation device for embodied intelligence models, the device comprising a multi-layered architectural boundary, the evaluation device for embodied intelligence models comprising: The task adaptation module is used to receive evaluation task requests, which include target benchmark information and constraints, and the constraints include at least a simulator type identifier. The execution binding module is used to send a resource acquisition instruction carrying the simulator type identifier to the sandbox plane through the unified handle protocol of the sandbox plane, and receive the simulator handle returned by the sandbox plane, wherein the simulator handle is used to uniquely identify the simulator backend instance called by the sandbox plane according to the simulator type identifier. The interaction module is used to interact with the simulator backend instance through the simulator handle to execute the evaluation task corresponding to the evaluation task request and obtain the evaluation trajectory data generated during the execution process. The evaluation module is used to calculate the evaluation trajectory data to obtain the evaluation data, and generate the evaluation results of the model under test under the target evaluation benchmark indicated by the target benchmark information based on the evaluation data. The task adaptation module, the execution binding module, the interaction module, and the evaluation module are configured to run on their respective architectural boundaries and interact with each other through standardized interfaces.

[0014] Furthermore, to achieve the above objectives, this application also proposes an evaluation device for embodied intelligence models, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the evaluation method for embodied intelligence models as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the evaluation method for embodied intelligence models as described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the evaluation method for embodied intelligence models as described above.

[0017] This application proposes one or more technical solutions that, by constructing a unified interaction protocol and standardized data interface, adapt to the evaluation needs of different types of simulators and real machine platforms, incorporating model performance indicators and simulation-real machine gaps across simulators and benchmarks into a unified quantitative evaluation framework. Firstly, this solution encapsulates the interaction logic of different simulation environments through an abstraction layer, enabling various benchmark testing tasks to access the evaluation system through a unified interface. This eliminates the need to reconstruct the entire evaluation process for new benchmarks, significantly reducing system maintenance and expansion costs. Simultaneously, it establishes a unified indicator system covering all performance dimensions of the model, transforming the evaluation outputs of simulation and real machine environments into quantifiable results in the same format. This allows for the quantitative measurement and difference localization of the simulation-real machine gap during the evaluation phase, providing a clear quantitative reference for the iterative optimization of embodied intelligent models. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an embodiment of the evaluation method for embodied intelligence models in this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the evaluation method for embodied intelligence models in this application. Figure 3 This is a flowchart illustrating Embodiment 3 of the evaluation method for embodied intelligence models in this application; Figure 4 This is a schematic diagram of the module structure of the evaluation device for embodied intelligence models according to an embodiment of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the evaluation method for embodied intelligent models in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] The main solution of this application embodiment is as follows: receiving an evaluation task request, the evaluation task request including target benchmark information and constraints, the constraints including at least a simulator type identifier; sending a resource acquisition instruction carrying the simulator type identifier to the sandbox plane through the unified handle protocol of the sandbox plane, and receiving a simulator handle returned by the sandbox plane, wherein the simulator handle is used to uniquely identify the simulator backend instance called by the sandbox plane according to the simulator type identifier; interacting with the simulator backend instance through the simulator handle to execute the evaluation task corresponding to the evaluation task request, and obtaining the evaluation trajectory data generated during the execution; calculating the indicators on the evaluation trajectory data to obtain indicator data, and generating the evaluation result of the model under test under the target evaluation benchmark indicated by the target benchmark information based on the indicator data.

[0025] Because existing technologies typically build independent evaluation scripts and runtime environments for specific simulators or benchmarks—for example, navigation tasks rely on Habitat-Sim, operation tasks are scattered across different simulators such as MuJoCo and SAPIEN, and whole-body control evaluation may use other platforms such as Isaac Sim—this fragmented evaluation architecture leads to a lack of unified interaction protocols and data standards between different simulators. This makes it difficult to compare model performance across simulators and benchmarks in the same coordinate system, and the integration of new benchmarks often requires the reconstruction of the entire evaluation system, resulting in high maintenance costs. Furthermore, existing simulation evaluations and real-device evaluations are usually executed by independent systems, using different indicator systems and result formats. This means that the simulation-to-real gap can only be qualitatively perceived through manual observation after deployment, and cannot be transformed into measurable, comparable, and quantifiable indicators during the evaluation phase.

[0026] This application provides a solution that adapts to the evaluation needs of different types of simulators and real machine platforms by constructing a unified interaction protocol and standardized data interface, incorporating model performance indicators and simulation-to-real machine gaps across simulators and benchmarks into a unified quantitative evaluation framework. First, the solution encapsulates the interaction logic of different simulation environments through an abstraction layer, enabling various benchmark testing tasks to access the evaluation system through a unified interface. This eliminates the need to reconstruct the entire evaluation process for new benchmarks, significantly reducing system maintenance and expansion costs. Simultaneously, it establishes a unified indicator system covering all performance dimensions of the model, transforming the evaluation outputs of simulation and real machine environments into quantifiable results in the same format. This allows for the quantitative measurement and difference localization of the simulation-to-real machine gap during the evaluation phase, providing a clear quantitative reference for the iterative optimization of embodied intelligent models.

[0027] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, or an evaluation system for embodied intelligence models. The following description uses an evaluation system for embodied intelligence models as an example to illustrate this embodiment and the subsequent embodiments. All actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection regulations of the country where the application is located and with authorization from the owner of the corresponding device.

[0028] Based on this, embodiments of this application provide an evaluation method for embodied intelligence models, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the evaluation method for embodied intelligence models in this application.

[0029] In this embodiment, the evaluation method for embodied intelligence models includes steps S10 to S40: Step S10: Receive an evaluation task request, which includes target benchmark information and constraints, and the constraints include at least a simulator type identifier.

[0030] It should be noted that the structural challenge of embodied evaluation lies in the fact that different benchmarks are bound to different simulators, and the scene formats, action spaces, physics engines, and turn lifecycle interfaces of different simulators are all different. The core idea of ​​this application is to treat the simulator as a type of backend in the Sandbox Plane, on the same level as the code container and terminal sandbox, and to access it through a unified acquire / release handle protocol, making it transparent to the orchestrator.

[0031] The evaluation method in this embodiment strictly distinguishes five architectural boundaries when the simulator is connected. At each boundary, "reusable shared infrastructure" and "benchmark-specific binding code" are clearly defined, as shown in Table 1. Table 1 lists the different architectural boundaries and specific shared and proprietary functions. The architectural boundaries include scene / data binding, task adapter, execution binding, model binding, and evaluation / reporting. At each of the five architectural boundaries, the binding code for reusable shared basic functions and proprietary functions is strictly distinguished. The evaluation orchestrator calls the shared basic functions to execute round-driven processes, resource scheduling, and tracking reports, and adapts the scene content, return rules, and metric definitions of the target benchmark by loading the binding code for proprietary functions. The five boundaries can be merged as needed (e.g., merging scene binding and task adapter) or further subdivided (e.g., splitting execution binding into physics engine binding and controller binding), but the core principle remains unchanged: the separation of shared infrastructure and benchmark-specific binding.

[0032] The scene / data binding boundary is used to separate scene content from scene abstraction. Specifically, the scene / data binding boundary can determine the scene content corresponding to the target simulator through the scene binding layer according to the scene configuration description. The scene content is the specific implementation of the scene configuration description in the target simulator, thereby initializing the simulation environment of the target simulator based on the scene content.

[0033] The task adapter boundary is used to separate return-compliant contracts from round-driven processes. Evaluation task requests contain round configuration information. When interacting with the simulator backend instance via the simulator handle, the execution of evaluation rounds can be driven according to the round configuration information, recording evaluation trajectory data. The unified mapping of reset / step in the round interface mapping can be replaced with an event-driven interface (pub / sub mode), a ROS 2 Action Server interface, or a custom gRPC streaming interface, as long as the semantics of action space normalization, observation space normalization, and time step alignment remain unchanged.

[0034] The execution binding boundary is used to separate the baseline-specific execution parameters from the resource model. It is mainly implemented in the unified handle protocol of the sandbox plane, so that the resource model hides the underlying simulator type from the upper layer logic. Model binding boundaries are used to separate model-specific adapters from model endpoint abstractions. This is mainly achieved through a pre-defined interface mapping layer, which is used to convert unified action commands into native action commands adapted to the simulator backend instance. The evaluation / report boundary is used to separate the benchmark-specific metrics from the report abstraction. It is mainly implemented through a scorer component that is independent of the execution process. The scorer is used to read the evaluation trajectory data and execute the scoring logic that is orthogonal to the evaluation task.

[0035] Table 1

[0036] Understandably, the evaluation task request is a declarative call initiated by the upper-layer orchestrator to the evaluation system. The target benchmark information specifies the test standard to be executed, such as an R2R navigation benchmark, a LIBERO operation benchmark, or a WBC whole-body control benchmark. This target benchmark information includes the benchmark identifier, benchmark name, version number, task definition, scene / environment, and evaluation metrics. Constraints are an abstract description of the evaluation environment configuration. Besides the required simulator type identifier (such as Habitat-Sim, MuJoCo, SAPIEN, or Isaac Sim), they can also include the scene configuration file path, physics engine parameters, random seed, parallelism requirements, and ontology model identifier. Decoupling the evaluation intent from the underlying implementation allows the upper-layer business logic to express its requirements through standardized constraints without needing to concern itself with the API differences of specific simulators, thus providing a unified input for subsequent resource scheduling.

[0037] In one feasible implementation, the evaluation task request may include a batch evaluation request for multiple model identifiers.

[0038] It should be understood that the constraints can also cover any computational backend with physical simulation capabilities, as long as it can be managed by the sandbox plane.

[0039] Step S20: Send a resource acquisition command carrying the simulator type identifier to the sandbox plane through the unified handle protocol of the sandbox plane, and receive the simulator handle returned by the sandbox plane, wherein the simulator handle is used to uniquely identify the simulator backend instance called by the sandbox plane according to the simulator type identifier.

[0040] It should be noted that the sandbox plane, as the core middleware layer of this application, is responsible for shielding the differences in deployment forms of the underlying heterogeneous emulators. Different types of emulator backends are connected to the sandbox plane in a pluggable manner. Regardless of whether the emulator backend instance runs in a Docker container, Kubernetes Pod, microvirtual machine, or local process, the sandbox plane abstracts it into a emulator resource that can be uniformly allocated, accessed, and released, and provides calling capabilities to the upper-layer orchestrator through a unified handle access interface.

[0041] Upon receiving a resource acquisition command, the sandbox plane allocates or dynamically launches the corresponding simulator backend instance from the resource pool based on the simulator type identifier, and generates a globally unique simulator handle to return to the caller. This simulator handle is an opaque reference token; it serves not only as the addressing basis for all subsequent interactive operations but also as the management anchor point for the lifecycle of the simulator backend instance. Through this handle mechanism, the orchestrator is completely unaware whether the backend is a single-machine process or a distributed cluster, and it does not need to handle complex network connections or inter-process communication details, achieving transparent access at the infrastructure level. For example, for the same handle, the underlying layer may correspond to an independent MuJoCo simulation process or a group of parallel-running Habitat-Sim container instances, but to the upper layer, they all appear as the same standardized interaction endpoint.

[0042] In one feasible implementation, step S20 may include steps S21 to S23: Step S21: Parse the parallelism parameter in the resource acquisition instruction, wherein the parallelism parameter is used to specify the required number of parallel simulation processes; The parallelism parameter is part of the constraints in the evaluation task request. Its semantics go beyond a simple integer count; it represents the upper-layer orchestrator's expected evaluation throughput. For example, during reinforcement learning policy training or large-scale benchmarking, a single evaluation might require hundreds or even thousands of independent physical rounds (Rollouts) to collect enough trajectory data to guarantee statistical significance. The parallelism parameter is used to declare this concurrency requirement.

[0043] It should be understood that the parsing of this parameter can also be combined with the current system load, available computing resource quota, and preset rate limiting policies for comprehensive verification. If the requested parallelism exceeds the system's carrying capacity, the sandbox plane can return the actual maximum number of parallelisms that can be allocated or queue up for notification, thereby ensuring the stability of the system and the reasonable scheduling of resources.

[0044] Step S22: Send a resource acquisition command carrying the parallelism parameter to the sandbox plane through the unified handle protocol of the sandbox plane, so that the sandbox plane starts the corresponding number of parallel simulation processes according to the simulator type identifier and returns the process group identifier. Upon receiving instructions containing parallelism parameters, the sandbox plane performs specific resource orchestration operations at the infrastructure layer. This process is completely transparent to the upper layers. For example, when the emulator type is identified as Habitat-Sim and the parallelism parameter is 64, the sandbox plane may activate 64 Docker container instances from a preheated container pool; or when the emulator type is MuJoCo, it may start 64 lightweight emulation processes on a single machine via multi-process fork; or it may dynamically request 64 Kubernetes Pods in a cloud-native environment. Regardless of the underlying concurrency implementation technology, the sandbox plane logically aggregates these parallel entities into a whole and generates a unique ProcessGroup ID.

[0045] It should be noted that this solution uses Docker containerization to encapsulate the emulator, which can be replaced with Kubernetes Pod, Firecracker microvirtual machine or process-level isolation, while the acquire / release handle protocol remains unchanged.

[0046] The process group identifier serves as an index key for internal management, linking the network address, process ID, and status information of all parallel instances belonging to this evaluation task. This design strictly confines the allocation and management of parallel resources within the sandbox plane, avoiding coupling between upper-layer business logic and lower-layer operational details.

[0047] Step S23: Receive the process group identifier returned by the sandbox plane, and encapsulate the process group identifier into an emulator handle.

[0048] The sandbox plane does not directly expose the complex structure containing multiple IP addresses or port lists to the caller. Instead, it encapsulates the process group identifier in a standard emulator handle object and returns it. For the upper-layer orchestrator, this emulator handle is formally indistinguishable from a single emulator handle; it remains an opaque reference token. However, when the orchestrator subsequently uses this handle to initiate interactive requests (such as Reset or Step), the sandbox plane automatically broadcasts or distributes the request to all parallel simulation processes within the process group based on internal mappings, and aggregates the responses from each process according to a predefined format before returning them. For example, a Reset call with a parallel handle actually triggers the simultaneous initialization of 64 simulation environments; a Step call executes 64 time-step physical simulations in parallel. Through this encapsulation mechanism, this application achieves the technical effect of a single handle driving a large-scale Rollout, allowing the orchestrator to transparently utilize massively parallel computing power without writing any multi-threaded, multi-process, or distributed communication code, simply by holding a single emulator handle. By sinking the management of parallel simulation processes into the sandbox plane and exposing a single simulator handle to the outside world through process group identifiers, the orchestrator can transparently request large-scale parallel rollout resources, which not only meets the high-concurrency simulation requirements of evaluation scenarios such as System 0 full-body control, but also simplifies the complexity of upper-layer scheduling logic.

[0049] Step S30: Interact with the simulator backend instance through the simulator handle to execute the evaluation task corresponding to the evaluation task request, and obtain the evaluation trajectory data generated during the execution process.

[0050] After obtaining the simulator handle, the orchestrator or round driver uses the simulator handle to drive the simulator backend instance to run according to the preset task logic.

[0051] The interaction process typically follows a unified turn-based interface paradigm, including macro-level stages such as initialization and reset, state observation, action execution, and termination determination. In this step, the focus is on establishing a stable communication channel through handles and transforming the raw state flow generated internally by the simulator into standardized evaluation trajectory data.

[0052] Evaluation trajectory data refers to structured records organized in chronological order of time steps, which include information such as agent observations, actions performed, reward feedback, physical events, and termination markers at each moment.

[0053] In one feasible implementation, in addition to the verified Habitat-Sim, MuJoCo, SAPIEN / ManiSkill, and Isaac, other simulators (such as Gazebo, PyBullet, and Unity ML-Agents) can be connected using the same "containerization + unified round interface + handle protocol" paradigm, without changing the architectural boundary division.

[0054] Step S40: Calculate the indicators from the evaluation trajectory data to obtain indicator data, and generate the evaluation results of the model under test under the target evaluation benchmark indicated by the target benchmark information based on the indicator data.

[0055] After acquiring complete evaluation trajectory data, the system calls a pre-built evaluation reducer or metric calculation module to process it. Metric calculation is the process of aggregating the raw time-series trajectory data into performance metrics with business meaning, such as success rate, confidence level of the simulation environment, effectiveness, or predictive ability. The generated evaluation result is a structured data object that not only contains the numerical values ​​of various metrics, such as the values ​​of benchmark metrics and reproducible metrics, but also associates metadata of this evaluation (such as model version, benchmark name, environment configuration, etc.) to facilitate subsequent leaderboard generation, historical comparison, or diagnostic analysis.

[0056] The generated evaluation result is a standardized, structured data object, not an isolated list of numerical values. This object contains at least three core fields: a state enumeration value representing task success or failure, a time floating-point value representing efficiency, and a reward floating-point value representing quality. In addition, this data structure can be extended to include metadata information such as task ID, model version, simulator type, and random seed for traceability and reproducibility. For example, a typical evaluation result JSON object might look like this: {"status": "SUCCESS", "completion_time": 12.5, "cumulative_reward": 85.3, "benchmark": "R2R", "model": "StreamVLN_v2"}. This composite result encapsulation allows the upper application layer to directly consume the object for multi-dimensional analysis: it can individually filter models with high success rates, further sort successful samples by completion time to find the optimal strategy, and use the cumulative reward as a feedback signal for reinforcement learning training. The standardized result format provides a unified interface contract for cross-simulator comparisons. Regardless of whether the underlying technology is Habitat-Sim or MuJoCo, as long as the evaluation results generated in accordance with this paradigm, they can be included in the same leaderboard or analysis report for processing, completely eliminating the comparison obstacles caused by inconsistent indicator definitions.

[0057] It should be noted that, in addition to calculating the basic metrics for discrete tasks, this application also provides a metric calculation paradigm for simulation-machine comparability measurement. This paradigm aims to solve the problem of the disconnect between simulation evaluation and machine evaluation in the prior art, and the inability to quantify the gap. Therefore, step S40 may include steps A11~A14: Step A11: Obtain the first evaluation trajectory data of the simulation backend and the second evaluation trajectory data of the physical machine backend based on the evaluation trajectory data; It should be noted that this application treats the physical device as an evaluation backend with the same status as the simulator. The physical device accesses the sandbox plane through the same unified handle protocol as the simulator. For example, the orchestrator sends a resource acquisition command carrying constraints such as robot identification, sensor configuration, and safety restrictions, and the sandbox plane returns the corresponding physical device handle. The model under test, driven by the same set of evaluation logic, performs the same evaluation task on both the simulation backend and the physical device backend. The generated trajectory events are all written to the same tracking system in a unified TraceRecord mode. The physical device can access the acquire / release protocol through ROS bridge, Hardware Abstraction Layer (HAL), or cloud robot agent, and is not limited to direct connection.

[0058] To distinguish the data source, the system adds a backend type label to each trajectory data when recording, such as "backend=sim" or "backend=real", thereby obtaining the first evaluation trajectory data and the second evaluation trajectory data.

[0059] Step A12: Based on the first evaluation trajectory data and the second evaluation trajectory data, calculate the first indicator value of the same evaluation index in the simulation environment and the second indicator value in the real machine environment; The system calls the same metric calculation function or reducer, and can produce two sets of values ​​simply by switching the input data filtering conditions (i.e., backend type labels). For example, for the Mean Joint Position Error (MPJPE) metric in a full-body control task, the system calculates the first metric value V_{sim} from the trajectory labeled "sim" and the second metric value V_{real} from the trajectory labeled "real". Since both originate from the same set of metric definition code and are based on the same time step alignment standard, V_{sim} and V_{real} are strictly comparable.

[0060] It should be understood that the same evaluation index is not limited to scalar performance indexes, but can also be vectorized dynamic characteristics, trajectory smoothness, or the number of safety boundary violations, as long as the index can be observed and calculated in both simulation and real machine environments.

[0061] Step A13: Calculate the numerical deviation between the first indicator value and the second indicator value, and calculate the ranking consistency coefficient between the first indicator value and the second indicator value; Numerical deviation is used to quantify the degree of systematic deviation between the simulation environment and the real physical world on specific indicators. It can be directly taken as the absolute difference between the first and second indicator values. The ranking consistency coefficient is used to evaluate the predictive ability of simulation evaluation on real-world performance ranking; that is, whether a strategy with a high simulation score among multiple candidate strategies also performs well on the actual real-world system. This application preferably uses Spearman's rank correlation coefficient (\rho) or Kendall's rank correlation coefficient (\tau) as the metric. Compared to Pearson's correlation coefficient, which assumes that the data follows a normal distribution and has a linear relationship, Spearman's \rho and Kendall's \tau are non-parametric statistical methods that only focus on the ranking relationship of the data. They are more robust to outliers and nonlinear monotonic relationships, and are more suitable for non-Gaussian distributed performance data commonly found in embodied intelligence evaluation. For example, when testing N different control strategies, the system lists their performance rankings on simulation and real-world systems respectively, and calculates the rank correlation between the two using a formula. If the coefficient is close to 1, it indicates that the simulator can accurately predict the order of merits on the real machine; if the coefficient is close to 0 or negative, it indicates that the simulator's fidelity is insufficient to support the model selection decision.

[0062] In one feasible implementation, the step of calculating the ranking consistency coefficient between the first indicator value and the second indicator value may include: ranking the first indicator value and the second indicator value of multiple test models in a simulation environment to obtain a first ranking set and a second ranking set; pairing the multiple test models one by one to determine whether the ranking order of each test model in the first ranking set is consistent with its ranking order in the second ranking set, obtaining a consistent logarithm and an inconsistent logarithm; calculating the Kendall rank correlation coefficient based on the consistent logarithm and the inconsistent logarithm; and obtaining the ranking consistency coefficient between the first indicator value and the second indicator value based on the Kendall rank correlation coefficient.

[0063] It should be noted that, assuming there are 3 models to be tested (A, B, C), and the same evaluation index, such as navigation success rate, is run in both simulation and real device environments, the first index value of model A is 0.85 and the second index value is 0.72, the first index value of model B is 0.60 and the second index value is 0.80, and the first index value of model C is 0.90 and the second index value is 0.65. Ranking the two sets of index values ​​from largest to smallest, the simulation ranking of model AC is: C, A, B, and the real device ranking of model AC is: B, A, C. The first ranking set = {A:2, B:3, C:1}, and the second ranking set = {A:2, B:1, C:3}.

[0064] Pair the three models two by two, resulting in a total of 3 pairs. For each pair, check whether their relative order in the simulation ranking and the real device ranking is the same. For example, in the simulation: A(2) < B(3), meaning A ranks before B; in the real device: A(2) > B(1), meaning A ranks after B, the rankings are inconsistent. Through sorting, the number of concordant pairs is obtained as 0, and the number of discordant pairs is 3. Then, based on the number of concordant pairs and the number of discordant pairs, the Kendall rank correlation coefficient is calculated as follows:

[0065] Wherein, the total number of pairs = the number of concordant pairs + the number of discordant pairs. Through the above example, the Kendall rank correlation coefficient can be obtained as -1, and the Kendall rank correlation coefficient is taken as the final ranking consistency coefficient, which indicates that the simulation ranking is completely opposite to the real device ranking, and the evaluation result of the simulation environment cannot reflect the performance of the real device at all.

[0066] In another feasible implementation, the step of calculating the ranking consistency coefficient of the first indicator value and the second indicator value may include: respectively ranking the first indicator value and the second indicator value of multiple models to be tested in the simulation environment to obtain a first ranking set and a second ranking set; calculating the Spearman rank correlation coefficient based on the ranking difference of the same model to be tested in the first ranking set and the second ranking set; and obtaining the ranking consistency coefficient of the first indicator value and the second indicator value according to the Spearman rank correlation coefficient.

[0067] It can be understood that ranking the first indicator value and the second indicator value is the same as the above content. For example, the obtained first ranking set = {A: 2, B: 3, C: 1}, and the second ranking set = {A: 2, B: 1, C: 3}.

[0068] By calculating the ranking difference d of each model i = simulation ranking - real device ranking, the ranking difference is used to obtain , and then the Spearman rank correlation coefficient is calculated according to the ranking difference as follows:

[0069] Wherein, n is the number of models. For example, n=3. By substituting the above ranking differences, the Spearman rank correlation coefficient is obtained as -1, and the Spearman rank correlation coefficient is taken as the final ranking consistency coefficient.

[0070] In addition to absolute numerical deviation, Spearman ρ, Kendall τ, and MMRV, statistical methods such as CCC (concordance correlation coefficient), Bland-Altman analysis, and Bootstrap confidence interval can be added.

[0071] Step A14: Obtain the evaluation result of the model under test under the target evaluation benchmark indicated by the target benchmark information based on the numerical deviation and the ranking consistency coefficient.

[0072] In practice, the generated evaluation result is no longer a single performance score, but a composite object containing multi-dimensional diagnostic information. This object includes at least two core dimensions: first, numerical deviation, which helps researchers pinpoint the gap between simulation parameters, such as friction, damping, and sensor noise, and reality; and second, ranking consistency coefficient, which helps determine whether the current simulation environment has reached a confidence level suitable for algorithm iteration. This transforms the vague concept of the "Sim-to-Real Gap" into quantifiable, comparable, and sustainably trackable engineering indicators, providing clear data support for the improvement direction of the simulator and the assessment of the model's transfer potential. Specifically, based on numerical deviation and ranking consistency coefficient, an evaluation result representing the confidence level of the simulation backend against the physical machine backend can be generated. This evaluation result characterizes the degree of matching between the simulation evaluation result and the physical machine evaluation result. The smaller the numerical deviation and the higher the ranking consistency coefficient, the higher the evaluation confidence. Thus, the evaluation confidence level indicates the correlation between the performance of the model under test in the simulation environment and its performance in the physical machine environment. When the numerical deviation is lower than the preset threshold and the ranking consistency coefficient is higher than the preset threshold, it can be determined that the current simulation backend evaluation results have reference value. Models can be directly selected based on the simulation evaluation results, reducing the frequency and cost of real machine evaluation.

[0073] In practical implementation, the target evaluation benchmark is represented by one or more evaluation tasks from the target benchmark information. Each evaluation task corresponds to a unique environment configuration, task rules, and indicator requirements. Under the multi-scenario evaluation requirements for embodied intelligence, the target evaluation benchmark can be a single-task benchmark for a specific capability dimension, or it can be a comprehensive benchmark set covering multiple scenarios and multiple capability dimensions.

[0074] For example, select 3 candidate WBC strategies and 4 representative actions; Each policy-action combination is rolled out K times on both the simulation (MuJoCo) and the real machine; The trajectory events of the simulation and the real machine are written to the same track with the tags backend=sim and backend=real, respectively; Calculate the sim and real values ​​for each metric (SR, MPJPE, hip dynamics, ground contact posture); Calculate the absolute gap (sim value - real value) and ranking consistency (Spearman ρ); Output Sim-to-Real correlation table (indicators × sim / real / gap / ρ).

[0075] Output: Sim-to-real gap quantization table, which verifies the predictive ability of simulation evaluation to rank real machine performance in the form of existence proof.

[0076] This embodiment provides an evaluation method for embodied intelligent models. By abstracting different types of simulators into pluggable back-ends of the sandbox plane and using a unified handle protocol for resource acquisition and interaction, it shields the heterogeneity of the underlying simulators in terms of scene format, action space, and lifecycle management. This allows the upper-layer orchestrator to be unaware of the implementation details of specific simulators, thereby achieving unified scheduling and standardized collection of evaluation trajectories across simulators within a single evaluation runtime. It also measures the comparability between simulators and physical machines, ensuring that the evaluation results of the same strategy on both simulators and physical machines are written into the same track and use the same indicators to quantify the magnitude of differences and ranking consistency, thus improving the comparability and interpretability of the evaluation results.

[0077] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Step S30 includes steps S301 to S303: Step S301: At the current time step of the evaluation round, obtain the unified action instruction output by the model under test.

[0078] It should be noted that the unified action instruction is the decision output made by the model under test based on the current perception state. Its format follows the general protocol predefined by the evaluation framework, such as using normalized floating-point vectors or discrete enumeration values, without containing any specific simulator's private semantics.

[0079] Each time step in the evaluation round executes a closed-loop process of perception-decision-execution: the sandbox plane first outputs the current environment state to the front-end orchestrator, the orchestrator inputs the state into the model under test, the model outputs the action and then forwards it back to the sandbox plane, and the sandbox plane completes the adaptation and forwarding of the action to the back-end.

[0080] In this step, the unified action instructions are directly forwarded to the sandbox plane by the front-end orchestrator. The sandbox plane only needs to read the standardized action dimensions and numerical ranges in the instructions to complete the subsequent distribution, without the need for secondary parsing and forwarding of the instruction format.

[0081] Step S302: Convert the unified action command into the native action command adapted to the simulator backend instance.

[0082] It should be noted that due to significant differences in the motion space definitions among different simulators and their supporting robot bodies—for example, some simulators use joint torque control, some use joint position control, and others use end effector pose control—and the number of joint degrees of freedom also varies among the bodies, dynamic adaptation is necessary through an interface mapping layer. This transforms the normalized, unified motion commands into a native command format that the target simulator's backend instance can directly recognize and invoke. This interface mapping layer uses a configurable JSON mapping table for dynamic adaptation. Different robot body and simulator combinations only need to submit the corresponding mapping table without modifying the core framework code, meeting the needs of rapid expansion across multiple scenarios.

[0083] In one feasible implementation, step S302 may include steps B11-B12: Step B11: Call the preset interface mapping layer to obtain the motion space definition corresponding to the simulator backend instance, wherein the motion space definition includes the number of joint degrees of freedom and the range of motion values; It should be noted that the interface mapping layer indexes the corresponding action space definition information from the configuration center based on the unique identifier carried by the simulator backend instance. This configuration center centrally stores the ontology and environment adaptation parameters of all connected backends and can be dynamically updated and expanded when a new backend is connected.

[0084] Step B12: Based on the action space definition, perform dimensional matching and numerical normalization on the unified action instruction to generate native action instructions that can be recognized by the simulator backend instance.

[0085] In practice, each dimension of the unified action instruction is rescaled to the numerical range required by the original action space according to the mapping rules. If the dimensions do not match, they are adjusted by padding with zeros or dimensionality reduction. The final output is a native instruction that can be directly executed by the simulator backend. In this way, the unified abstraction of action instructions by the upper-level evaluation logic is preserved, while flexibly adapting to the underlying differences of different backends, realizing the cross-simulator evaluation capability of "one definition, multiple execution".

[0086] Dimension matching aims to resolve inconsistencies between the model's output dimensions and the simulator's input dimensions. For example, if the model under test outputs a unified motion vector for a 7-DOF robotic arm, but the current simulator backend only supports a 6-DOF robotic arm, the interface mapping layer will automatically identify and remove redundant dimensions based on the preset ontology configuration file, or project the high-dimensional motion into the low-dimensional space using an inverse kinematics solver. Conversely, if the simulator has more degrees of freedom than the model output, the missing dimensions can be filled with default safety values ​​or the previous state can be maintained. Numerical normalization aims to address differences in control dimensions. Unified motion commands are typically limited to the standard range of [-1, 1] or [0, 1], while native simulators may require specific physical quantities, such as torque values ​​in Newton-meters (N·m) or position values ​​in radians (rad). The interface mapping layer restores standardized values ​​to physical quantities based on the upper and lower bound parameters in the action space definition. Through the above-mentioned dimensional matching and numerical normalization processing, this application realizes lossless translation from abstract strategy to specific physical execution, ensuring the evaluation framework's broad compatibility with different robot bodies and control modes.

[0087] Step S303: Send the native action command to the simulator backend instance through the simulator handle for interaction, so that the simulator backend instance executes the evaluation task corresponding to the evaluation task request based on the native action command, and obtains the evaluation trajectory data generated during the execution process.

[0088] In practice, the system sends native action commands to the simulator backend instance through the simulator handle to interact and execute the evaluation task corresponding to the evaluation task request. This drives the virtual objects in the simulator backend instance to perform corresponding physical actions, thereby obtaining the evaluation trajectory data generated during the execution process.

[0089] In one feasible implementation, step S303 may include steps B21 to B23: Step B21: Send the native action command to the simulator backend instance through the simulator handle to drive the virtual object in the simulator backend instance to execute the evaluation task corresponding to the evaluation task request; It should be noted that the system can send native action commands to the simulator backend instance to execute tasks, and generate native observation data during the execution process.

[0090] Step B22: Receive the raw observation data returned by the simulator backend instance after executing the evaluation task, and convert the raw observation data into unified observation data; The raw observation data may contain various modalities such as RGB images, depth maps, point clouds, joint encoder readings, and IMU data, with different data arrangements. The reverse transformation logic of the interface mapping layer encapsulates these heterogeneous data into a unified observation dictionary with modal labels. For example, it reorganizes the one-dimensional array returned by MuJoCo into a structured joint state object, or marks the tensor returned by Habitat-Sim as a standard visual observation field. This bidirectional mapping mechanism ensures that the data flowing into the model under test and the trajectory record maintains a consistent semantic structure regardless of changes in the underlying simulator.

[0091] Step B23: The unified action command and the unified observation data are appended to the trajectory record in time step order until the round termination condition is met, and the evaluation trajectory data is generated.

[0092] To support accurate evaluation and analysis, this embodiment also introduces a strict time step alignment mechanism. In embodied intelligence simulation, there are typically three time scales: the simulation step size of the physics engine, the update cycle of the controller, and the decision step of the agent. Different simulators often have different default configurations for these three scales. Without alignment, trajectory data will be misaligned on the time axis, thus affecting the accuracy of index calculations. Therefore, during interaction, the system explicitly maintains the synchronization relationship of these three time scales, ensuring that the uniform observation data recorded in each frame precisely corresponds to the same physical moment and decision node.

[0093] The round termination condition can be task success, failure, timeout, or reaching the maximum number of steps. The generated evaluation trajectory data is a time-series structured sequence that not only records the agent's behavioral trajectory but also fully preserves environmental feedback and performance signals.

[0094] Although this embodiment describes a loop within a single time step as an example, in actual operation, this loop will continue to execute until the end of the entire evaluation round. Furthermore, the specific implementation of the interface mapping layer is not limited to a software function library; it can also be an independent microservice or a hardware accelerator, as long as it can achieve bidirectional real-time conversion between the unified protocol and the native protocol. Through the above complete interactive closed-loop design, this application effectively shields the heterogeneity of the underlying simulator, providing a stable and standardized evaluation data stream for the upper layer, making fair comparisons across platforms and ontologies possible.

[0095] This embodiment introduces an interface mapping layer to establish a conversion mechanism between unified action commands and native simulator action commands. This ensures that the standardized control signals output by the model under test can be correctly parsed and executed by different physics engines or simulation environments, solving the problem of incompatibility between heterogeneous simulator action spaces. Furthermore, by converting native observation data into unified observation data, the consistency of the input data format of the model under test is guaranteed. This allows the evaluation process to flexibly adapt to the differentiated outputs of different simulation backends without modifying the inference logic of the upper-level model, further consolidating the framework's cross-heterogeneous compatibility.

[0096] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 Step S40 may include steps S401 to S404: Step S401: Calculate each evaluation index based on the evaluation trajectory data corresponding to each model to be tested.

[0097] After completing the unified scenario evaluation of multiple models, the system collects a dataset of evaluation trajectories containing identifiers for multiple models under test. This step utilizes an index calculation paradigm to independently calculate indices for each model's trajectory data, producing various evaluation metrics including success rate, cumulative reward, task completion time, or simulation-to-real-device gap coefficient. This process ensures that all models participating in the ranking generate indices based on the exact same data processing pipeline, eliminating system errors introduced by differences in calculation script versions. For example, when evaluating the three navigation models StreamVLN, QwenNav, and AgentVLN, the system extracts key event frames from their respective trajectory records, calculates their respective success rates (SR) and path-weighted scores (SPL), forming a set of structure-aligned performance vectors.

[0098] Step S402: Associate and display the identifiers of each model to be tested with the corresponding evaluation metrics to generate a cross-model evaluation leaderboard.

[0099] The performance vectors calculated above are arranged according to preset sorting rules, such as descending order of success rate or ascending order of time consumption. Model identifiers are then linked to their respective metric values ​​to construct a visual cross-model evaluation leaderboard. This leaderboard is not only a static data table but also supports dynamic filtering and multi-dimensional sorting, allowing users to examine model performance from different perspectives. For example, users can switch between viewing the "Navigation Success Rate Leaderboard" and the "Inference Efficiency Leaderboard" with a single click, quickly identifying the model that performs best in a specific dimension. This standardized leaderboard generation mechanism provides a unified horizontal comparison benchmark for models from different research teams and using different technical approaches, breaking the previous situation where individual papers presented their own arguments and direct comparisons were impossible.

[0100] Step S403: Obtain the benchmark indicator data published by each model under test in historical evaluations, and use the evaluation indicators as the reproduction indicator data.

[0101] Understandably, the system has a built-in or connection to an authoritative benchmark database that stores historical performance data of various mainstream models published in original papers or official benchmark tests, i.e., benchmark metric data (Reference Value, or Ref value). Simultaneously, the system explicitly marks the currently calculated evaluation metrics as reproducible metric data (DeepInsight Value, or DI value). This step essentially establishes a mapping relationship between "historical claimed values" and "current measured values" at the data level.

[0102] It should be understood that the source of benchmark data can be a pre-built JSON configuration file, a remote API interface, or a user-uploaded validation file, as long as it can represent the performance baseline of the model under recognized standards. By explicitly defining the current evaluation result as a "reproducibility metric," this application emphasizes the validation attribute of this evaluation, that is, not only to obtain a score, but also to confirm the position of that score in the historical coordinate system.

[0103] Step S404: In the cross-model evaluation leaderboard, for the same model to be tested, the benchmark index data and the reproduction index data are displayed in parallel, and the evaluation results of each model to be tested under the target evaluation benchmark indicated by the target benchmark information are obtained based on the benchmark index data and the reproduction index data.

[0104] When rendering the leaderboard, the system allocates two parallel data display areas for each model: one column displays the Ref value and the other column displays the DI value. The benchmark index data and the reproduction index data are used together as the evaluation results of the model under test, thus obtaining the final evaluation results of each model under test.

[0105] For example, in the R2R navigation benchmark leaderboard, if the StreamVLN model's Ref success rate is 58.0% and the DI success rate is 57.8%, it indicates successful reproduction. If the DI success rate is only 45.0%, the system can automatically highlight a warning, prompting the user to check the scene asset version or inference parameter settings. Through this mechanism, this application effectively solves the "reproducibility crisis" caused by the privatization of evaluation code and the lack of transparency in environment configuration in existing technologies. This makes the evaluation results not only valuable for horizontal comparison but also capable of vertical traceability and diagnosis, greatly enhancing the credibility and scientific rigor of the embodied intelligent evaluation ecosystem.

[0106] For example, in the Habitat-Sim simulation environment, two visual language navigation benchmarks, R2R and RxR, are configured. Each benchmark uses val-unseen splits to unify the number of episodes and initialization conditions. Five navigation models—StreamVLN, QwenNav, JanusVLN, AgentVLN, and Base—are connected to the DeepInsight inference plane. The execution flow is as follows: The orchestrator obtains the Habitat-Sim simulator handle through the sandbox plane acquire(kind='simulator', constraints={simulator='habitat', scene_config=...}); The navigation model inference handle is obtained through the inference plane acquire(kind='inference', constraints={model='StreamVLN',...}); The round-driven process executes each episode sequentially: reset(handle) → observation → model.predict(observation) → action → step(handle, action) → ... until navigation terminates or times out; The trajectory events (position, orientation, observation, action, reward) for each step are written to the trace using TraceRecord; The evaluation reducer calculates metrics such as SR, SPL, nDTW, SDTW, and NE from the tracking data; For each model, both Ref (original paper value) and DI (DeepInsight reproduction value) are displayed simultaneously to verify the fidelity of reproduction.

[0107] Outputs: R2R leaderboard (5 models × 6 metrics) and RxR leaderboard (5 models × 6 metrics), which can be directly used for cross-model comparisons.

[0108] In one feasible implementation, this embodiment proposes an explicit resource release and reclamation mechanism. After the evaluation task is completed, a resource release instruction is generated, wherein the resource release instruction carries the simulator handle; the resource release instruction is sent to the sandbox plane so that the sandbox plane cancels the simulator handle and reclaims the computing resources occupied by the simulator backend instance.

[0109] When an evaluation session transitions from an active state to a terminated state, the triggering conditions include not only normal round termination, such as task success or reaching the maximum number of steps, but also abnormal interruptions, such as model inference timeouts, simulator process crashes, user-initiated cancellations, or system-level circuit breakers. Regardless of the reason for task termination, the system must deterministically generate resource release instructions, rather than relying on garbage collection mechanisms or implicit cleanup upon process exit. This instruction uses the simulator handle as the core index, ensuring precise targeting of the release operation. Furthermore, the resource release instruction can be extended to carry cleanup strategy parameters, such as specifying whether to retain trajectory logs for post-event diagnosis, whether to forcibly terminate suspended child processes, or whether to save a snapshot of the current environment as a checkpoint for subsequent recovery. This explicit instruction generation design fundamentally eliminates resource hanging problems caused by upper-layer orchestrator logic defects or abnormal exits, providing the first line of defense for the long-term stable operation of the system.

[0110] Upon receiving this instruction, the sandbox plane executes a strict resource reclamation protocol. First, it deregisters the emulator handle from its internal routing table or handle registry, immediately invalidating it and preventing any subsequent erroneous operations or malicious requests from accessing the released backend instance through the handle. Second, based on the metadata associated with the handle, the sandbox plane locates the underlying emulator backend instance and performs physical resource reclamation. The granularity and depth of the reclamation operation depend on the specific type of backend instance: for Docker containerized emulators, reclamation actions include stopping the container, deleting the container instance and its mounted temporary volumes, and releasing occupied network ports; for local process-level emulators, reclamation actions include sending termination signals (SIGTERM / SIGKILL), waiting for the process to exit, and cleaning up shared memory segments and temporary files; for emulators using GPU acceleration, the device management interface must be explicitly called to release GPU memory usage, preventing memory fragmentation or leakage from affecting subsequent tasks.

[0111] As an alternative implementation, the resource release and reclamation mechanism can also adopt asynchronous or delayed strategies to adapt to different performance requirements. For example, in high-throughput reinforcement learning training scenarios, frequent creation and destruction of simulator instances can lead to significant time overhead. In this case, after receiving a resource release instruction, the sandbox plane may not immediately destroy the backend instance, but instead reset it to its initial state and put it back into the preheating resource pool for direct reuse in the next resource acquisition instruction. In this case, "unregistering the simulator handle" only means releasing the current orchestrator from the instance, while "reclaiming computing resources" is postponed until the instance is actually retired or the system load decreases. As long as the sandbox plane can ensure that the semantics of the exposed handles are correct (i.e., released handles are unusable, and newly acquired handles correspond to available resources), the specific internal reclamation sequence does not constitute a limitation on the scope of protection of this application. Through this flexible reclamation strategy, this application ensures both resource security and evaluation efficiency, further enhancing the system's adaptability and engineering practicality in diverse application scenarios.

[0112] This embodiment provides an intuitive means of verifying reproducibility by constructing a comparison leaderboard that includes the original baseline value and the current reproducible value. This helps to quickly identify performance fluctuations caused by differences in the evaluation environment configuration or model version iterations, thereby enhancing the credibility and traceability of the evaluation results.

[0113] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the evaluation method of this application for embodied intelligence models. Any simple modifications based on this technical concept are within the protection scope of this application.

[0114] This application also provides an evaluation device for embodied intelligence models, please refer to... Figure 5 The evaluation device for embodied intelligence models includes: The task adaptation module 10 is used to receive evaluation task requests, which include target benchmark information and constraints, and the constraints include at least a simulator type identifier.

[0115] The execution binding module 20 is used to send a resource acquisition instruction carrying the simulator type identifier to the sandbox plane through the unified handle protocol of the sandbox plane, and receive the simulator handle returned by the sandbox plane, wherein the simulator handle is used to uniquely identify the simulator backend instance called by the sandbox plane according to the simulator type identifier.

[0116] The interaction module 30 is used to interact with the simulator backend instance through the simulator handle to execute the evaluation task corresponding to the evaluation task request and obtain the evaluation trajectory data generated during the execution process.

[0117] The evaluation module 40 is used to calculate the indicators from the evaluation trajectory data to obtain indicator data, and to generate the evaluation results of the model under test under the target evaluation benchmark indicated by the target benchmark information based on the indicator data.

[0118] The task adaptation module 10, the execution binding module 20, the interaction module 30, and the evaluation module 40 are configured to run on their respective architectural boundaries and interact with each other through standardized interfaces. The evaluation apparatus for embodied intelligence models provided in this application employs the evaluation method for embodied intelligence models described in the above embodiments. This addresses the technical problems of current solutions where simulator fragmentation leads to difficulties in cross-platform comparisons and the inability to measure the difference between simulation and real-device performance. Compared to existing technologies, the beneficial effects of the evaluation apparatus for embodied intelligence models provided in this application are the same as those of the evaluation method for embodied intelligence models provided in the above embodiments. Furthermore, other technical features of the evaluation apparatus for embodied intelligence models are the same as those disclosed in the methods of the above embodiments, and will not be elaborated upon here.

[0119] In one embodiment, the interaction module 30 is further configured to obtain the unified action instruction output by the model under test at the current time step of the evaluation round; The unified action instructions are converted into native action instructions adapted to the simulator backend instance. The simulator handle sends the native action command to the simulator backend instance for interaction, so that the simulator backend instance executes the evaluation task corresponding to the evaluation task request based on the native action command, and obtains the evaluation trajectory data generated during the execution process.

[0120] In one embodiment, the interaction module 30 is further configured to call a preset interface mapping layer to obtain the motion space definition corresponding to the simulator backend instance, wherein the motion space definition includes the number of joint degrees of freedom and the range of motion values; Based on the defined action space, the unified action instructions are subjected to dimensional matching and numerical normalization to generate native action instructions that can be recognized by the simulator backend instance.

[0121] In one embodiment, the interaction module 30 is further configured to send the native action instruction to the simulator backend instance through the simulator handle, so as to drive the virtual object in the simulator backend instance to execute the evaluation task corresponding to the evaluation task request; Receive the raw observation data returned by the simulator backend instance after executing the evaluation task, and convert the raw observation data into unified observation data; The unified action instructions and the unified observation data are appended to the trajectory record in time step order until the round termination condition is met, thereby generating evaluation trajectory data.

[0122] In one embodiment, the execution binding module 20 is further configured to parse the parallelism parameter in the resource acquisition instruction, wherein the parallelism parameter is used to specify the required number of parallel simulation processes; Through the unified handle protocol of the sandbox plane, a resource acquisition command carrying the parallelism parameter is sent to the sandbox plane, so that the sandbox plane starts the corresponding number of parallel simulation processes according to the simulator type identifier and returns the process group identifier. Receive the process group identifier returned by the sandbox plane, and encapsulate the process group identifier into an emulator handle.

[0123] In one embodiment, the evaluation module 40 is further configured to obtain first evaluation trajectory data of the simulation backend and second evaluation trajectory data of the physical machine backend based on the evaluation trajectory data; Based on the first evaluation trajectory data and the second evaluation trajectory data, respectively, calculate the first indicator value of the same evaluation index in the simulation environment and the second indicator value in the real machine environment; Calculate the numerical deviation between the first indicator value and the second indicator value, and calculate the ranking consistency coefficient between the first indicator value and the second indicator value; The evaluation results of the model under test under the target evaluation benchmark indicated by the target benchmark information are obtained based on the numerical deviation and the ranking consistency coefficient.

[0124] In one embodiment, the evaluation module 40 is further configured to rank the first index value and the second index value of multiple test models in a simulation environment, respectively, to obtain a first ranking set and a second ranking set; Pair multiple models to be tested together to determine whether the ranking order of each model to be tested in the first ranking set is consistent with the ranking order in the second ranking set, and obtain the consistent logarithm and inconsistent logarithm. Calculate the Kendall rank correlation coefficient based on the consistent logarithm and the inconsistent logarithm; The ranking consistency coefficient between the first indicator value and the second indicator value is obtained based on the Kendall rank correlation coefficient.

[0125] In one embodiment, the evaluation module 40 is further configured to rank the first index value and the second index value of multiple test models in a simulation environment, respectively, to obtain a first ranking set and a second ranking set; The Spearman rank correlation coefficient is calculated based on the ranking difference of the same model under test in the first ranking set and the second ranking set. The ranking consistency coefficient between the first indicator value and the second indicator value is obtained based on the Spearman rank correlation coefficient.

[0126] In one embodiment, the evaluation module 40 is further configured to calculate various evaluation indicators based on the evaluation trajectory data corresponding to each model under test; The system associates and displays the identifiers of each model under test with their corresponding evaluation metrics, generating a cross-model evaluation leaderboard. Obtain the benchmark indicator data of each model under test in historical evaluations, and use the aforementioned evaluation indicators as reproducible indicator data; In the cross-model evaluation leaderboard, for the same model under test, the benchmark index data and the reproduction index data are displayed in parallel, and the evaluation results of each model under test under the target evaluation benchmark indicated by the target benchmark information are obtained based on the benchmark index data and the reproduction index data.

[0127] This application provides an evaluation device for embodied intelligence models. The evaluation device for embodied intelligence models includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the evaluation method for embodied intelligence models in the first embodiment described above.

[0128] The following is for reference. Figure 5 This document illustrates a structural schematic diagram of an evaluation device suitable for implementing embodiments of the present application regarding embodied intelligence models. The evaluation device for embodied intelligence models in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), tablets, PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The evaluation device for embodied intelligence models shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0129] like Figure 5As shown, the evaluation device for embodied intelligent models may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to programs stored in ROM (Read Only Memory) 1002 or programs loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the evaluation device for embodied intelligent models. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, LCDs (Liquid Crystal Displays), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the evaluation device for embodied intelligence models to exchange data wirelessly or via wired communication with other devices. Although the figure shows an evaluation device for embodied intelligence models with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0130] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0131] The evaluation device for embodied intelligence models provided in this application, employing the evaluation method for embodied intelligence models described in the above embodiments, can solve the technical problems of current solutions where simulator fragmentation leads to difficulties in cross-platform comparison and the inability to measure the difference between simulation and real machines. Compared with the prior art, the beneficial effects of the evaluation device for embodied intelligence models provided in this application are the same as those of the evaluation method for embodied intelligence models provided in the above embodiments, and other technical features in this evaluation device for embodied intelligence models are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0132] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0133] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0134] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the evaluation method for embodied intelligence models in the above embodiments.

[0135] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory or Flash Memory), optical fibers, CD-ROM (CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0136] The aforementioned computer-readable storage medium may be included in an evaluation device for embodied intelligence models; or it may exist independently and not be assembled into an evaluation device for embodied intelligence models.

[0137] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by an evaluation device for an embodied intelligent model, the evaluation device for the embodied intelligent model: receives an evaluation task request, the evaluation task request including target benchmark information and constraints, the constraints including at least a simulator type identifier; sends a resource acquisition instruction carrying the simulator type identifier to the sandbox plane via a unified handle protocol of the sandbox plane, and receives a simulator handle returned by the sandbox plane, wherein the simulator handle is used to uniquely identify the simulator backend instance invoked by the sandbox plane according to the simulator type identifier; interacts with the simulator backend instance through the simulator handle to execute the evaluation task corresponding to the evaluation task request, and obtains evaluation trajectory data generated during execution; performs index calculations on the evaluation trajectory data to obtain index data, and generates an evaluation result of the model under test under the target evaluation benchmark indicated by the target benchmark information based on the index data.

[0138] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LAN (Local Area Network) or WAN (Wide Area Network)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0140] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0141] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described evaluation method for embodied intelligence models. This solves the technical problems of current solutions, such as simulator fragmentation leading to difficulties in cross-platform comparisons and the inability to measure the difference between simulation and real-device performance. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the evaluation method for embodied intelligence models provided in the above embodiments, and will not be repeated here.

[0142] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the evaluation method for embodied intelligence models as described above.

[0143] The computer program product provided in this application can solve the technical problems of current solutions in embodied intelligence evaluation, such as the difficulty of cross-platform comparison caused by simulator fragmentation and the inability to measure the difference between simulation and real machine. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the evaluation method for embodied intelligence models provided in the above embodiments, and will not be repeated here.

[0144] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for evaluating embodied intelligence models, characterized in that, The evaluation method for embodied intelligence models includes: Receive an evaluation task request, the evaluation task request including target benchmark information and constraints, the constraints including at least a simulator type identifier; Through the unified handle protocol of the sandbox plane, a resource acquisition command carrying the simulator type identifier is sent to the sandbox plane, and the simulator handle returned by the sandbox plane is received. The simulator handle is used to uniquely identify the simulator backend instance called by the sandbox plane according to the simulator type identifier. The simulator handle interacts with the simulator backend instance to execute the evaluation task corresponding to the evaluation task request and obtain the evaluation trajectory data generated during the execution process. The evaluation trajectory data is used to calculate the index data, and the evaluation results of the model under test are generated based on the index data under the target evaluation benchmark indicated by the target benchmark information.

2. The method as described in claim 1, characterized in that, The steps of interacting with the simulator backend instance through the simulator handle to execute the evaluation task corresponding to the evaluation task request and obtaining the evaluation trajectory data generated during the execution process include: At the current time step of the evaluation round, obtain the unified action command output by the model under test; The unified action instructions are converted into native action instructions adapted to the simulator backend instance. The simulator handle sends the native action command to the simulator backend instance for interaction, so that the simulator backend instance executes the evaluation task corresponding to the evaluation task request based on the native action command, and obtains the evaluation trajectory data generated during the execution process.

3. The method as described in claim 2, characterized in that, The step of converting the unified action command into a native action command adapted to the simulator backend instance includes: Call the preset interface mapping layer to obtain the motion space definition corresponding to the simulator backend instance, wherein the motion space definition includes the number of joint degrees of freedom and the range of motion values; Based on the defined action space, the unified action instructions are subjected to dimensional matching and numerical normalization to generate native action instructions that can be recognized by the simulator backend instance.

4. The method as described in claim 2, characterized in that, The step of sending the native action command to the simulator backend instance through the simulator handle for interaction, so that the simulator backend instance executes the evaluation task corresponding to the evaluation task request based on the native action command, and obtains the evaluation trajectory data generated during the execution process includes: The native action command is sent to the simulator backend instance through the simulator handle to drive the virtual object in the simulator backend instance to execute the evaluation task corresponding to the evaluation task request; Receive the raw observation data returned by the simulator backend instance after executing the evaluation task, and convert the raw observation data into unified observation data; The unified action instructions and the unified observation data are appended to the trajectory record in time step order until the round termination condition is met, thereby generating evaluation trajectory data.

5. The method as described in claim 1, characterized in that, The step of sending a resource acquisition command carrying the simulator type identifier to the sandbox plane via the unified handle protocol of the sandbox plane, and receiving the simulator handle returned by the sandbox plane, includes: Parse the parallelism parameter in the resource acquisition instruction, wherein the parallelism parameter is used to specify the required number of parallel simulation processes; Through the unified handle protocol of the sandbox plane, a resource acquisition command carrying the parallelism parameter is sent to the sandbox plane, so that the sandbox plane starts the corresponding number of parallel simulation processes according to the simulator type identifier and returns the process group identifier. Receive the process group identifier returned by the sandbox plane, and encapsulate the process group identifier into an emulator handle.

6. The method as described in claim 1, characterized in that, The steps of calculating index data from the evaluation trajectory data and generating evaluation results of the model under test under the target evaluation benchmark indicated by the target benchmark information based on the index data include: Based on the evaluation trajectory data, the first evaluation trajectory data of the simulation backend and the second evaluation trajectory data of the physical machine backend are obtained; Based on the first evaluation trajectory data and the second evaluation trajectory data, respectively, calculate the first indicator value of the same evaluation index in the simulation environment and the second indicator value in the real machine environment; Calculate the numerical deviation between the first indicator value and the second indicator value, and calculate the ranking consistency coefficient between the first indicator value and the second indicator value; The evaluation results of the model under test under the target evaluation benchmark indicated by the target benchmark information are obtained based on the numerical deviation and the ranking consistency coefficient.

7. The method as described in claim 6, characterized in that, The steps for calculating the ranking consistency coefficient between the first indicator value and the second indicator value include: The first index value and the second index value of multiple models under test are ranked in the simulation environment to obtain a first ranking set and a second ranking set; Pair multiple models to be tested together to determine whether the ranking order of each model to be tested in the first ranking set is consistent with the ranking order in the second ranking set, and obtain the consistent logarithm and inconsistent logarithm. Calculate the Kendall rank correlation coefficient based on the consistent logarithm and the inconsistent logarithm; The ranking consistency coefficient between the first indicator value and the second indicator value is obtained based on the Kendall rank correlation coefficient.

8. The method as described in claim 6, characterized in that, The steps for calculating the ranking consistency coefficient between the first indicator value and the second indicator value include: The first index value and the second index value of multiple models under test are ranked in the simulation environment to obtain a first ranking set and a second ranking set; The Spearman rank correlation coefficient is calculated based on the ranking difference of the same model under test in the first ranking set and the second ranking set. The ranking consistency coefficient between the first indicator value and the second indicator value is obtained based on the Spearman rank correlation coefficient.

9. The method as described in claim 1, characterized in that, The steps of calculating index data from the evaluation trajectory data and generating evaluation results of the model under test under the target evaluation benchmark indicated by the target benchmark information based on the index data include: Based on the evaluation trajectory data corresponding to each model to be tested, calculate each evaluation index separately; The system associates and displays the identifiers of each model under test with their corresponding evaluation metrics, generating a cross-model evaluation leaderboard. Obtain the benchmark indicator data of each model under test in historical evaluations, and use the aforementioned evaluation indicators as reproducible indicator data; In the cross-model evaluation leaderboard, for the same model under test, the benchmark index data and the reproduction index data are displayed in parallel, and the evaluation results of each model under test under the target evaluation benchmark indicated by the target benchmark information are obtained based on the benchmark index data and the reproduction index data.

10. An evaluation device for embodied intelligence models, characterized in that, The device includes a multi-layered architectural boundary, and the device includes: The task adaptation module is used to receive evaluation task requests, which include target benchmark information and constraints, and the constraints include at least a simulator type identifier. The execution binding module is used to send a resource acquisition instruction carrying the simulator type identifier to the sandbox plane through the unified handle protocol of the sandbox plane, and receive the simulator handle returned by the sandbox plane, wherein the simulator handle is used to uniquely identify the simulator backend instance called by the sandbox plane according to the simulator type identifier. The interaction module is used to interact with the simulator backend instance through the simulator handle to execute the evaluation task corresponding to the evaluation task request and obtain the evaluation trajectory data generated during the execution process. The evaluation module is used to calculate the evaluation trajectory data to obtain the evaluation data, and generate the evaluation results of the model under test under the target evaluation benchmark indicated by the target benchmark information based on the evaluation data. The task adaptation module, the execution binding module, the interaction module, and the evaluation module are configured to run on their respective architectural boundaries and interact with each other through standardized interfaces.

11. An evaluation device for embodied intelligence models, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the evaluation method for embodied intelligence models as described in any one of claims 1 to 9.

12. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the evaluation method for embodied intelligent models as described in any one of claims 1 to 9.

Images