Body model evaluation method, system, and computing device cluster

Abstract

The application provides a body model evaluation method, system and computing device cluster, and relates to the technical field of cloud computing. The body model evaluation method comprises the following steps: inputting an initial observation graph and an initial ontology state parameter of a real robot into a to-be-tested body model to determine a current action instruction; driving a simulation robot based on a trajectory of the real robot to operate a first simulation object according to the current action instruction, to obtain an ontology state parameter and a state parameter; generating a first simulation object segmentation graph and a simulation robot pose graph according to the ontology state parameter and the state parameter, a reference observation graph for providing a distribution of a real environment observation graph, and an observation graph of a next moment; inputting the observation graph of the next moment and the ontology state parameter into the to-be-tested body model, and iteratively performing the above steps of driving to generating until an evaluation result is obtained when a task ends. The embodiment of the application can balance efficiency, cost and accuracy in body model evaluation.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular to an embodied model evaluation method, system, and computing device cluster. Background Technology

[0002] Embodied intelligence, such as in robotics, refers to giving artificial intelligence a "body" capable of perception and action. It uses multiple sensors, including vision, depth, and touch, to perceive its environment, and then uses actuators such as robotic arms and mobile platforms to manipulate the environment (e.g., objects within it), achieving a closed-loop intelligence of perception, decision-making, and action. Embodied intelligence goes beyond mere computation in the information world; it acquires knowledge through interaction with the real physical world to form behavioral strategies and complete tasks, more closely resembling the intelligence humans develop based on their physical interaction with the environment.

[0003] Embodied intelligence behavioral strategies are generally generated through embodied models configured for robots. This means the embodied model must perform tasks in the physical world, making evaluation far more important than for pure algorithmic models. Current embodied model evaluation schemes mainly include simulation evaluation and real-machine evaluation. Simulation evaluation is low-cost and efficient, but the significant difference between the simulation environment and the real world can easily lead to distorted evaluation results. Real-machine evaluation provides realistic results but is costly and inefficient. Therefore, how to balance high efficiency, low cost, and accuracy in embodied model evaluation is a pressing issue that needs to be addressed. Summary of the Invention

[0004] To address the aforementioned issues, this application provides an embodied model evaluation method, system, and computing device cluster. In this embodied model evaluation method, physical alignment ensures that the dynamic characteristics of the simulation environment are consistent with the real environment, and visual alignment ensures that the visual representation of the simulation environment is consistent with the real environment. The two work synergistically to achieve high-fidelity simulation-to-real-world transfer, thus balancing high efficiency, low cost, and accuracy in embodied model evaluation.

[0005] The following is a detailed description of the embodied model evaluation method, system, and computing device cluster provided in the embodiments of this application.

[0006] In a first aspect, embodiments of this application provide an embodied model evaluation method, which includes: inputting an initial observation map and initial body state parameters collected by a real robot into the embodied model under test to determine the current action command; driving a simulated robot deployed in a first simulation environment to operate a first simulated object according to the current action command, obtaining the body state parameters of the simulated robot and the state parameters of the first simulated object, wherein the physical parameters of the simulated robot are obtained based on the trajectory of the real robot; generating a segmentation map of the first simulated object and a pose map of the simulated robot according to the body state parameters of the simulated robot and the state parameters of the first simulated object; generating an observation map for the next moment according to the segmentation map of the first simulated object, the pose map of the simulated robot, and a reference observation map, wherein the reference observation map is used to provide the distribution of observation maps of the environment in which the real robot is located; inputting the visually aligned observation map for the next moment and the body state parameters of the simulated robot into the embodied model under test to obtain the next action command, and using the next action command as the current action command, iteratively executing driving the simulated robot to operate the first simulated object according to the current action command until the iteration termination condition is met to obtain the task execution result; and providing an evaluation result of the embodied model under test according to the task execution result.

[0007] In this embodiment, the initial observation map and body state parameters collected by the real robot are input into the embodied model under test to determine the action instructions. In the simulation environment, the simulated robot, which obtains physical parameters based on the trajectory of the real robot, is driven to perform operations. The observation map of the next moment is generated by providing the observation map distribution of the real environment through the reference observation map. This forms a closed-loop iterative evaluation process that coordinates physical alignment and visual alignment. In this way, the visual input received by the embodied model under test in the simulation environment and the state generated by the simulated robot, i.e., the dynamic feedback, are significantly reduced from the real environment. This ensures that the evaluation results can accurately reflect the capabilities of the embodied model under test in the real environment. It is equivalent to achieving high-fidelity simulation to reality transfer, thus balancing high efficiency, low cost and accuracy in embodied model evaluation.

[0008] According to the first aspect, the trajectory of the real robot includes the real body state sequence collected by the real robot; the acquisition of the physical parameters of the simulated robot includes: optimizing the physical parameters of the differentiable neural network simulator based on the error between the real body state sequence collected by the real robot and the simulated body state sequence generated by the differentiable neural network simulator to obtain the physically aligned physical parameters; and configuring the physically aligned parameters for the simulated robot.

[0009] In this embodiment, the physical parameters of a differentiable neural network simulator are optimized by using the error between the real ontology state sequence and the simulated ontology state sequence. The differentiability property ensures that the physical parameters can be updated in reverse by adjusting the error, thereby improving the efficiency of obtaining the physical parameters after physical alignment.

[0010] According to the first aspect, or any implementation of the first aspect above, based on the error between the real body state sequence collected by the real robot and the simulated body state sequence generated by the differentiable neural network simulator, the physical parameters of the differentiable neural network simulator are optimized to obtain the physically aligned physical parameters, including: acquiring the real body state sequence collected by the real robot based on the second real action command sequence; inputting the simulated physical parameters and the second real action command sequence into the differentiable neural network simulator to generate the simulated state sequence; acquiring the error between the real body state sequence and the simulated state sequence; updating the simulated physical parameters through the backpropagation of the gradient of the error by the differentiable neural network simulator to obtain the physically aligned physical parameters.

[0011] In this embodiment, the real body state sequence acquired by the real robot based on the second real action command sequence is compared with the simulation state sequence generated by the differentiable neural network simulator. The error between the two is calculated and the simulation physical parameters are updated through gradient backpropagation. In this way, the gradient-based optimization process can be directly used to update the physical parameters: the parameters are updated in each iteration along the direction of error descent, which further improves the efficiency and accuracy of obtaining the physical parameters after physical alignment.

[0012] According to the first aspect, or any implementation of the first aspect above, before inputting the simulated physical parameters and the second real action command sequence into the differentiable neural network simulator to generate the simulated state sequence, the method further includes: inputting the initial physical parameters of the real robot into the parameter mapping network to obtain the simulated physical parameters; updating the simulated physical parameters by backpropagating the gradient of the error through the differentiable neural network simulator to obtain the physically aligned physical parameters, including: backpropagating the gradient of the error through the differentiable neural network simulator to obtain the gradient of the weights of the parameter mapping network; updating the parameter mapping network according to the gradient of the weights and the optimization objective of minimizing the error to obtain the updated parameter mapping network, and determining the physical parameters output by the updated parameter mapping network as the physically aligned physical parameters.

[0013] In this embodiment, by introducing a parameter mapping network, the optimization process of physical parameters is transformed into the optimization process of network weights. In this way, optimization based on the network weight space is smoother and the gradient direction is clearer than optimization based directly on the original physical parameter space. This can avoid the problems of numerical instability and local optima that may occur when searching for relatively better parameters directly in the physical parameter space, thereby further improving the accuracy of the physical parameters after physical alignment.

[0014] According to the first aspect, or any implementation of the first aspect above, before inputting the simulated physical parameters and the second real action command sequence into the differentiable neural network simulator to generate the simulated state sequence, the method further includes: sampling the physical parameters, action command sequence, and first body state sequence corresponding to the action command sequence of the simulated robot to obtain a first training sample and a label of the first training sample, wherein the first training sample includes the sampled physical parameters, the action command at time i in the action command sequence, and the body state parameters at time i in the first body state sequence, and the label of the training sample at time i in the first training sample includes the body state parameters at time i+1 in the first body state sequence; using the first training sample and the label of the first training sample, a multilayer perceptron is trained to obtain the differentiable neural network simulator.

[0015] In this embodiment, training samples, i.e., a large amount of simulation dynamics data, are constructed by sampling the physical parameters, action command sequences, and body state sequences of the simulated robot. A multilayer perceptron is then trained as a differentiable neural network simulator. In this way, after learning a large amount of simulation dynamics data, the multilayer perceptron can fit the physical system of the simulated robot. Moreover, the neural network has differentiable characteristics, which can ensure that the physical parameters can be optimized through the simulator in the future.

[0016] According to the first aspect, or any implementation of the first aspect above, generating an observation map for the next moment based on the first simulated object segmentation map, the simulated robot pose map, and the reference observation map includes: inputting the first simulated object segmentation map, the simulated robot pose map, and the reference observation map into a visual alignment generation model to generate a visually aligned observation map for the next moment, wherein the training data of the visual alignment generation model includes a sequence of real observation maps pre-collected by the real robot based on a first real action command sequence, and a sequence of simulated object segmentation maps and a sequence of simulated robot pose maps corresponding to the first real action command sequence.

[0017] In this embodiment, a visually aligned observation map is generated by inputting a simulated object segmentation map, a simulated robot pose map, and a reference observation map into a visual alignment generation model. The training data of the visual alignment generation model uses real observation map sequences as labels and simulated segmentation maps and pose maps as input conditions. In this way, the visual alignment generation model can learn the mapping relationship from simulated geometric conditions to real observation maps. The generated observation map not only conforms to the geometric layout of the simulated scene but also has the visual distribution characteristics of the real environment, thereby ensuring the effect of visual alignment.

[0018] According to the first aspect, or any implementation of the first aspect above, the process of acquiring training data for the visual alignment generation model includes: acquiring a sequence of real observation maps and a sequence of real action commands collected by a real robot in a real environment based on a first sequence of real action commands; driving a simulated robot deployed in a second simulation scene to operate a second simulated object according to the first sequence of real action commands, obtaining a second body state sequence of the simulated robot and a state sequence of the second simulated object, wherein the second simulation scene matches the real scene; generating a second training sample according to the second body state sequence of the simulated robot and the state sequence of the second simulated object, wherein the second training sample includes a simulated object segmentation map at time i in the simulated object segmentation map sequence and a simulated robot pose map at time i in the simulated robot pose map sequence, where i is an integer greater than or equal to 1; determining the real observation map at time i in the real observation map sequence as the sample label of the training sample at time i in the second training sample, thereby obtaining training data for the visual alignment generation model.

[0019] In this embodiment, the operation process is reproduced in a simulation scene using real action command sequences, and corresponding segmentation maps and pose maps are generated as training samples. Real observation maps are used as labels to construct training data for the visual alignment generation model. In this way, the segmentation maps and pose maps retain the geometric structure information of the simulation scene, and the real observation maps, as labels, provide visual distribution information of the real environment. The visual alignment generation model can learn the mapping relationship from simulation geometric conditions to real observation maps.

[0020] According to the first aspect, or any implementation of the first aspect above, the visual alignment generation model adopts the form of a conditional video diffusion model; the first simulated object segmentation map, the simulated robot pose map, and the reference observation map are input into the visual alignment generation model to generate the visually aligned observation map at the next moment, including: summing the pixels of the first simulated object segmentation map and the simulated robot pose map, and extracting features from the result of the pixel summation to obtain simulated observation features; extracting features from the reference observation map to obtain real observation features; determining the conditional signal of the visual alignment generation model based on the simulated observation features and the real observation features, and inputting the conditional signal into the visual alignment generation model to generate the visually aligned observation map at the next moment.

[0021] In this embodiment, a conditional video diffusion model is used as the visual alignment generation model. By summing the pixels of the simulated object segmentation map and the simulated robot pose map, the simulation observation features are extracted from the summing result, and the real observation features are extracted from the reference observation map. The two types of features are fused as a conditional signal to guide the denoising generation process of the diffusion model. In this way, the conditional signal contains both the geometric layout information of the simulation environment and the visual information of the real environment. The generated observation map not only conforms to the spatial layout of the simulation scene, but also has the visual characteristics of the real world.

[0022] Secondly, embodiments of this application provide a physical alignment method for embodied models. The method includes: acquiring a real body state sequence collected by a real robot based on a real action command sequence; inputting the simulated physical parameters and the real action command sequence into a differentiable neural network simulator to generate a simulated state sequence; and updating the simulated physical parameters based on the error between the real body state sequence and the simulated state sequence by backpropagating the gradient of the error through the differentiable neural network simulator to obtain the physically aligned physical parameters.

[0023] In this embodiment, the real robot's state sequence, acquired based on real action command sequences, is compared with the simulated state sequence generated by a differentiable neural network simulator. The error between the two sequences is used to update the simulated physical parameters through gradient backpropagation, thus achieving automatic alignment between the physical parameters of the simulated environment and the real environment. Applying these aligned physical parameters to the simulated robot ensures that the robot driven in the simulation environment exhibits dynamic behavior consistent with that of the real robot, narrowing the gap in dynamic characteristics between simulation and reality.

[0024] According to the second aspect, before inputting the simulated physical parameters and the real action command sequence into the differentiable neural network simulator to generate the simulated state sequence, the method further includes: inputting the initial physical parameters of the real robot into the parameter mapping network to obtain the simulated physical parameters; updating the simulated physical parameters by backpropagating the gradient of the error through the differentiable neural network simulator to obtain the physically aligned physical parameters, including: backpropagating the gradient of the error through the differentiable neural network simulator to obtain the gradient of the weights of the parameter mapping network; updating the parameter mapping network according to the gradient of the weights and the optimization objective of minimizing the error to obtain the updated parameter mapping network, and determining the physical parameters output by the updated parameter mapping network as the physically aligned physical parameters.

[0025] In this embodiment, by introducing a parameter mapping network, the optimization process of physical parameters is transformed into the optimization process of network weights. In this way, the optimization of the network weight space is smoother and the gradient direction is clearer than that of the original physical parameter space. This avoids the problems of numerical instability and local optima that may occur when searching directly in the physical parameter space, thereby further improving the efficiency and accuracy of obtaining physical parameters after physical alignment.

[0026] According to the second aspect, or any implementation of the second aspect above, before inputting the simulated physical parameters and the real action command sequence into the differentiable neural network simulator to generate the simulated state sequence, the method further includes: sampling the physical parameters, action command sequence, and the body state sequence corresponding to the action command sequence of the simulated robot to obtain a first training sample and a label of the first training sample, wherein the training sample at time i in the first training sample includes the sampled physical parameters, the action command at time i in the action command sequence, and the body state parameters at time i in the body state sequence, and the label of the training sample at time i in the first training sample includes the body state parameters at time i+1 in the body state sequence, where i is an integer greater than or equal to 1; using the training sample and the label of the training sample, a multilayer perceptron is trained to obtain the differentiable neural network simulator.

[0027] In this embodiment, training samples are constructed by sampling the physical parameters, action command sequences, and body state sequences of the simulated robot. A multilayer perceptron is then trained as a differentiable neural network simulator. In this way, the multilayer perceptron fits the state transition function under different physical parameter configurations, and the neural network has differentiable characteristics. The optimization process of physical parameters can be performed through gradient calculation and backpropagation by the simulator, thereby further improving the optimization efficiency and accuracy.

[0028] According to the second aspect, or any implementation of the second aspect above, the physical parameters, motion command sequence, and corresponding ontological state sequence of the simulated robot are sampled to obtain a first training sample and a label for the first training sample. This includes: sampling the physical parameters of the simulated robot and configuring the sampled physical parameters into the model description file of the simulated robot, wherein the model description file is used to describe the kinematic and dynamic information of the simulated robot; sampling the motion command sequence of the simulated robot, and driving the simulated robot to execute the motion command at each moment based on the configured model description file and the motion command at each moment in the motion command sequence, thereby obtaining the ontological state parameters of the simulated robot at each moment and forming an ontological state sequence corresponding to the motion command sequence; constructing a training sample at moment i in the first training sample based on the ontological state parameters at moment i in the ontological state sequence, the motion command at moment i in the motion command sequence, and the sampled physical parameters, and determining the ontological state parameters at moment i+1 in the ontological state sequence as the label for the training sample at moment i in the first training sample.

[0029] In this embodiment, by sampling physical parameters and configuring them into a model description file, and then sampling action command sequences to drive the simulated robot to execute, dynamic data covering different physical configurations is generated. In this way, the trained differentiable neural network simulator can accurately fit the state transition relationship under different physical parameters, thus improving the generalization ability of physical alignment.

[0030] Thirdly, embodiments of this application provide a visual alignment method for embodied models. The method includes: acquiring the body state parameters of a simulated robot and the state parameters of a simulated object; generating a simulated object segmentation map and a simulated robot pose map based on the body state parameters of the simulated robot and the state parameters of the simulated object; and generating a visually aligned observation map based on the simulated object segmentation map, the simulated robot pose map, a reference observation map, and a visual alignment generation model, wherein the reference observation map is used to provide the distribution of observation maps of the environment in which the real robot is located.

[0031] In this embodiment, a segmentation map and a pose map are generated by simulating the body state parameters of the robot and the state parameters of the simulated object. Then, a reference observation map is used to provide the observation map distribution of the real environment. The visual alignment generation model is used to fuse the geometric layout information of the simulation with the visual distribution information of the real environment to generate a visually aligned observation map. In this way, the observation map output by the simulation environment is consistent with the real environment in terms of visual representation, thus narrowing the gap between simulation and reality at the visual level.

[0032] According to the third aspect, the process of acquiring training data for the visual alignment generation model includes: acquiring a sequence of real observation maps and a sequence of real action commands collected by a real robot in a real environment based on a sequence of real action commands; driving a simulated robot deployed in a simulated scene to operate a simulated object according to the sequence of real action commands, obtaining a sequence of the robot's body state and a sequence of the simulated object's state, wherein the simulated scene matches the real scene; generating a second training sample according to the sequence of the robot's body state and the sequence of the simulated object's state, wherein the training sample at time i in the second training sample includes the simulated object segmentation map at time i in the simulated object segmentation map sequence and the simulated robot pose map at time i in the simulated robot pose map sequence, where i is an integer greater than or equal to 1; determining the real observation map at time i in the sequence of real observation maps as the sample label of the training sample at time i in the second training sample, thereby obtaining the training data for the visual alignment generation model.

[0033] In this embodiment, the operation process is reproduced in a simulation scene using real action command sequences, and corresponding segmentation maps and pose maps are generated as training samples. Real observation maps are used as labels to construct training data for the visual alignment generation model. In this way, the visual alignment generation model can learn the mapping relationship from simulation observation conditions to real observation maps.

[0034] According to the third aspect, or any implementation of the third aspect above, the visual alignment generation model adopts the form of a conditional video diffusion model; based on the simulated object segmentation map, the simulated robot pose map, the reference observation map, and the visual alignment generation model, a visually aligned observation map is generated, including: summing the pixels of the simulated object segmentation map and the simulated robot pose map, and extracting features from the result of the pixel summation to obtain simulated observation features; extracting features from the reference observation map to obtain real observation features; determining the conditional signal of the visual alignment generation model based on the simulated observation features and the real observation features, and inputting the conditional signal into the visual alignment generation model to generate the visually aligned observation map.

[0035] In this embodiment, a conditional video diffusion model is used as the visual alignment generation model. The simulation observation features and the real observation features are fused as the conditional signal to guide the denoising generation process of the diffusion model. In this way, the conditional signal contains both the geometric layout information of the simulation environment and the visual information of the real environment. The generated observation map not only conforms to the spatial layout of the simulation scene, but also has the visual characteristics of the real world.

[0036] Fourthly, embodiments of this application provide an embodied model evaluation system, which includes: a physically aligned state update module, used to input an initial observation map and initial body state parameters collected by a real robot into the embodied model under test to obtain the current action command output by the embodied model under test; according to the current action command, driving a simulated robot deployed in a first simulation environment to operate a first simulated object to obtain the body state parameters of the simulated robot and the state parameters of the first simulated object, wherein the physical parameters of the simulated robot are obtained based on the trajectory of the real robot; and a visually aligned environment observation module, used to generate a first simulated object profile based on the body state parameters of the simulated robot and the state parameters of the first simulated object. The system comprises: a segmentation map and a simulated robot pose map; a reference observation map for the next time step, generated based on the first simulated object segmentation map, the simulated robot pose map, and the reference observation map, which provides the distribution of observation maps of the real robot's environment; a model simulation execution module, which inputs the visually aligned observation map for the next time step and the body state parameters of the simulated robot into the embodied model under test to obtain the next action command, and uses the next action command as the current action command, iteratively executing the simulation robot to operate the first simulated object according to the current action command until the iteration termination condition is met, and obtaining the task execution result; and an evaluation result acquisition module, which provides the evaluation results of the embodied model under test based on the task execution results.

[0037] According to the fourth aspect, the trajectory of the real robot includes the real body state sequence collected by the real robot; the system also includes a physical alignment module, which is used to: optimize the physical parameters of the differentiable neural network simulator based on the error between the real body state sequence collected by the real robot and the simulated body state sequence generated by the differentiable neural network simulator to obtain the physically aligned physical parameters; and configure the physically aligned physical parameters for the simulated robot.

[0038] According to the fourth aspect, or any implementation of the fourth aspect above, the physical alignment module is specifically used for: inputting the initial physical parameters of the real robot into the parameter mapping network to obtain the simulated physical parameters; updating the simulated physical parameters by backpropagating the gradient of the error through a differentiable neural network simulator to obtain the physically aligned physical parameters, including: backpropagating the gradient of the error through a differentiable neural network simulator to obtain the gradient of the weights of the parameter mapping network; updating the parameter mapping network according to the gradient of the weights and the optimization objective of minimizing the error to obtain the updated parameter mapping network, and determining the physical parameters output by the updated parameter mapping network as the physically aligned physical parameters.

[0039] According to the fourth aspect, or any implementation of the fourth aspect above, the visual alignment environment observation module is specifically used to: input the first simulated object segmentation map, the simulated robot pose map, and the reference observation map into the visual alignment generation model to generate the visually aligned observation map at the next moment. The training data of the visual alignment generation model includes the sequence of real observation maps pre-collected by the real robot based on the first real action command sequence, as well as the sequence of simulated object segmentation maps and the sequence of simulated robot pose maps corresponding to the first real action command sequence.

[0040] According to the fourth aspect, or any implementation of the fourth aspect above, the visual alignment generation model adopts the form of a conditional video diffusion model; the visual alignment environment observation module is specifically used for: summing pixels of the first simulated object segmentation map and the simulated robot pose map, and extracting features from the pixel summation result to obtain simulated observation features; extracting features from the reference observation map to obtain real observation features; determining the conditional signal of the visual alignment generation model based on the simulated observation features and the real observation features, and inputting the conditional signal into the visual alignment generation model to generate the visually aligned observation map at the next moment.

[0041] The fourth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the fourth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0042] Fifthly, embodiments of this application provide a physical alignment device, which includes: a real data acquisition module for acquiring a real body state sequence collected by a real robot based on a real action command sequence; a simulation data acquisition module for inputting simulation physical parameters and the real action command sequence into a differentiable neural network simulator to generate a simulation state sequence; and a physical alignment module for updating the simulation physical parameters based on the error between the real body state sequence and the simulation state sequence, through backpropagation of the gradient of the error by the differentiable neural network simulator, to obtain the physically aligned physical parameters.

[0043] According to the fifth aspect, the physical alignment module is specifically used for: inputting the initial physical parameters of the real robot into the parameter mapping network to obtain the simulated physical parameters; backpropagating the gradient of the error through a differentiable neural network simulator to obtain the gradient of the weights of the parameter mapping network; updating the parameter mapping network according to the gradient of the weights and the optimization objective of minimizing the error to obtain the updated parameter mapping network, and determining the physical parameters output by the updated parameter mapping network as the physically aligned physical parameters.

[0044] The fifth aspect and any implementation thereof correspond to the second aspect and any implementation thereof, respectively. The technical effects of the fifth aspect and any implementation thereof are similar to those of the second aspect and any implementation thereof, and will not be repeated here.

[0045] Sixthly, embodiments of this application provide a visual alignment device, which includes: a real state acquisition module for acquiring the body state parameters of a simulated robot and the state parameters of a simulated object; a simulated observation image acquisition module for generating a simulated object segmentation map and a simulated robot pose map based on the body state parameters of the simulated robot and the state parameters of the simulated object; and a visual alignment module for generating a visually aligned observation map based on the simulated object segmentation map, the simulated robot pose map, a reference observation map, and a visual alignment generation model, wherein the reference observation map is used to provide the distribution of observation maps of the environment in which the real robot is located.

[0046] According to the sixth aspect, the visual alignment generation model adopts the form of a conditional video diffusion model; the visual alignment module is specifically used for: summing pixels of the simulated object segmentation map and the simulated robot pose map, and extracting features from the pixel summation result to obtain simulated observation features; extracting features from the reference observation map to obtain real observation features; determining the conditional signal of the visual alignment generation model based on the simulated observation features and the real observation features, and inputting the conditional signal into the visual alignment generation model to generate the visually aligned observation map.

[0047] The sixth aspect and any implementation thereof correspond to the third aspect and any implementation thereof, respectively. The technical effects of the sixth aspect and any implementation thereof can be found in the technical effects of the third aspect and any implementation thereof, as described above, and will not be repeated here.

[0048] In a seventh aspect, embodiments of this application provide a computing device, including: a processor and a memory; the processor and the memory are connected; the memory is used to store one or more programs; when one or more programs are executed by one or more processors, the one or more processors implement a method as described in the first aspect, the second aspect, the third aspect, any one implementation of the first aspect, any one implementation of the second aspect, and any one implementation of the third aspect.

[0049] Eighthly, embodiments of this application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs a method as described in the first aspect, the second aspect, the third aspect, any implementation of the first aspect, any implementation of the second aspect, and any implementation of the third aspect.

[0050] Ninthly, embodiments of this application provide a computer program product containing instructions that, when executed by a computing device cluster, cause the computing device cluster to perform a method as described in the first aspect, the second aspect, the third aspect, any implementation of the first aspect, any implementation of the second aspect, and any implementation of the third aspect.

[0051] In a tenth aspect, embodiments of this application provide a computer-readable storage medium including computer program instructions, which, when executed by a computing device cluster, enable the computing device cluster to perform a method as described in the first aspect, the second aspect, the third aspect, any implementation of the first aspect, any implementation of the second aspect, and any implementation of the third aspect. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1This is a schematic diagram of an implementation scenario involving an embodied model evaluation method provided in this application embodiment.

[0054] Figure 2 This is a schematic diagram of the software architecture of an embodied model evaluation system provided in an embodiment of this application.

[0055] Figure 3 This is a flowchart illustrating an embodied model evaluation method provided in an embodiment of this application.

[0056] Figure 4 This is a schematic diagram of the physical alignment process provided in the embodiments of this application.

[0057] Figure 5 This is an example diagram illustrating the process of acquiring simulation dynamics data provided in the embodiments of this application.

[0058] Figure 6 This is an example diagram of the training process of the differentiable neural network simulator provided in the embodiments of this application.

[0059] Figure 7A This is a flowchart illustrating a physical alignment method provided in an embodiment of this application.

[0060] Figure 7B This is an example diagram of the physical alignment process provided in the embodiments of this application.

[0061] Figure 8A This is a flowchart illustrating a visual alignment method provided in an embodiment of this application.

[0062] Figure 8B This is a schematic diagram of the visual alignment process provided in the embodiments of this application.

[0063] Figure 9 This is a schematic diagram illustrating the training data acquisition process of the visual alignment generation model provided in this application embodiment.

[0064] Figure 10 These are comparative example diagrams of real-world and simulated scenarios provided in the embodiments of this application.

[0065] Figure 11 This is a schematic diagram of the training process of the visual alignment generation model provided in the embodiments of this application.

[0066] Figure 12 This is a schematic diagram of the cross-loop evaluation process provided in the embodiments of this application.

[0067] Figure 13 This is an example diagram illustrating the visual and physical alignment processes during the cross-loop evaluation provided in this application embodiment.

[0068] Figure 14This is an example diagram of the cross-loop evaluation process provided in the embodiments of this application.

[0069] Figure 15 This is one of the structural block diagrams of the embodied model evaluation system provided in the embodiments of this application.

[0070] Figure 16 This is a structural block diagram of the physical alignment device provided in the embodiments of this application.

[0071] Figure 17 This is a structural block diagram of the visual alignment device provided in the embodiments of this application.

[0072] Figure 18 This is one of the structural block diagrams of a computing device provided in the embodiments of this application.

[0073] Figure 19 This is one of the structural block diagrams of a computing device provided in the embodiments of this application.

[0074] Figure 20 This is one of the structural block diagrams of a computing device provided in the embodiments of this application.

[0075] Figure 21 This is one of the structural block diagrams of a computing device cluster provided in the embodiments of this application.

[0076] Figure 22 This is one of the structural block diagrams of a computing device cluster provided in the embodiments of this application.

[0077] Figure 23 This is one of the structural block diagrams of a computing device cluster provided in the embodiments of this application.

[0078] Figure 24 This is one of the structural block diagrams of a computing device cluster provided in the embodiments of this application.

[0079] Figure 25 This is one of the structural block diagrams of a computing device cluster provided in the embodiments of this application.

[0080] Figure 26 This is one of the structural block diagrams of a computing device cluster provided in the embodiments of this application. Detailed Implementation

[0081] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0082] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0083] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0084] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0085] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0086] To facilitate understanding of this embodiment, some technical terms and background technologies involved in this embodiment will be introduced first.

[0087] (1) Simulation to real environment (Sim2Real): The process of transferring and deploying a model and its capabilities trained or evaluated in a simulation environment to a real physical environment. The differences between simulation and real environment in terms of physical dynamics characteristics, sensor observation characteristics, etc., are usually referred to as Sim2RealGap.

[0088] (2) Physical alignment: In this embodiment, it refers to the mechanism of eliminating the difference in robot physical parameters between the simulation environment and the real environment through data-driven means, so as to ensure that the motion trajectory of the simulated robot tends to be consistent with the motion trajectory of the real robot under the same action command.

[0089] (3) Visual alignment: In this embodiment, it refers to the mechanism of eliminating the difference in visual distribution between the observation image generated by the simulation environment rendering and the observation image collected in the real environment through data-driven means, so as to ensure that the simulation observation image has the visual characteristics of the real physical environment such as lighting, texture, and sensor noise.

[0090] (4) Cross-loop evaluation: In this embodiment, the initial observation map and initial ontology state parameters collected in the real environment are used as the starting point for inference in the simulation environment. Closed-loop inference is performed on the embodied model under test in the simulation environment. Furthermore, the observation input received by the embodied model under test is a highly realistic observation image beautified by the visually aligned environment observation module, rather than the original rendered image of the simulation environment. By reproducing the physical and visual conditions of the real world in the simulation environment, the accuracy of the evaluation results is close to that of the real machine evaluation.

[0091] (5) Body state parameters: Various parameters used to describe the robot's own motion state. For example, body state parameters include, but are not limited to, kinematic and dynamic variables that change over time, such as the angular position, angular velocity, and joint torque of each joint of the robot.

[0092] Robot physical parameters: Parameters used to describe the inherent physical properties (such as mass, coefficient of friction, etc.) of a robot (such as a real robot in a real physical environment or a simulated robot deployed in a simulated environment).

[0093] (6) Differentiable Neural Network Simulator (NN Simulator): A neural network model that learns the mapping relationship between physical parameters, current body state, and current action to the next body state from the dynamic data of a simulated robot using a multilayer perceptron (MLP). Due to the continuity of the MLP, the differentiable neural network simulator is differentiable with respect to the input physical parameters, thus supporting the direct calculation of the gradient of the physical parameters through backpropagation.

[0094] (7) Multilayer perceptron (MLP): This is a fully connected neural network, meaning that every neuron in each layer is fully connected to the next layer, and layers are connected through nonlinear activation functions. MLPs can be used to approximate functions, that is, to use a parameterized network to approximate complex unknown functions with a certain margin of error. The nonlinear activation function can be, for example, a rectified linear unit (ReLU).

[0095] (8) Segmentation map: An image used to identify different objects or regions in an image. It is usually represented as a matrix of the same size as the original image, where the value of each pixel position represents the object category or instance identifier to which the pixel belongs. Visually, segmentation maps are often rendered as different colored or grayscale regions to clearly distinguish foreground objects from the background environment.

[0096] (9) Conditional Video Diffusion Model: A generative model that generates temporally consistent image sequences through a progressive denoising process guided by an additional conditional signal. The processing of input data by the conditional video diffusion model mainly includes a forward diffusion process and a reverse denoising process. In the forward diffusion process, Gaussian noise is progressively added to the images in the original video frames or image sequence until it becomes a pure noise distribution; in the reverse denoising process, guided by a given conditional signal, clear video frames are progressively recovered from the pure noise. By using the conditional video diffusion model instead of a frame-by-frame independent image generation model, the natural temporal continuity between the continuously generated multiple frames of observed images can be guaranteed.

[0097] (10) Vision Transformer (ViT): A neural network model that applies the Transformer architecture to image processing. ViT segments the input image into fixed-size image patches, linearly unfolds each image patch into a one-dimensional vector, adds a learnable feature summary label before the vector sequence, and then processes it through a multi-layer Transformer encoder. In each Transformer encoder layer, the global correlation between image patches is calculated through a multi-head self-attention mechanism, thereby extracting the global contextual features of the image.

[0098] Current embodied model evaluation solutions mainly include simulation evaluation and real-device evaluation. Simulation evaluation offers advantages in scalability and controllability, allowing for the efficient testing of embodied models by constructing numerous scenes, objects, and tasks in a virtual environment. Simulation platforms based on high-fidelity rendering and physics engines can measure metrics such as path length, number of movement steps, grasping contact stability, target approach accuracy, and behavior completion rate. Furthermore, they can artificially introduce perceptual noise, lighting perturbations, and physical friction errors to test the robustness of strategies. Simulation can also perform large-scale extreme case testing, such as object occlusion, texture randomization, random background interference, and action delay injection. These extreme case tests, crucial for improving robustness, can be implemented in simulation testing at a lower cost than real-device testing. However, simulation evaluation faces the challenge of migrating from simulation to reality (sim2real), where gaps exist between simulation and reality (such as differences in physical properties, sensor noise, and unrealistic visual rendering).

[0099] Real-world evaluation requires testing the embodied model on a real robot to assess its task completion rate, perceptual stability, physical interaction capabilities, and safety in a real physical environment. At the task execution level, it's necessary to directly measure the success rate, time taken, and operational quality of tasks such as grasping, moving, switching, organizing, and transporting in a real-world environment. At the perception level, it's necessary to examine the model's recognition and localization capabilities under realistic lighting, noise, occlusion, reflection, and texture clutter conditions. Regarding safety, it's necessary to evaluate collision risk, trajectory stability, motion acceleration, contact force, and emergency stop response capabilities to ensure that the agent will not pose a danger when deployed in homes, factories, or public spaces.

[0100] In summary, while simulation-based evaluation is fast and low-cost, it suffers from the Sim2Real Gap, leading to inaccurate results. Real-device evaluation, on the other hand, provides reliable and authentic results, but it is costly, inefficient, difficult to scale up, and its security is questionable. Therefore, finding a balance between efficiency, low cost, and accuracy in embodied model evaluation is a pressing issue that needs to be addressed.

[0101] To address this issue, this application provides an embodied model evaluation method. This method optimizes the physical parameters of the differentiable neural network simulator by pre-calculating the error between the real-body state sequence collected from a real robot and the simulated-body state sequence generated by a differentiable neural network simulator. This results in physically aligned physical parameters, which are then configured for the simulated robot, thereby reducing the difference between the simulated robot and the real robot, i.e., achieving physical alignment. Furthermore, a visual alignment generation model is trained using a sequence of real observation maps pre-collected by the real robot based on a first real action command sequence, as well as a sequence of simulated object segmentation maps and a sequence of simulated robot pose maps corresponding to the first real action command sequence. This ensures that the visual alignment model, based on the simulated object style map, the simulated robot pose map, and observation maps that provide the distribution of the initial observation map, generates a visually aligned observation map that reflects the image of the simulated physical environment in a way that is consistent with the real physical environment captured in the initial observation map, thus achieving visual alignment. In this way, by eliminating the Sim2Real Gap in the simulation evaluation environment through physical and visual alignment, a cross-loop evaluation is performed: the embodied model under test receives visually aligned observation images from the simulation environment during the evaluation process, and is driven by a simulated robot in the simulation environment configured with physically aligned physical parameters, which can significantly improve the accuracy of simulation evaluation. Therefore, this embodied model evaluation method achieves more accurate and reliable simulation evaluation by eliminating the Sim2Real Gap in both visual and physical dimensions, thus balancing high efficiency, low cost, and accuracy in embodied model evaluation.

[0102] In another implementation, the embodied model evaluation method provided in this application optimizes the physical parameters of the differentiable neural network simulator by pre-calculating the error between the real ontology state sequence collected from the real robot and the simulated ontology state sequence generated by the differentiable neural network simulator. This results in physically aligned physical parameters, which are then configured for the simulated robot, thereby reducing the difference between the simulated robot and the real robot, i.e., achieving physical alignment. In this way, the embodied model under test drives a simulated robot with physically aligned physical parameters configured in the simulation environment during the evaluation process. This eliminates the Sim2Real Gap in the inherent physical parameter dimension of the simulated robot, achieving more accurate and reliable simulation evaluation. This significantly improves the accuracy of simulation evaluation, thus balancing high efficiency, low cost, and accuracy in embodied model evaluation.

[0103] In another implementation, the embodied model evaluation method provided in this application trains a visual alignment generation model using a sequence of real observation maps pre-collected by a real robot based on a first real action command sequence, as well as a sequence of simulated object segmentation maps and a sequence of simulated robot pose maps corresponding to the first real action command sequence. This ensures that the visual alignment model can generate a visually aligned observation map that reflects the image of the simulated physical environment, which is consistent with the real physical environment of the initial observation map, based on the simulated object style map, the simulated robot pose map, and the observation maps that provide the distribution of the initial observation map. In this way, the embodied model under test receives a visually aligned observation map of the simulated environment during the evaluation process, eliminating the Sim2Real Gap in the physical environment observed by the simulated robot, i.e., the visual dimension, achieving more accurate and reliable simulation evaluation. This can significantly improve the accuracy of simulation evaluation, thus balancing high efficiency, low cost, and accuracy in embodied model evaluation.

[0104] To facilitate understanding, before describing the technical solutions of the embodiments of this application, the implementation scenarios of the embodied model evaluation method of the embodiments of this application will first be described in conjunction with the accompanying drawings.

[0105] Figure 1 This is a schematic diagram illustrating an implementation scenario of an embodied model evaluation method provided in this application. For example... Figure 1 As shown, the embodied model evaluation method provided in this application embodiment can be applied to cloud service system 100. Cloud service system 100 may include public cloud 101 and one or more tenants (here, one tenant 102 is taken as an example).

[0106] Among them, the public cloud 101 may include a embodied model evaluation platform 103 for communication connectivity and cloud infrastructure 104.

[0107] The embodied model evaluation platform 103, also known as a cloud platform or simply a cloud management platform, is a software system used by cloud providers to offer cloud technology (also known as cloud computing) services. It can be used to provide embodied model evaluation services through cloud infrastructure 104; that is, embodied model evaluation services are a type of cloud service. In one example, the embodied model evaluation platform 103 can be used to manage cloud infrastructure 104, for example, by creating virtual instances within the cloud infrastructure.

[0108] Cloud infrastructure 104 refers to the hardware devices that provide cloud services. Cloud infrastructure 104 may include multiple data centers (DCs) located in different regions, with at least one data center in each region. Each data center may contain multiple physical servers, and each physical server can be used to support various cloud services. For example, physical servers can be bare-metal servers; there are no restrictions on this.

[0109] Cloud services may include embodied model evaluation services. In one example, cloud services may also include training services for differentiable neural network simulators and / or training services for visual alignment generative models. Devices or functions accessible to tenant 102 upon logging into embodied model evaluation platform 103 can be considered as cloud services provided by cloud infrastructure 104.

[0110] like Figure 1 As shown, the embodied model evaluation platform 103 provides an interface related to cloud services for tenant 102 (or a client) to remotely access cloud services. Tenant 102 can log in to the embodied model evaluation platform 103 through a pre-registered account and password on the cloud service access page, and after successful login, purchase and use the corresponding cloud services on the cloud service access page. Since the embodied model evaluation platform 103 is communicatively connected to the cloud infrastructure 104, the embodied model evaluation platform 103 can provide various cloud services supported by the cloud infrastructure 104 and purchased by tenant 102 to tenant 102 for use.

[0111] In one example, Figure 1 The client shown can be a terminal device used by tenant 102 that can access cloud services, or client software such as a browser or application (App) on the terminal device. This terminal device can include, but is not limited to, mobile phones, tablets, computers, personal computers (PCs), and devices in internet systems. For ease of understanding and description, the term "client" will be used to refer to the client software in the subsequent embodiments of the embodied model evaluation method. In specific applications, the "client" in the embodiments of the embodied model evaluation method can be replaced with the terminal device.

[0112] Tenants can send instructions to the embodied model evaluation platform 103 via their client 2 to create, manage, log in to, and operate virtual instances on the infrastructure's servers, and use the cloud services provided by these virtual instances, such as the embodied model evaluation service. For example, the embodied model evaluation platform 103 can provide an access interface. This access interface can be provided in the form of a user interface or an API. Tenants can use client 2 to remotely access the access interface to register a cloud account and password on the embodied model evaluation platform 103, and log in to the platform using the cloud account and password. The embodied model evaluation platform 103 can also authenticate the cloud account and password. After successful authentication, tenants can further select and purchase virtual instances of specific specifications (processor, memory, disk) on the embodied model evaluation platform 103. After successfully purchasing a virtual instance, the embodied model evaluation platform 103 provides the tenant with a remote login account and password for the purchased virtual instance. Tenants can use the remote login account and password to remotely log in to the virtual instance on their client and use the embodied model evaluation service provided by the virtual instance.

[0113] In one implementation, the cloud service system 100 can implement the embodied model evaluation method provided in this application embodiment by running the executable program of the embodied model evaluation method provided in this application embodiment. Furthermore, the executable program implementing the embodied model evaluation method can optionally be presented in the form of an application installation package. After the computing nodes in the cloud infrastructure 104 install the application installation package, they can implement the embodied model evaluation method provided in this application embodiment by running the executable program therein.

[0114] For example, Figure 2 This is a schematic diagram of the software architecture of an embodied model evaluation system provided in an embodiment of this application. For example... Figure 2 As shown, the embodied model evaluation system may include: The physically aligned state update module is used to input the initial observation map and initial body state parameters collected by the real robot into the embodied model under test to obtain the current action command output by the embodied model under test; according to the current action command, it drives the simulated robot deployed in the first simulation environment to operate the first simulated object, and obtains the body state parameters of the simulated robot and the state parameters of the first simulated object. The simulated robot is configured with physically aligned physical parameters, which are used to indicate the error between the real body state sequence collected in advance based on the real robot and the simulated body state sequence generated by the differentiable neural network simulator, and the parameters obtained by optimizing the physical parameters of the differentiable neural network simulator. The visual alignment environment observation module is used to generate a first simulated object segmentation map and a simulated robot pose map based on the body state parameters of the simulated robot and the state parameters of the first simulated object. The first simulated object segmentation map, the simulated robot pose map, and the reference observation map are input into the visual alignment generation model to generate the visually aligned observation map at the next moment. The reference observation map is used to indicate the distribution of the observation map that can provide the initial observation map. The training data of the visual alignment generation model includes a sequence of real observation maps pre-collected by the real robot based on a first real action command sequence, as well as a sequence of simulated object segmentation maps and a sequence of simulated robot pose maps corresponding to the first real action command sequence. The model simulation operation module is used to input the visually aligned observation image and the body state parameters of the simulated robot into the embodied model under test at the next moment, obtain the next action command, and use the next action command as the current action command. Iteratively execute the simulation robot to operate the first simulated object according to the current action command until the embodied model under test outputs a task completion signal or reaches the preset number of iterations, then terminate the iteration and obtain the task execution result. The evaluation result acquisition module is used to provide evaluation results for the embodied model under test based on the task execution results.

[0115] It should be understood that the above content is an exemplary description of the implementation scenarios and embodied model evaluation system provided in the embodiments of this application, and does not constitute a limitation on the implementation scenarios and embodied model evaluation system. Those skilled in the art will understand that as business needs change, the implementation scenarios and specific system structures can be adjusted according to application requirements, and the embodiments of this application do not impose specific limitations on them. For example, the embodiments of this application are also applicable to users who pay to obtain the executable program of the embodied model evaluation method provided in the embodiments of this application. Furthermore, the executable program implementing the embodied model evaluation method can be presented in the form of an application installation package. After the user's local computing device or private cloud computing node installs the application installation package, they can implement the embodied model evaluation method provided in the embodiments of this application by running the executable program therein. Different modules in the embodied model evaluation system can be functionally aggregated, the system can contain more or fewer modules, and the corresponding system structure can be adapted accordingly.

[0116] The following is combined with Figures 3 to 14 The embodied model evaluation method provided in the embodiments of this application will be described in detail.

[0117] Example 1: For example, Figure 3 This is a flowchart illustrating an embodied model evaluation method provided in an embodiment of this application. Figure 3 As shown, the method may include the following steps: S100 inputs the initial observation map and initial body state parameters collected by the real robot into the embodied model under test to determine the current action command.

[0118] In this step, the embodied model under test refers to the robot control strategy model to be evaluated, such as a visual-language-action model. The initial observation image can be image data of the real robot in a real environment captured by a camera at the current moment. The initial ontological state parameters include ontological perception data such as the current joint angles, angular velocities, and end effector states of the real robot. The embodied model under test infers based on this input information and outputs the action command for the current moment. This action command is used to control the robot to perform corresponding operational tasks, such as grasping, moving, or rotating objects.

[0119] S101, according to the current action command, drive the simulated robot deployed in the first simulation environment to operate the first simulated object, and obtain the body state parameters of the simulated robot and the state parameters of the first simulated object, wherein the physical parameters of the simulated robot are obtained based on the trajectory of the real robot.

[0120] The simulated robot is equipped with physically aligned physical parameters, which are used to indicate the error between the real body state sequence collected in advance based on the real robot and the simulated body state sequence generated by the differentiable neural network simulator. The parameters are obtained by optimizing the physical parameters of the differentiable neural network simulator.

[0121] This step is a crucial entry point for physical alignment features in the evaluation closed loop. In traditional simulation evaluation, the physical parameters of the simulated robot (such as mass, inertia tensor, friction coefficient, joint damping, etc.) are usually set based on theoretical or empirical values. This often leads to differences between the dynamic response of the simulated robot and that of the real robot, i.e., a Sim2RealGap at the physical level. In this embodiment, the physical parameters configured for the simulated robot are not fixed theoretical values, but rather parameters optimized through physical alignment. Specifically, these parameters are obtained by comparing the motion trajectory (real body state sequence) of the real robot in the real environment with the motion trajectory (simulated body state sequence) of the simulated robot in the simulated environment, using the gradient backpropagation mechanism of a differentiable neural network simulator. In this way, when the simulated robot executes action commands, its motion trajectory, contact force changes, and other dynamic behaviors can closely approximate the performance of the real robot, thereby narrowing the physical gap. The specific implementation process of physical alignment, such as the construction and training method of the differentiable neural network simulator, will be detailed in subsequent embodiments.

[0122] S103, Based on the body state parameters of the simulated robot and the state parameters of the first simulated object, generate the segmentation map of the first simulated object and the pose map of the simulated robot.

[0123] After the simulated robot performs an action, the simulation environment updates the robot's body state parameters (such as updated joint angles) and the state parameters of the first simulated object (such as the object's position and orientation). To provide structured input for subsequent visual alignment, this step does not directly render the RGB image of the simulation environment, but instead generates a segmentation map and a pose map with clear semantic information. The first simulated object segmentation map is a binarized or semantically labeled image used to identify which pixels in the image belong to the manipulated object; the simulated robot pose map is used to identify the position and orientation of each joint or end effector of the robot in the image. These image data strip away the typically rudimentary texture and lighting information of the simulation environment, preserving the key geometric and semantic structures as the skeleton for generating high-fidelity images.

[0124] S104, Based on the first simulated object segmentation map, the simulated robot pose map, and the reference observation map, generate the observation map for the next moment, wherein the reference observation map is used to provide the distribution of the observation map of the real robot's environment.

[0125] The reference observation map is used to indicate the distribution of the observation map that can provide the initial observation map. The training data of the visual alignment generation model includes a sequence of real observation maps pre-collected by the real robot based on the first real action command sequence, as well as a sequence of simulated object segmentation maps and a sequence of simulated robot pose maps corresponding to the first real action command sequence.

[0126] This step is a crucial entry point for visual alignment features in the evaluation loop. Since the rendering engines of simulation environments often struggle to perfectly reproduce the visual characteristics of the real world, such as lighting, material textures, and sensor noise, directly using simulation renderings as the visual input to the embodied model under test can lead to discrepancies in model performance between simulation and real environments. This embodiment introduces a visual alignment generation model, which is essentially a generative model, such as a conditional video diffusion model. The visual alignment generation model uses the segmentation map and pose map generated in step S103 as geometric conditions and a reference observation map as a distribution reference for the observation images in the real environment, generating an observation map that conforms to both the geometric layout of the simulation environment and the visual characteristics of the real world. The reference observation map can be a real image captured by the robot at the initial moment or an image from a real scene dataset, providing prior information such as lighting distribution and texture style in the real world. By learning the mapping relationship between the sequence of real observation maps and the sequence of simulation geometry, the visual alignment generation model can transform rudimentary simulation geometric information into realistic visual observations, thereby reducing the Sim2Real Gap at the visual level. The specific training and inference process of the visual alignment generation model will be detailed in subsequent embodiments.

[0127] S105, input the visually aligned observation image of the next moment and the body state parameters of the simulated robot into the embodied model to be tested, obtain the next action command, and use the next action command as the current action command. Iterate and execute according to the current action command to drive the simulated robot to operate the first simulated object until the iteration termination condition is met, and obtain the task execution result.

[0128] Through the above steps, the visual input received by the embodied model under test in the evaluation loop is a high-fidelity image after visual alignment, and the received ontological state is a realistic dynamic state derived based on physical alignment parameters. In this way, the embodied model under test can make decisions in a simulated environment that more closely resembles the decision-making process in a real environment. The embodied model evaluation system iteratively executes processes S101 to S105, simulating the task execution flow of a real robot. The loop terminates when the embodied model under test determines that the task has been completed and outputs a signal, or when the simulation steps reach a preset upper limit, thus satisfying the iteration termination condition. The task execution results can include quantitative indicators such as task success, trajectory smoothness, and operation time.

[0129] S106, based on the task execution results, provides the evaluation results of the embodied model under test.

[0130] The evaluation system generates an evaluation report based on the task execution results. The evaluation report can include key indicators such as task success rate, average completion time, trajectory deviation, and number of collisions. Because the evaluation process is aligned with the real environment in both physical and visual dimensions, the evaluation results can truly reflect the performance of the embodied model under test in the real environment, thus achieving low-cost, high-efficiency, and highly reliable evaluation.

[0131] Based on the above description, this embodiment drives the simulated robot through physically aligned physical parameters, narrowing the gap in dynamic characteristics between the simulated and real environments; it also generates observation maps that closely resemble the real distribution through visual alignment, thus reducing the visual gap. This cross-loop evaluation method, which combines physical and visual alignment, integrates the physical and visual characteristics of the real world within the simulated environment. The visual input and dynamic feedback received by the embodied model under test are consistent with the real environment, and its decision-making behavior can realistically reflect its performance in the real environment. This method retains the advantages of high efficiency, low cost, and automation of simulation evaluation while improving the consistency between the evaluation results and the performance in the real physical world.

[0132] Example 2: This embodiment, based on Embodiment 1, elaborates on the specific implementation of the physical alignment module. The core objective of physical alignment is to automatically optimize physical parameters in the simulation environment through a data-driven approach, thereby making the dynamic behavior of the simulated robot approximate that of a real robot. This process relies on a key component—a differentiable neural network simulator.

[0133] For example, Figure 4 This is a schematic diagram of the physical alignment process provided in an embodiment of this application. For example... Figure 4 As shown, before driving the simulated robot according to the current action command, a differentiable neural network simulator needs to be built and trained first. Specifically, for example, Figure 5 This is an example diagram illustrating the process of acquiring simulation dynamics data provided in an embodiment of this application. For example... Figure 5 As shown, firstly, the simulated robot (such as...) Figure 5 The simulated dexterous hand (as shown) samples physical parameters, a sequence of action commands, and a corresponding first body state sequence to obtain first training samples and their labels. The physical parameters can include dynamic parameters such as mass, inertia tensor, friction coefficient, and joint damping. These parameters can be randomized during sampling to cover a broad dynamic characteristic space. The first training sample includes the sampled physical parameters, the action command at time i in the action command sequence, and the body state parameters at time i in the first body state sequence. The label of the first training sample includes the body state parameters at time i+1 in the first body state sequence, where i is an integer greater than or equal to 1. This constructs a dataset mapping from "current state, action, and physical parameters" to "next moment state." Subsequently, using the first training samples and their labels, a multilayer perceptron is trained to obtain a differentiable neural network simulator. For example, Figure 6 This is an example diagram illustrating the training process of the differentiable neural network simulator provided in this application embodiment. For example... Figure 6 As shown, this multilayer perceptron fits the state transition function of the physical system by learning from a large amount of simulated dynamic data. Because neural networks are differentiable, this simulator can not only quickly predict the state at the next moment, but also output the gradient information of that state with respect to the physical parameters to achieve automatic parameter optimization in the future.

[0134] Once a differentiable neural network simulator is available, physical parameter alignment optimization can be performed. First, a real-world state sequence is acquired from the real robot based on a second real motion command sequence. This sequence records the actual motion trajectory of the real robot when performing a specific action, serving as the target ground truth for optimization. Next, the simulated physical parameters and the second real motion command sequence are input into the differentiable neural network simulator to generate a simulated state sequence. The simulated physical parameters here can be initial guesses or default values. Due to differences between the simulated and real environments, there are often errors between the generated simulated state sequence and the real-world state sequence. The system acquires the error between the real-world state sequence and the simulated state sequence, and updates the simulated physical parameters by backpropagating the gradient of the error through the differentiable neural network simulator, obtaining the physically aligned physical parameters. This process utilizes the gradient propagation capability of the differentiable neural network simulator to transform trajectory-level errors (such as position and velocity deviations) into corrective gradients for the physical parameters. The physical parameters are continuously adjusted using a gradient descent algorithm until the error between the simulated trajectory and the real trajectory is minimized. In this way, the physical properties of the simulated robot (such as joint friction and link mass) are calibrated to the same level as those of the real robot, reducing the Sim2RealGap at the dynamic level.

[0135] As an optional implementation of this embodiment, to avoid numerical instability or getting trapped in local optima that may result from directly optimizing physical parameters, a parameter mapping network can be introduced. Before inputting the simulated physical parameters and the second real motion command sequence into the differentiable neural network simulator, the initial physical parameters of the real robot are input into the parameter mapping network to obtain the simulated physical parameters. The parameter mapping network can be a multi-layer neural network, whose function is to map the initial physical parameters of the real robot (which may be coarse measurements or theoretical values) into simulated physical parameters that are more suitable for the simulation environment. During the optimization process, the gradient of the error is backpropagated through the differentiable neural network simulator to obtain the gradient of the weights of the parameter mapping network; based on the gradient of the weights and the optimization objective of minimizing the error, the parameter mapping network is updated to obtain the updated parameter mapping network, and the physical parameters output by the updated parameter mapping network are determined as the physically aligned physical parameters. By optimizing the weights of the parameter mapping network, the optimization of physical parameters is indirectly achieved. This method constrains the parameter search space within the network weight space, improves the robustness and convergence speed of the optimization process, and can more stably find the combination of physical parameters that conforms to the real dynamic characteristics.

[0136] Based on the above description, this embodiment constructs a differentiable neural network simulator to achieve differentiable calculation of the simulation state with respect to physical parameters. Then, it automatically optimizes the simulation physical parameters using real trajectory data through gradient backpropagation. Introducing a parameter mapping network transforms the optimization process of physical parameters into the optimization process of network weights. The optimization of the network weight space is smoother and the gradient direction is more explicit compared to the original physical parameter space, avoiding numerical instability and local optima problems that may occur when directly searching in the physical parameter space. This further improves the efficiency and accuracy of obtaining the aligned physical parameters. Under this physical alignment mechanism, the simulation environment can reproduce the dynamic response of the real robot, and the dynamic behavior of the simulated robot is consistent with that of the real robot. This ensures that when evaluating embodied models based on this simulated robot—that is, the simulated robot configured with physically aligned physical parameters—the evaluation results are more accurate.

[0137] Example 3: This embodiment, based on Embodiment 1, details the specific implementation of the visual alignment module. The goal of visual alignment is to resolve the differences in visual characteristics such as texture, lighting, and noise distribution between simulated environment rendered images and real-world observed images, i.e., the visual-level Sim2Real Gap. This process relies on a visual alignment generation model, and this embodiment explains the construction method of the training data and the inference process of the visual alignment generation model.

[0138] The training data acquisition process for the visual alignment generative model employs a "simulation observation reproduction" strategy. Specifically, it acquires a sequence of real observation images and a sequence of real action commands from a real robot in a real environment. This step is typically accomplished through teleoperation or teaching, recording the visual images and corresponding action commands of the real robot when performing a specific task. Next, based on the first real action command sequence, a simulated robot deployed in a second simulated scene is driven to operate a second simulated object, resulting in a second state sequence for the simulated robot and a second state sequence for the simulated object. The second simulated scene matches the real scene. Here, the "second simulated scene" refers to a simplified simulation environment constructed to facilitate visual alignment training; its geometric layout is consistent with the real scene, but high-fidelity texture rendering is not required. By replaying the real robot's action commands in the simulation environment, the robot's motion trajectory and the object's pose changes can be reproduced. Subsequently, based on the second body state sequence of the simulated robot and the state sequence of the second simulated object, a second training sample is generated. This second training sample includes the simulated object segmentation map at time i in the simulated object segmentation map sequence and the simulated robot pose map at time i in the simulated robot pose map sequence, where i is an integer greater than or equal to 1. When generating the segmentation map and pose map, the built-in functions of the simulation engine can be used to directly obtain the object's mask and the robot's pose image, or an image segmentation algorithm (such as GroundedSAM) can be used to process the simulated rendering map. Finally, the real observation map at time i in the real observation map sequence is determined as the sample label for the training sample at time i in the second training sample, obtaining the training data for the visual alignment generation model. In this way, a dataset is constructed that maps from "simulated geometry (segmentation map and pose map)" to "real visual observation." This data construction method decouples the scene's geometry from its visual appearance, allowing the model to learn how to generate realistic visual images based on the simulated geometric information, which can then be used for subsequent visual alignment.

[0139] After obtaining the training data, the visual alignment generation model is trained using a conditional video diffusion model. The diffusion model learns a progressive denoising process to generate high-quality images. In the inference phase of the evaluation loop, the first simulated object segmentation image, the simulated robot pose image, and the reference observation image are input into the visual alignment generation model. The process of generating the visually aligned observation image for the next time step is as follows: First, the pixel summation of the first simulated object segmentation map and the simulated robot pose map is performed, and feature extraction is performed on the result to obtain the simulation observation features. The pixel summation operation fuses the object mask and robot pose information onto the same feature map, forming a structured representation that includes the scene's geometric layout. Subsequently, a visual encoder (such as ViT or CNN) is used to extract features from this feature map, obtaining a high-dimensional simulation observation feature vector. This feature vector accurately describes the spatial relationship between objects and the robot in the simulation environment at the current moment.

[0140] Secondly, feature extraction is performed on the reference observation map to obtain the true observation features. The reference observation map can be a real image captured by the real robot at the initial moment, containing visual information such as lighting, material, and background texture of the real scene. Through feature extraction, the true observation feature vector captures the visual distribution characteristics of the real world. In another example, after generating the visually aligned observation map for the next moment based on the real image captured by the real robot at the initial moment (i.e., the initial observation map), the visually aligned observation map at each moment can be used as the reference observation map for the next moment.

[0141] Next, based on simulated and real observation features, a conditional signal for the visual alignment generation model is determined. This conditional signal is then input into the model to generate the visually aligned observation image for the next time step. The conditional signal integrates information from both the geometric structure and the distribution of the observed image in the real environment. During the denoising generation process of the diffusion model, guided by this conditional signal, the model starts with a purely noisy image, gradually removing noise until it generates an observation image that conforms to both the simulated geometric layout and realistic visual characteristics. Specifically, at each denoising step, the diffusion model predicts the noise component in the current image based on the conditional signal and subtracts it. After multiple iterations of denoising, the image gradually becomes clearer, presenting a realistic visual effect. Thus, the embodied model under test no longer sees a simulation rendering that differs greatly from the real environment in the simulation evaluation, but rather a more realistic image, thereby reducing the Sim2Real Gap at the visual level and improving the credibility of the evaluation.

[0142] Example 4: This embodiment provides a physical alignment method. This method can be used to align the physical parameters of simulated robots, addressing the differences in dynamic characteristics between the simulated and real environments, i.e., eliminating the Sim2Real Gap at the physical level. By automatically optimizing the simulation's physical parameters through a data-driven approach, the simulated robot's dynamic behavior closely approximates that of a real robot, thereby ensuring consistency at the physical level in subsequent simulation evaluations, strategy training, and other applications.

[0143] Specifically, Figure 7AThis is a flowchart illustrating a physical alignment method provided in an embodiment of this application. Figure 7A As shown, the method includes the following steps: S210, acquire the real body state sequence of the real robot based on the real action command sequence.

[0144] Specifically, the robot is controlled to execute a series of preset motion commands (such as joint movements and end effectors) in a real environment. The robot's body state parameters, such as angles, angular velocities, and joint torques, are recorded by data acquisition devices like joint encoders and torque sensors, forming a real body state sequence. This sequence reflects the robot's dynamic response under its actual physical parameters and serves as the benchmark data for subsequent physical alignment. It should be understood that the real motion command sequence can be a pre-designed standard motion sequence or a sequence of motions naturally generated by the robot while performing a specific task, as long as it covers a sufficiently rich range of dynamic behaviors.

[0145] S220 inputs the simulated physical parameters and the sequence of real action commands into the differentiable neural network simulator to generate a simulation state sequence.

[0146] The simulated physical parameters are the initial or current values ​​of the robot's physical parameters configured in the simulation environment. The differentiable neural network simulator is a neural network model capable of predicting the state at the next moment based on the input physical parameters, action commands, and the current state. This model is differentiable with respect to the physical parameters, meaning it can calculate the gradient of the state output with respect to the physical parameters. The simulated physical parameters and the same sequence of real action commands as in step S210 are input into the differentiable neural network simulator. The simulation model then progressively extrapolates according to the sequence of real action commands, generating a simulated state sequence. This simulated state sequence reflects the dynamic response of the simulated robot under the current simulated physical parameter configuration.

[0147] As one way of implementation, Figure 7B This is an example diagram illustrating the physical alignment process provided in an embodiment of this application. For example... Figure 7B As shown, before inputting the simulated physical parameters and the real motion command sequence into the differentiable neural network simulator, the initial physical parameters of the real robot (such as the initial estimated parameters of the real robot's dexterous hand) can be input into the parameter mapping network to obtain the simulated physical parameters. The parameter mapping network is a multilayer perceptron (MLP), whose function is to map the initial physical parameters of the real robot (such as parameters obtained from the real robot's specifications, estimated parameters of the real robot, or parameters obtained from rough measurements) to the initial estimated values ​​of the simulated physical parameters. The advantage of introducing the parameter mapping network is that it transforms the optimization process of physical parameters into the optimization process of network weights, simplifies the optimization process, improves the stability of parameter optimization, and avoids getting trapped in local optima when searching directly in the physical parameter space.

[0148] As another implementation, a differentiable neural network simulator can be constructed through the following process: sampling the physical parameters, action command sequence, and corresponding ontological state sequence of the simulated robot to obtain training samples and labels for the training samples. The training samples include the sampled physical parameters, the action command at time i in the action command sequence, and the ontological state parameters at time i in the ontological state sequence. The labels for the training samples include the ontological state parameters at time i+1 in the ontological state sequence. Using the training samples and their labels, a multilayer perceptron is trained to obtain the differentiable neural network simulator.

[0149] Specifically, the physical parameters of the simulated robot are randomly sampled and configured into the model description file (such as a URDF file) of the simulated robot. This model description file describes the kinematic and dynamic information of the simulated robot. A sequence of action commands is randomly sampled. Based on the configured model description file, the action commands at each moment of the action command sequence and the sampled physical parameters are input into the simulator (such as MuJoCo or PyBullet) to drive the simulated robot to execute the action commands at each moment, obtaining the body state parameters of the simulated robot at each moment, forming a body state sequence corresponding to the action command sequence. Training samples are constructed based on the body state parameters at moment i in the body state sequence, the action commands at moment i in the action command sequence, and the sampled physical parameters. The body state parameters at moment i+1 in the body state sequence are determined as the labels of the training samples. In this way, by covering dynamic data under different physical configurations, the trained differentiable neural network simulator can accurately fit the state transition relationships under different physical parameters, improving the generalization ability of physical alignment.

[0150] S230, based on the error between the real ontology state sequence and the simulated state sequence, updates the simulated physical parameters by backpropagating the gradient of the error through a differentiable neural network simulator, and obtains the physically aligned physical parameters.

[0151] Specifically, the error between the real-world state sequence obtained in step S210 and the simulated state sequence generated in step S220 is calculated. This error can be calculated using the mean squared error (MSE) function, which measures the deviation between the simulated robot's trajectory and the real robot's trajectory at each time step. Since the differentiable neural network simulator is differentiable with respect to the physical parameters, the gradient of this error with respect to the simulated physical parameters can be calculated using the backpropagation algorithm. Based on this gradient and the optimization objective of minimizing the error (such as gradient descent), the simulated physical parameters are updated. Steps S220 and S230 are executed iteratively, i.e., repeatedly generating the simulated state sequence, calculating the error, backpropagating the gradient, and updating the physical parameters, until the error converges to a preset threshold or reaches a preset number of iterations. The physical parameters obtained at this point are the physically aligned physical parameters.

[0152] When a parameter mapping network is introduced, the gradient of the error is backpropagated using a differentiable neural network simulator to obtain the gradient of the weights of the parameter mapping network. Based on the gradient of the weights and the optimization objective of minimizing the error, the parameter mapping network is updated, resulting in the updated parameter mapping network. The physical parameters output by the updated parameter mapping network are then determined as the physically aligned physical parameters. In this way, the optimization process of physical parameters is transformed into the optimization process of the parameter mapping network weights. The physical parameters output by the network gradually approach the true physical parameters as training progresses, improving the stability and convergence speed of parameter optimization.

[0153] In precision manipulation scenarios involving dexterous hands, which are characterized by multiple joints and high degrees of freedom, their physical parameters (such as joint damping, link mass, and fingertip friction coefficient) have a significant impact on operational performance and are difficult to accurately measure directly. This embodiment utilizes a physical alignment method that automatically infers accurate physical parameters using only a small amount of real-machine trajectory data. Specifically, a real dexterous hand is controlled to perform a series of finger opening and closing, wrist rotation, and other actions, recording the real-machine state sequence. Initially guessed simulated physical parameters and the same sequence of action commands are input into a differentiable neural network simulator to generate a simulated state sequence. The error between the two is calculated, and the simulated physical parameters are updated through backpropagation. After iterative optimization, the joint motion trajectory and contact force changes of the simulated dexterous hand closely approximate the performance of the real dexterous hand, ensuring consistency at the physical level in subsequent simulation evaluations or strategy training.

[0154] Based on the above description, this embodiment calculates the error between the simulated state sequence and the real-world state sequence using a differentiable neural network simulator, and updates the simulated physical parameters using the gradient backpropagation of the error. This achieves automatic alignment of the physical parameters of the simulated environment with those of the real environment, narrowing the gap in dynamic characteristics between the simulation and reality. This method is independent of the evaluation loop and focuses on the Sim2Real Gap problem at the physical level. The physically aligned physical parameters can be configured in the simulated robot in cross-loop evaluation to ensure dynamic consistency, or they can be applied independently to scenarios requiring physical parameter calibration (such as policy training, dynamic simulation, etc.).

[0155] Example 5: This embodiment provides a visual alignment method. This method is applicable to scenarios where it is necessary to eliminate the visual differences between simulated and real environments. For example, when training or evaluating embodied models that rely on visual perception in a simulated environment, the images rendered by the simulation engine differ from the real environment in visual characteristics such as texture, lighting, and noise, causing a significant decrease in the embodied model's visual perception ability after being transferred to the real environment. This method generates a model through visual alignment, transforming the structured geometric information in the simulated environment into an observation map consistent with the visual distribution of the real environment, thereby eliminating the Sim2Real Gap at the visual level.

[0156] Specifically, Figure 8A This is a flowchart illustrating a visual alignment method provided in an embodiment of this application. Figure 8A As shown, the method includes the following steps: S310: Obtain the body state parameters of the simulated robot and the state parameters of the simulated object.

[0157] Specifically, Figure 8B This is a schematic diagram of the visual alignment process provided in an embodiment of this application. Figure 8B As shown, after the simulated robot executes motion commands in the simulation environment, the simulation engine outputs the robot's body state parameters (such as the angles and angular velocities of each joint) and the simulated object's state parameters (such as position and orientation). These state parameters reflect the geometric and kinematic information of the robot and the object in the simulation environment and are the basis for subsequent generation of segmentation maps and pose maps. It should be understood that the robot's body state parameters and the simulated object's state parameters can be obtained directly from the simulation engine, or they can be obtained after driving the simulated robot to execute specific motion commands in the simulation environment.

[0158] S320 generates a segmentation map of the simulated object and a pose map of the simulated robot based on the body state parameters of the simulated robot and the state parameters of the simulated object.

[0159] Specifically, using the built-in functions of the simulation engine or image segmentation algorithms, a simulation object segmentation map and a simulation robot pose map are generated based on the geometric information of the robot and objects in the simulation environment. The simulation object segmentation map identifies the pixel regions of the simulation objects in the image, that is, which object each pixel belongs to; the simulation robot pose map identifies the spatial positions of each joint and end effector of the simulation robot.

[0160] S330 generates a visually aligned observation map based on the simulated object segmentation map, the simulated robot pose map, the reference observation map, and the visual alignment generation model. The reference observation map is used to provide the distribution of observation maps in the real robot's environment.

[0161] The reference observation map is used to indicate the distribution of observation maps that can provide real observation maps. The training data for the visual alignment generation model includes a sequence of real observation maps collected by the real robot based on the real action command sequence, as well as a sequence of simulated object segmentation maps and a sequence of simulated robot pose maps corresponding to the real action command sequence.

[0162] The visual alignment generative model receives the segmentation map and pose map generated in step S320, as well as the reference observation map, and transforms this structured geometric information into realistic visual observation maps through a generative model. Specifically, by learning the mapping relationship between the sequence of real observation maps and the sequence of simulated geometry, the visual alignment generative model can generate observation maps that both conform to the geometric layout of the simulation environment and possess the lighting, texture, and noise characteristics of the real world. In this way, the observation maps generated in the simulation environment can be consistent with the observation maps in the real environment in terms of distribution, narrowing the gap between simulation and reality at the visual level.

[0163] For example, Figure 10 These are comparative example diagrams of real-world and simulated scenarios provided in the embodiments of this application. For example... Figure 10 As shown, in a scenario where a dual-armed robot presses a button on a table, the wrapper image rendered by the simulation environment typically only contains simple color blocks and geometric outlines, lacking realistic tabletop texture, button feel, and lighting and shadows. This results in the embodied model, which relies on visual perception, failing to effectively transfer the strategies trained in the simulation to the real environment. Through the visual alignment method in this embodiment, the segmentation map generated in the simulation environment identifies the pixel region of the button, the pose map identifies the spatial position of the two arms, and the reference observation map contains the distribution of observation images of the table where the button is actually placed. The visual alignment generation model uses this information as a condition to generate images with realistic tabletop texture, button feel, and lighting and shadows, as well as images of the two arms with realistic metallic sheen and shadows. In this way, the visual input received by the embodied model in the simulation is consistent with the real environment, reducing the Sim2Real Gap at the visual level.

[0164] As one implementation method, the process of acquiring training data for the visual alignment generation model includes: acquiring a sequence of real observation maps and a sequence of real action commands collected by a real robot in a real environment based on real action command sequences; driving a simulated robot deployed in a simulated scene to operate simulated objects according to the real action command sequences, obtaining the body state sequence of the simulated robot and the state sequence of the simulated objects, wherein the simulated scene matches the real scene; generating training samples according to the body state sequence of the simulated robot and the state sequence of the simulated objects, wherein the training samples include the simulated object segmentation map at time i in the simulated object segmentation map sequence and the simulated robot pose map at time i in the simulated robot pose map sequence; determining the real observation map at time i in the real observation map sequence as the sample label of the training sample at time i in the training samples, thereby obtaining the training data for the visual alignment generation model.

[0165] For example, Figure 9 This is a schematic diagram illustrating the training data acquisition process of the visual alignment generation model provided in this application embodiment. For example... Figure 9 As shown, real-machine training data, namely, the real observation map sequence and the real action command sequence, can be acquired using methods such as teleoperation, master-slave, and motion capture. For simulation observation and reproduction, which involves driving a simulated robot deployed in a simulation scene to operate simulated objects based on the real action command sequence, the goal is to obtain the state sequence of the robot and the state sequence of the simulated object at the corresponding moment. A simplified operation scene can be constructed. Based on this simplified scene, which matches the real scene, the motion data acquired by the real machine is replayed to reproduce the current pose of the robot arm and the observed manipulated object. Automatic segmentation is then performed using GroundSAM to generate a segmentation map of the manipulated object.

[0166] For example, Figure 11 This is a schematic diagram illustrating the training process of the visual alignment generation model provided in an embodiment of this application. For example... Figure 11 As shown, the operation process is reproduced in the simulation scene by using real action command sequences and corresponding segmentation maps and pose maps are generated as training samples. The real observation map is used as the label to construct the training data of the visual alignment generation model. The visual alignment generation model can learn the mapping relationship from simulation observation conditions to real observation maps.

[0167] As an alternative implementation, the visual alignment generation model adopts the form of a conditional video diffusion model; Accordingly, the simulated object segmentation map, the simulated robot pose map, and the reference observation map are input into the visual alignment generation model to generate a visually aligned observation map. Specifically, this may include: summing the pixels of the simulated object segmentation map and the simulated robot pose map, and extracting features from the summed pixels to obtain simulated observation features; extracting features from the reference observation map to obtain real observation features; and determining the conditional signal of the visual alignment generation model based on the simulated observation features and the real observation features, and inputting the conditional signal into the visual alignment generation model to generate a visually aligned observation map.

[0168] Specifically, see still Figure 8B The simulation object segmentation map and the simulation robot pose map are summed at the pixel level to obtain a composite map that integrates object region and robot pose information. Features are extracted from this composite map using a convolutional neural network to obtain simulation observation features, which encode the geometric layout information of the simulation environment. Simultaneously, features are extracted from a reference observation map to obtain real observation features, which encode the visual information of the real environment (such as lighting, texture, and noise distribution). The simulation and real observation features are fused into a conditional signal, which is input into a conditional video diffusion model. A visually aligned observation map is generated through a denoising process. By employing a conditional video diffusion model and using simulation and real observation features as conditional signals, an observation map that both conforms to the layout of the simulation scene and possesses real-world visual characteristics can be generated, narrowing the gap between simulation and reality at the visual level.

[0169] Based on the above description, this embodiment generates a visually aligned observation map by using a simulated object segmentation map, a simulated robot pose map, and a reference observation map as conditions, thereby narrowing the gap in visual representation between the simulated and real environments. This method is independent of the evaluation loop and focuses on the Sim2Real Gap problem at the visual level. The visually aligned observation map can be used as visual input to the embodied model under test in cross-loop evaluation to ensure visual consistency, or it can be applied independently to scenarios requiring improved visual fidelity (such as policy training and visual data augmentation).

[0170] Example 6: This embodiment uses a scenario of a dexterous hand manipulating precision parts as an example to illustrate in detail the application of the aforementioned physical alignment method in the calibration of complex dynamic parameters. Dexterous hands are characterized by multiple joints and high degrees of freedom. Their physical parameters (such as joint damping, link mass, and fingertip friction coefficient) have a significant impact on operational performance, and accurate values ​​are difficult to obtain through direct measurement. If the physical parameters in the simulation environment are not set accurately, it will lead to a large deviation between the simulation results and the real situation, failing to effectively support the transfer of the embodied model from simulation to reality. This embodiment uses physical alignment technology to deduce accurate physical parameters using a small amount of real machine trajectory data, thereby reproducing realistic contact dynamics behavior in the simulation environment.

[0171] Specifically, the physical alignment task in this embodiment is set as "rotational insertion," which involves controlling a dexterous hand to rotate a precision part and insert it into a target hole. This task involves complex contact forces and precise force control operations, requiring extremely high accuracy in physical parameters. The physical alignment process is as follows: Step S610: Obtain the real body state sequence collected by the dexterous hand based on the real action command sequence.

[0172] Specifically, Figure 12 This is a schematic diagram of the cross-loop evaluation process provided in an embodiment of this application. For example... Figure 12 As shown, a real dexterous hand is controlled to execute a series of preset motion commands (such as finger opening and closing, wrist rotation, etc.) in a real environment. The angles, angular velocities, and joint torques of each joint of the dexterous hand are recorded by joint encoders and torque sensors to form a sequence of real-body states. This sequence reflects the dynamic response of the real dexterous hand under specific physical parameters.

[0173] Step S620: Input the simulated physical parameters and the real motion command sequence into the differentiable neural network simulator to generate a simulated state sequence. Specifically, input the initially guessed simulated physical parameters (such as joint damping, link mass, etc.) and the real motion command sequence collected in step S610 into the differentiable neural network simulator. This model predicts the motion state of the dexterous hand under given physical parameters and motion commands, and outputs the simulated state sequence. As one implementation method, the differentiable neural network simulator can be constructed through the following process: sample the physical parameters, motion command sequence, and corresponding body state sequence of the simulated dexterous hand to obtain training samples and labels for the training samples. The training samples include the sampled physical parameters, the motion command at time i in the motion command sequence, and the body state parameters at time i in the body state sequence. The labels of the training samples include the body state parameters at time i+1 in the body state sequence. Use the training samples and their labels to train a multilayer perceptron to obtain the differentiable neural network simulator. In this way, the differentiable neural network simulator can quickly predict the motion state of a dexterous hand and provide gradient information of the state with respect to physical parameters, supporting subsequent parameter optimization through differentiable calculations. As another implementation, before inputting the simulated physical parameters into the differentiable neural network simulator, the initial physical parameters of the real dexterous hand can be input into a parameter mapping network to obtain the simulated physical parameters. The parameter mapping network maps the real physical parameters to the simulated physical parameter space, serving as the initial search point for subsequent gradient optimization, which can improve the stability of parameter optimization.

[0174] Step S630: Based on the error between the real entity state sequence and the simulated state sequence, the simulated physical parameters are updated by backpropagating the gradient of the error through a differentiable neural network simulator, resulting in physically aligned physical parameters. Specifically, the error (such as joint angle error and torque error) between the real entity state sequence collected in step S610 and the simulated state sequence generated in step S620 is calculated. Since the differentiable neural network simulator is differentiable, the gradient of the error with respect to the physical parameters can be calculated using the backpropagation algorithm, and the simulated physical parameters are updated using the gradient descent method. This process is iteratively executed until the error converges, resulting in physically aligned physical parameters consistent with the dynamic characteristics of the real dexterous hand. When a parameter mapping network is used, the gradient of the error is backpropagated through the differentiable neural network simulator to obtain the gradient of the weights of the parameter mapping network; based on the gradient of the weights and the optimization objective of minimizing the error, the parameter mapping network is updated, resulting in an updated parameter mapping network, and the physical parameters output by the updated parameter mapping network are determined as the physically aligned physical parameters. This improves the stability of parameter optimization and avoids getting trapped in local optima.

[0175] This embodiment utilizes physical alignment technology and the gradient backpropagation capability of a differentiable neural network simulator to automatically optimize the simulation's physical parameters, ensuring they align with the dynamic characteristics of a real dexterous hand. Thus, after configuring the physically aligned physical parameters in the simulation environment, the simulated dexterous hand's motion trajectory, contact force changes, and other dynamic behaviors can closely approximate the performance of a real dexterous hand, supporting the accuracy requirements of physical parameters during the transfer of embodied models from simulation to reality.

[0176] Example 7: This embodiment uses a dual-arm robot sorting parcels as an example to illustrate in detail the application of the aforementioned embodied model evaluation method in complex interactive tasks. Dual-arm sorting tasks require the robot to possess high-precision visual perception capabilities to identify parcels of different sizes and label information. Simultaneously, it requires precise coordination between the two arms to complete grasping, handling, and placement actions, while avoiding collisions between the arms. This scenario involves both complex dynamic interactions and extremely high demands on visual perception, making it the best vehicle to demonstrate the advantages of collaborative work between physical and visual alignment.

[0177] Specifically, the evaluation process in this embodiment constructs a high-fidelity cross-loop evaluation environment, and the specific steps are as follows: Step S710: Input the initial observation map and initial body state parameters collected by the real robot into the embodied model under test to obtain the current action command output by the embodied model under test.

[0178] For example, Figure 13 This is an example diagram illustrating the visual and physical alignment processes during the cross-loop evaluation provided in this application embodiment. For example... Figure 13 As shown, assuming a real dual-arm robot is located at a sorting station, the initial observation image is a sorting basket image captured by a camera, containing haphazardly stacked express parcels. The initial body state parameters include the angles and angular velocities of each joint of the two arms, as well as the opening and closing states of the end effectors. Based on these inputs, the embodied model under test (such as a visual language action model) infers the action command at the current moment, for example, "move the left arm above parcel A, and the right arm above parcel B."

[0179] Step S720: According to the current action command, drive the simulation robot deployed in the first simulation environment to operate the first simulation object, and obtain the body state parameters of the simulation robot and the state parameters of the first simulation object.

[0180] The simulated robot is configured with physically aligned physical parameters. For example, Figure 13 This is an example diagram illustrating the visual and physical alignment processes during the cross-loop evaluation provided in this application embodiment. For example... Figure 13As shown, the accuracy of physical parameters is crucial in dual-arm sorting scenarios. For example, during coordinated grasping by both arms, if the joint damping or link mass in the simulation environment is inconsistent with that of the real robot, it will cause deviations between the simulated trajectory and the real trajectory at high speeds, leading to collisions on the planned collision-free path in the simulation in the real environment. This embodiment utilizes the physical alignment method described in Embodiment 2 to pre-collect trajectory data of the real dual-arm robot when performing coordinated actions. The physical parameters of the simulated robot are optimized using a differentiable neural network simulator. In this way, the dynamic response of the simulated robot is consistent with that of the real robot, and its motion trajectory, inertial effects, and contact force changes can realistically reflect the characteristics of the physical world when executing action commands.

[0181] Step S730: Generate a segmentation map of the first simulated object and a pose map of the simulated robot based on the body state parameters of the simulated robot and the state parameters of the first simulated object.

[0182] After updating the joint states and the position and pose of the wrapped parts of the arms, the simulation environment generates corresponding segmentation and pose maps. The first simulated object segmentation map identifies the pixel regions wrapped in the image, while the simulated robot pose map identifies the spatial positions of the joints of the arms and the end effector.

[0183] Step S740: Input the first simulated object segmentation map, the simulated robot pose map, and the reference observation map into the visual alignment generation model to generate the visually aligned observation map for the next time step.

[0184] See also Figure 13 As shown, the reference observation map can be an image of the actual sorting station at the initial moment, containing the distribution of lighting, background texture, and package label observation images in a real scene. The visual alignment generation model adopts the conditional video diffusion model trained in Example 3. This model uses the segmentation map and pose map as geometric conditions and the reference observation map as style conditions to generate an observation map that conforms to the current simulation geometry and has realistic visual characteristics. For example, the model can generate a package image with realistic cardboard texture, label text, and lighting and shadows based on the package outline in the segmentation map, and simultaneously generate a double-arm image with realistic metallic sheen and shadows based on the pose map. In this way, the visual input received by the embodied model under test is no longer a simple color block in the simulation environment, but a realistic "real" image, thereby reducing the Sim2Real Gap at the visual level.

[0185] Step S750: Input the visually aligned observation image of the next moment and the body state parameters of the simulated robot into the embodied model under test to obtain the next action command. Use the next action command as the current action command and iteratively execute the simulation robot to operate the first simulated object according to the current action command until the embodied model under test outputs a task completion signal or reaches the preset number of iterations, then terminate the iteration and obtain the task execution result.

[0186] For example, Figure 14 This is an example diagram of the cross-loop evaluation process provided in the embodiments of this application. For example... Figure 14 As shown, the embodied model evaluation system iteratively executes the above steps until the iteration termination condition is met. The embodied model under test makes its next decision based on realistic visual feedback and accurate dynamic state feedback, for example, "left arm descends to grab package A, right arm remains idle." Because physical alignment ensures the realism of motion feedback and visual alignment ensures the realism of visual feedback, the embodied model under test can achieve an effect that more closely resembles making decisions in a real sorting environment. During this process, the embodied model evaluation system automatically records various indicators during task execution, such as grab success rate, placement accuracy, number of collisions between the two arms, and task completion time.

[0187] Step S760: Based on the task execution results, provide the evaluation results of the embodied model to be tested.

[0188] The evaluation system generates an evaluation report based on the recorded metrics. Because the evaluation process is aligned with the real environment in both physical and visual dimensions, the evaluation results accurately reflect the performance of the tested embodied model in real-world sorting scenarios. For example, if the tested embodied model frequently experiences double-arm collisions in the simulation evaluation, it indicates that it also faces collision risks in the real environment; if the model can accurately identify waybill information and complete sorting in the simulation evaluation, it indicates that it also possesses high visual perception capabilities in the real environment.

[0189] Based on the above description, this embodiment reduces the dynamic gap in the coordination process of the two arms through a physical alignment module and reduces the visual gap in the package recognition process through a visual alignment module. This cross-loop evaluation method with dual physical and visual alignment ensures that the sorting skill evaluation conducted in the simulation environment can truly reflect the model's performance in the real environment, avoiding the high cost and safety risks of real machine testing, and achieving low-cost, high-efficiency, and high-reliability evaluation.

[0190] This application also provides an embodied model evaluation system. Figure 15 This is one of the structural block diagrams of the embodied model evaluation system provided in the embodiments of this application. For example... Figure 15 As shown, the embodied model evaluation system includes: The physical alignment state update module is used to input the initial observation map and initial body state parameters collected by the real robot into the embodied model under test to obtain the current action command output by the embodied model under test; according to the current action command, it drives the simulated robot deployed in the first simulation environment to operate the first simulated object, and obtains the body state parameters of the simulated robot and the state parameters of the first simulated object, wherein the physical parameters of the simulated robot are obtained based on the trajectory of the real robot. The visually aligned environment observation module is used to generate a first simulated object segmentation map and a simulated robot pose map based on the body state parameters of the simulated robot and the state parameters of the first simulated object; and to generate an observation map for the next moment based on the first simulated object segmentation map, the simulated robot pose map, and the reference observation map, wherein the reference observation map is used to provide the distribution of observation maps of the environment in which the real robot is located. The model simulation operation module is used to input the visually aligned observation image and the body state parameters of the simulated robot into the physical model under test to obtain the next action command, and use the next action command as the current action command. Iteratively executes the simulation robot to operate the first simulated object according to the current action command until the iteration termination condition is met, and obtains the task execution result. The evaluation result acquisition module is used to provide evaluation results for the embodied model under test based on the task execution results.

[0191] Optionally, the trajectory of the real robot includes a sequence of real-body states collected by the real robot. The system also includes a physical alignment module for: Based on the error between the real body state sequence collected by the real robot and the simulated body state sequence generated by the differentiable neural network simulator, the physical parameters of the differentiable neural network simulator are optimized to obtain the physically aligned physical parameters. Configure the physical parameters of the simulated robot after physical alignment.

[0192] Optionally, the physical alignment module is specifically used for: Obtain the real-body state sequence of the real robot based on the second real action command sequence; The simulated physical parameters and the second real action command sequence are input into the differentiable neural network simulator to generate a simulation state sequence. The error between the real ontology state sequence and the simulated state sequence is obtained, and the gradient of the error is backpropagated through a differentiable neural network simulator to update the simulated physical parameters, thus obtaining the physically aligned physical parameters.

[0193] Optionally, the physical alignment module is specifically used for: The initial physical parameters of the real robot are input into the parameter mapping network to obtain the simulated physical parameters; By backpropagating the gradient of the error using a differentiable neural network simulator, the simulated physical parameters are updated to obtain the physically aligned physical parameters, including: By using a differentiable neural network simulator, the gradient of the error is backpropagated to obtain the gradient of the weights of the parameter mapping network. Based on the gradient of the weights and the optimization objective of minimizing the error, the parameter mapping network is updated to obtain the updated parameter mapping network. The physical parameters output by the updated parameter mapping network are then determined as the physical parameters after physical alignment.

[0194] Optionally, the visually aligned environment observation module is specifically used for: The first simulated object segmentation map, the simulated robot pose map, and the reference observation map are input into the visual alignment generation model to generate the visually aligned observation map at the next moment. The training data of the visual alignment generation model includes the sequence of real observation maps pre-collected by the real robot based on the first real action command sequence, as well as the sequence of simulated object segmentation maps and the sequence of simulated robot pose maps corresponding to the first real action command sequence.

[0195] Optionally, the visual alignment generation model takes the form of a conditional video diffusion model; The visually aligned environment observation module is specifically used for: The first simulated object segmentation map and the simulated robot pose map are pixel summed, and the result of pixel summation is used to extract features to obtain the simulated observation features; Feature extraction is performed on the reference observation map to obtain the true observation features; Based on simulated and real observation features, the conditional signals for the visual alignment generation model are determined, and the conditional signals are input into the visual alignment generation model to generate the visually aligned observation map at the next time step.

[0196] For example, Figure 16 This is a structural block diagram of the physical alignment device provided in an embodiment of this application. For example... Figure 16 As shown, the physical alignment device includes: The real data acquisition module is used to acquire the real body state sequence of the real robot based on the real action command sequence; The simulation data acquisition module is used to input the simulation physical parameters and the sequence of real action commands into the differentiable neural network simulator to generate the simulation state sequence; The physical alignment module is used to update the simulation physical parameters based on the error between the real ontology state sequence and the simulation state sequence, by backpropagating the gradient of the error through a differentiable neural network simulator, and thus obtain the physically aligned physical parameters.

[0197] Optionally, the physical alignment module is specifically used for: The initial physical parameters of the real robot are input into the parameter mapping network to obtain the simulated physical parameters; By using a differentiable neural network simulator, the gradient of the error is backpropagated to obtain the gradient of the weights of the parameter mapping network. Based on the gradient of the weights and the optimization objective of minimizing the error, the parameter mapping network is updated to obtain the updated parameter mapping network. The physical parameters output by the updated parameter mapping network are then determined as the physical parameters after physical alignment.

[0198] For example, Figure 17 This is a structural block diagram of the visual alignment device provided in an embodiment of this application. For example... Figure 17 As shown, the visual alignment device includes: The real-state acquisition module acquires the body state parameters of the simulated robot and the state parameters of the simulated object; The simulation observation image acquisition module is used to generate a simulation object segmentation map and a simulation robot pose map based on the body state parameters of the simulation robot and the state parameters of the simulation object. The visual alignment module is used to generate a visually aligned observation map based on the simulated object segmentation map, the simulated robot pose map, the reference observation map, and the visual alignment generation model. The reference observation map is used to provide the distribution of observation maps in the real robot's environment.

[0199] Optionally, the visual alignment generation model takes the form of a conditional video diffusion model; The visual alignment module is specifically used for: The pixel summation of the simulated object segmentation map and the simulated robot pose map is performed, and the feature extraction of the pixel summation result is performed to obtain the simulation observation features; Feature extraction is performed on the reference observation map to obtain the true observation features; Based on simulated and real observation features, the conditional signals for the visual alignment generation model are determined, and the conditional signals are input into the visual alignment generation model to generate the visually aligned observation map.

[0200] In the embodied model evaluation system, the functions of the physical alignment state update module, the visual alignment environment observation module, the model simulation operation module, and the evaluation result acquisition module can all be implemented in software or hardware. Similarly, the functions of the real data acquisition module, simulation data acquisition module, and physical alignment module in the physical alignment device can all be implemented in software or hardware. Likewise, the functions of the real state acquisition module, simulation observation image acquisition module, and visual alignment module in the visual alignment device can all be implemented in software or hardware. For example, the implementation method of the physical alignment state update module will be described below. Similarly, the implementation methods of the visual alignment environment observation module, model simulation operation module, evaluation result acquisition module, real data acquisition module, simulation data acquisition module, physical alignment module, real state acquisition module, simulation observation image acquisition module, and visual alignment module can refer to the implementation method of the physical alignment state update module.

[0201] As an example of a software functional unit, a physically aligned state update module may include code running on compute instances. A compute instance may include at least one of a physical host (compute device), a virtual machine, or a container. Furthermore, the aforementioned compute instance may be one or more. For example, a physically aligned state update module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0202] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0203] As an example of a hardware functional unit, a physically aligned state update module may include at least one computing device, such as a server. Alternatively, a physically aligned state update module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0204] The physically aligned state update module includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the physically aligned state update module includes multiple computing devices that can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the physically aligned state update module includes multiple computing devices that can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0205] It should be noted that, in other embodiments, the physically aligned state update module can be used to execute any step in the embodied model evaluation method, the visually aligned environment observation module can be used to execute any step in the embodied model evaluation method, the model simulation running module can be used to execute any step in the embodied model evaluation method, and the evaluation result acquisition module can be used to execute any step in the embodied model evaluation method. The steps implemented by the physically aligned state update module, the visually aligned environment observation module, the model simulation running module, and the evaluation result acquisition module can be specified as needed. By implementing different steps in the embodied model evaluation method through the physically aligned state update module, the visually aligned environment observation module, the model simulation running module, and the evaluation result acquisition module, all functions of the embodied model evaluation system can be realized.

[0206] Similarly, in other embodiments, the real data acquisition module can be used to execute any step in the physical alignment method, the simulation data acquisition module can be used to execute any step in the physical alignment method, and the physical alignment module can be used to execute any step in the physical alignment method. The steps implemented by the real data acquisition module, the simulation data acquisition module, and the physical alignment module can be specified as needed. The physical alignment device can achieve all its functions by implementing different steps in the physical alignment method through the real data acquisition module, the simulation data acquisition module, and the physical alignment module respectively.

[0207] Similarly, in other embodiments, the real state acquisition module can be used to execute any step in the visual alignment method, the simulation observation image acquisition module can be used to execute any step in the visual alignment method, and the visual alignment module can be used to execute any step in the visual alignment method. The steps implemented by the real state acquisition module, the simulation observation image acquisition module, and the visual alignment module can be specified as needed. By implementing different steps in the visual alignment method through the real state acquisition module, the simulation observation image acquisition module, and the visual alignment module, all functions of the visual alignment device can be realized.

[0208] This application also provides a computing device 100. For example, Figure 18 This is one of the structural block diagrams of a computing device provided in an embodiment of this application. For example... Figure 18 As shown, the computing device 100 includes a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other via the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 100.

[0209] Bus 102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 18 The bus 102 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 102 may include a path for transmitting information between various components of the computing device 100 (e.g., memory 106, processor 104, communication interface 108).

[0210] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0211] Memory 106 may include volatile memory, such as random access memory (RAM). Memory 106 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0212] The memory 106 stores executable program code, which the processor 104 executes to implement the functions of the aforementioned physical alignment state update module, visual alignment environment observation module, model simulation operation module, and evaluation result acquisition module, thereby realizing the embodied model evaluation method. In other words, the memory 106 stores instructions for executing the embodied model evaluation method.

[0213] The communication interface 108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 100 and other devices or communication networks.

[0214] This application also provides a computing device 200. Figure 19 This is one of the structural block diagrams of a computing device provided in an embodiment of this application. For example... Figure 19 As shown, computing device 200 and Figure 18 The computing device 100 shown is similar; for the same parts, please refer to the above. Figure 18 As already described, it should be noted that the memory 106 of the computing device 200 stores executable program code, and the processor 104 executes this executable program code to implement the functions of the aforementioned real data acquisition module, simulation data acquisition module, and physical alignment module, thereby realizing the physical alignment method. That is, the memory 106 stores instructions for executing the physical alignment method.

[0215] This application also provides a computing device 300. Figure 20 This is one of the structural block diagrams of a computing device provided in an embodiment of this application. For example... Figure 20As shown, computing device 300 and Figure 18 The computing device 100 shown is similar; for the same parts, please refer to the above. Figure 18 As already described, it should be noted that the memory 106 of the computing device 300 stores executable program code, and the processor 104 executes this executable program code to implement the functions of the aforementioned real state acquisition module, simulation observation image acquisition module, and visual alignment module, thereby realizing the visual alignment method. That is, the memory 106 stores instructions for executing the visual alignment method.

[0216] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0217] For example, Figure 21 This is one of the structural block diagrams of a computing device cluster provided in the embodiments of this application. For example... Figure 21 As shown, the computing device cluster includes at least one computing device 100. The memory 106 of one or more computing devices 100 in the computing device cluster may store the same instructions for executing the embodied model evaluation method.

[0218] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store a portion of the instructions for executing the embodied model evaluation method. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for executing the embodied model evaluation method.

[0219] It should be noted that the memory 106 in different computing devices 100 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the embodied model evaluation system. That is, the instructions stored in the memory 106 of different computing devices 100 can implement the functions of one or more modules among the physically aligned state update module, the visually aligned environment observation module, the model simulation execution module, and the evaluation result acquisition module.

[0220] For example, Figure 22 This is one of the structural block diagrams of a computing device cluster provided in the embodiments of this application. For example... Figure 22 As shown, the computing device cluster includes at least one computing device 200. The memory 106 of one or more computing devices 200 in the computing device cluster may store the same instructions for executing the physical alignment method.

[0221] In some possible implementations, the memory 106 of one or more computing devices 200 in the computing device cluster may also store partial instructions for executing the physical alignment method. In other words, a combination of one or more computing devices 200 can jointly execute the instructions for executing the physical alignment method.

[0222] It should be noted that the memory 106 in different computing devices 200 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the physical alignment device. That is, the instructions stored in the memory 106 of different computing devices 200 can implement the functions of one or more modules among the real data acquisition module, the simulation data acquisition module, and the physical alignment module.

[0223] For example, Figure 23 This is one of the structural block diagrams of a computing device cluster provided in the embodiments of this application. For example... Figure 23 As shown, the computing device cluster includes at least one computing device 300. The memory 106 in one or more computing devices 300 within the computing device cluster may store the same instructions for executing the visual alignment method.

[0224] In some possible implementations, the memory 106 of one or more computing devices 300 in the computing device cluster may also store partial instructions for executing the visual alignment method. In other words, a combination of one or more computing devices 300 can jointly execute the instructions for executing the visual alignment method.

[0225] It should be noted that the memory 106 in different computing devices 300 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the visual alignment device. That is, the instructions stored in the memory 106 of different computing devices 300 can implement the functions of one or more modules among the real-state acquisition module, the simulated observation image acquisition module, and the visual alignment module.

[0226] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. For example, Figure 24 This is one of the structural block diagrams of a computing device cluster provided in the embodiments of this application. For example... Figure 24As shown, the two computing devices 100A and 100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 106 in computing device 100A stores instructions for performing the functions of a physically aligned state update module and a visually aligned environment observation module. Simultaneously, the memory 106 in computing device 100B stores instructions for performing the functions of a model simulation execution module and an evaluation result acquisition module.

[0227] Figure 24 The connection method between the computing device clusters shown can be such that, considering the embodied model evaluation method provided in this application requires processing a large amount of image data, the functions of the visually aligned environment observation module and the physically aligned state update module are delegated to the computing device 100A.

[0228] It should be understood that Figure 24 The functions of the computing device 100A shown can also be performed by multiple computing devices 100. Similarly, the functions of the computing device 100B can also be performed by multiple computing devices 100.

[0229] For example, Figure 25 This is one of the structural block diagrams of a computing device cluster provided in the embodiments of this application. For example... Figure 25 As shown, the two computing devices 200A and 200B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 106 in computing device 200A stores instructions for executing the functions of the physical alignment module. Simultaneously, the memory 106 in computing device 200B stores instructions for executing the functions of the real data acquisition module and the simulation data acquisition module.

[0230] Figure 25 The connection method between the computing device clusters shown can be such that, considering the physical alignment method provided in this application requires a large amount of model inference, the functions of the real data acquisition module and the simulation data acquisition module are delegated to the computing device 200A.

[0231] It should be understood that Figure 25 The functions of the computing device 200A shown can also be performed by multiple computing devices 200. Similarly, the functions of the computing device 200B can also be performed by multiple computing devices 200.

[0232] For example, Figure 26 This is one of the structural block diagrams of a computing device cluster provided in the embodiments of this application. For example... Figure 26As shown, the two computing devices 300A and 300B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 106 in computing device 300A stores instructions for executing the functions of the visual alignment module. Simultaneously, the memory 106 in computing device 300B stores instructions for executing the functions of the real-state acquisition module and the simulated observation image acquisition module.

[0233] Figure 26 The connection method between the computing device clusters shown can be such that, considering the visual alignment method provided in this application requires a large amount of model inference, the functions of the real state acquisition module and the simulation observation image acquisition module are delegated to the computing device 300A.

[0234] It should be understood that Figure 26 The functions of the computing device 300A shown can also be performed by multiple computing devices 300. Similarly, the functions of the computing device 300B can also be performed by multiple computing devices 300.

[0235] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to perform an embodied model evaluation method, a physical alignment method, or a visual alignment method.

[0236] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform an embodied model evaluation method, a physical alignment method, or a visual alignment method.

[0237] In this embodiment, the computing device, computing device cluster, computer storage medium, or computer program product are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0238] Any content in the various embodiments of this application, as well as any content in the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0239] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating embodied models, characterized in that, The method includes: Input the initial observation map and initial body state parameters collected by the real robot into the embodied model under test to determine the current action command; According to the current action command, drive the simulated robot deployed in the first simulation environment to operate the first simulated object, and obtain the body state parameters of the simulated robot and the state parameters of the first simulated object, wherein the physical parameters of the simulated robot are obtained based on the trajectory of the real robot; Based on the body state parameters of the simulated robot and the state parameters of the first simulated object, a segmentation map of the first simulated object and a pose map of the simulated robot are generated. Based on the first simulated object segmentation map, the simulated robot pose map, and the reference observation map, an observation map for the next moment is generated, wherein the reference observation map is used to provide the distribution of observation maps of the environment in which the real robot is located; The visually aligned observation image of the next moment and the body state parameters of the simulated robot are input into the embodied model to be tested to obtain the next action instruction. The next action instruction is used as the current action instruction. The simulation robot is driven to operate the first simulated object according to the current action instruction until the iteration termination condition is met to obtain the task execution result. Based on the task execution results, the evaluation results of the embodied model under test are provided.

2. The method according to claim 1, characterized in that, The trajectory of the real robot includes the real robot's actual body state sequence collected by the real robot; The acquisition of the physical parameters of the simulated robot includes: Based on the error between the real body state sequence collected by the real robot and the simulated body state sequence generated by the differentiable neural network simulator, the physical parameters of the differentiable neural network simulator are optimized to obtain the physically aligned physical parameters. Configure the physically aligned parameters for the simulated robot.

3. The method according to claim 2, characterized in that, The error between the real-world state sequence collected by the real robot and the simulated-world state sequence generated by the differentiable neural network simulator is used to optimize the physical parameters of the differentiable neural network simulator, resulting in physically aligned physical parameters, including: Obtain the real body state sequence of the real robot based on the second real action command sequence; The simulated physical parameters and the second real action command sequence are input into the differentiable neural network simulator to generate a simulation state sequence. Obtain the error between the real ontology state sequence and the simulated state sequence; By backpropagating the gradient of the error through the differentiable neural network simulator, the simulated physical parameters are updated to obtain the physically aligned physical parameters.

4. The method according to claim 3, characterized in that, Before inputting the simulated physical parameters and the second real action command sequence into the differentiable neural network simulator to generate the simulation state sequence, the method further includes: The initial physical parameters of the real robot are input into a parameter mapping network to obtain the simulated physical parameters; The backpropagation of the gradient of the error through the differentiable neural network simulator to update the simulated physical parameters and obtain the physically aligned physical parameters includes: The gradient of the error is backpropagated using the differentiable neural network simulator to obtain the gradient of the weights of the parameter mapping network. Based on the gradient of the weights and the optimization objective of minimizing the error, the parameter mapping network is updated to obtain the updated parameter mapping network, and the physical parameters output by the updated parameter mapping network are determined as the physical parameters after physical alignment.

5. The method according to claim 3 or 4, characterized in that, Before inputting the simulated physical parameters and the second real action command sequence into the differentiable neural network simulator to generate the simulation state sequence, the method further includes: The simulated robot is sampled for physical parameters, action command sequence, and a first body state sequence corresponding to the action command sequence to obtain a first training sample and a label for the first training sample. The first training sample includes the sampled physical parameters, the action command at time i in the action command sequence, and the body state parameters at time i in the first body state sequence. The label of the training sample at time i in the first training sample includes the body state parameters at time i+1 in the first body state sequence. Using the first training sample and its label, a multilayer perceptron is trained to obtain the differentiable neural network simulator.

6. The method according to any one of claims 1 to 5, characterized in that, The step of generating the observation map for the next moment based on the first simulated object segmentation map, the simulated robot pose map, and the reference observation map includes: The first simulated object segmentation map, the simulated robot pose map, and the reference observation map are input into the visual alignment generation model to generate the visually aligned observation map at the next moment. The training data of the visual alignment generation model includes the sequence of real observation maps pre-collected by the real robot based on the first real action command sequence, as well as the sequence of simulated object segmentation maps and the sequence of simulated robot pose maps corresponding to the first real action command sequence.

7. The method according to claim 6, characterized in that, The process of acquiring training data for the visual alignment generation model includes: Obtain the sequence of real observation images and the sequence of real action commands collected by the real robot in a real environment based on the first sequence of real action commands; Based on the first real action instruction sequence, the simulated robot deployed in the second simulation scene is driven to operate the second simulated object, thereby obtaining the second body state sequence of the simulated robot and the state sequence of the second simulated object, wherein the second simulation scene matches the real scene; A second training sample is generated based on the second body state sequence of the simulated robot and the state sequence of the second simulated object. The second training sample includes the simulated object segmentation map at time i in the simulated object segmentation map sequence and the simulated robot pose map at time i in the simulated robot pose map sequence, where i is an integer greater than or equal to 1. The real observation map at time i in the real observation map sequence is determined as the sample label of the training sample at time i in the second training sample, and the training data of the visual alignment generation model is obtained.

8. The method according to claim 6 or 7, characterized in that, The visual alignment generation model adopts the form of a conditional video diffusion model; The step of inputting the first simulated object segmentation map, the simulated robot pose map, and the reference observation map into the visual alignment generation model to generate the visually aligned observation map for the next time step includes: The first simulated object segmentation map and the simulated robot pose map are summed pixel by pixel, and the result of the pixel summation is used to extract features to obtain the simulation observation features; Feature extraction is performed on the reference observation map to obtain the true observation features; Based on the simulated observation features and the real observation features, the conditional signal of the visual alignment generation model is determined, and the conditional signal is input into the visual alignment generation model to generate the visually aligned observation map at the next time step.

9. A physical alignment method, characterized in that, The method includes: Obtain the real-world state sequence of a real robot based on real action command sequences; The simulated physical parameters and the sequence of real action commands are input into a differentiable neural network simulator to generate a simulation state sequence. Based on the error between the real ontology state sequence and the simulated state sequence, the simulated physical parameters are updated by backpropagating the gradient of the error through the differentiable neural network simulator, thus obtaining the physically aligned physical parameters.

10. The method according to claim 9, characterized in that, Before inputting the simulated physical parameters and the real action command sequence into the differentiable neural network simulator to generate the simulation state sequence, the method further includes: The initial physical parameters of the real robot are input into a parameter mapping network to obtain the simulated physical parameters; The step of backpropagating the gradient of the error through the differentiable neural network simulator to update the simulated physical parameters and obtain the physically aligned physical parameters includes: The gradient of the error is backpropagated using the differentiable neural network simulator to obtain the gradient of the weights of the parameter mapping network. Based on the gradient of the weights and the optimization objective of minimizing the error, the parameter mapping network is updated to obtain the updated parameter mapping network, and the physical parameters output by the updated parameter mapping network are determined as the physical parameters after physical alignment.

11. The method according to claim 9 or 10, characterized in that, Before inputting the simulated physical parameters and the real action command sequence into the differentiable neural network simulator to generate the simulation state sequence, the method further includes: The simulation robot is sampled for physical parameters, action command sequences, and body state sequences corresponding to the action command sequences to obtain a first training sample and a label for the first training sample. The training sample at time i in the first training sample includes the sampled physical parameters, the action command at time i in the action command sequence, and the body state parameters at time i in the body state sequence. The label of the training sample at time i in the first training sample includes the body state parameters at time i+1 in the body state sequence, where i is an integer greater than or equal to 1. Using the training samples and their labels, a multilayer perceptron is trained to obtain the differentiable neural network simulator.

12. The method according to claim 11, characterized in that, The sampling of physical parameters, action command sequences, and corresponding body state sequences of the simulated robot to obtain a first training sample and its label includes: The physical parameters of the simulated robot are sampled, and the sampled physical parameters are configured into the model description file of the simulated robot, wherein the model description file is used to describe the kinematic and dynamic information of the simulated robot; The motion command sequence of the simulated robot is sampled. Based on the configured model description file and the motion command at each moment in the motion command sequence, the simulated robot is driven to execute the motion command at each moment to obtain the body state parameters of the simulated robot at each moment, forming a body state sequence corresponding to the motion command sequence. The training sample at time i in the first training sample is constructed based on the ontology state parameters at time i in the ontology state sequence, the action command at time i in the action command sequence, and the sampled physical parameters, and the ontology state parameters at time i+1 in the ontology state sequence are determined as the label of the training sample at time i in the first training sample.

13. A visual alignment method, characterized in that, The method includes: Obtain the body state parameters of the simulated robot and the state parameters of the simulated object; Based on the body state parameters of the simulated robot and the state parameters of the simulated object, a segmentation map of the simulated object and a pose map of the simulated robot are generated. Based on the simulated object segmentation map, the simulated robot pose map, the reference observation map, and the visual alignment generation model, a visually aligned observation map is generated, wherein the reference observation map is used to provide the distribution of observation maps of the real robot's environment.

14. The method according to claim 13, characterized in that, The process of acquiring training data for the visual alignment generation model includes: Acquire the sequence of real observation images and the sequence of real action commands collected by the real robot in a real environment based on the sequence of real action commands; Based on the real action instruction sequence, the simulated robot deployed in the simulation scene is driven to operate the simulated object, thereby obtaining the body state sequence of the simulated robot and the state sequence of the simulated object, wherein the simulation scene matches the real scene; A second training sample is generated based on the body state sequence of the simulated robot and the state sequence of the simulated object. The training sample at time i in the second training sample includes the simulated object segmentation map at time i in the simulated object segmentation map sequence and the simulated robot pose map at time i in the simulated robot pose map sequence, where i is an integer greater than or equal to 1. The real observation map at time i in the real observation map sequence is determined as the sample label of the training sample at time i in the second training sample, and the training data of the visual alignment generation model is obtained.

15. The method according to claim 13 or 14, characterized in that, The visual alignment generation model adopts the form of a conditional video diffusion model; The step of generating the visually aligned observation map based on the simulated object segmentation map, the simulated robot pose map, the reference observation map, and the visual alignment generation model includes: The simulated object segmentation map and the simulated robot pose map are summed pixel by pixel, and the result of the pixel summation is used to extract features to obtain the simulation observation features; Feature extraction is performed on the reference observation map to obtain the true observation features; Based on the simulated observation features and the real observation features, the conditional signal of the visual alignment generation model is determined, and the conditional signal is input into the visual alignment generation model to generate the visually aligned observation map.

16. An embodied model evaluation system, characterized in that, The system includes: The physical alignment state update module is used to input the initial observation map and initial body state parameters collected by the real robot into the embodied model under test to obtain the current action command output by the embodied model under test; according to the current action command, drive the simulated robot deployed in the first simulation environment to operate the first simulated object to obtain the body state parameters of the simulated robot and the state parameters of the first simulated object, wherein the physical parameters of the simulated robot are obtained based on the trajectory of the real robot; The visually aligned environment observation module is used to generate a first simulation object segmentation map and a simulation robot pose map based on the body state parameters of the simulation robot and the state parameters of the first simulation object; and to generate an observation map for the next moment based on the first simulation object segmentation map, the simulation robot pose map, and a reference observation map, wherein the reference observation map is used to provide the distribution of the observation map of the environment in which the real robot is located. The model simulation running module is used to input the visually aligned observation image of the next moment and the body state parameters of the simulated robot into the embodied model under test, obtain the next action instruction, and use the next action instruction as the current action instruction to iteratively execute the operation of the simulated robot on the first simulated object according to the current action instruction until the iteration termination condition is met, and obtain the task execution result. The evaluation result acquisition module is used to provide the evaluation results of the embodied model under test based on the task execution results.

17. The system according to claim 16, characterized in that, The trajectory of the real robot includes the real robot's actual body state sequence collected by the real robot; The system also includes a physical alignment module for: Based on the error between the real body state sequence collected by the real robot and the simulated body state sequence generated by the differentiable neural network simulator, the physical parameters of the differentiable neural network simulator are optimized to obtain the physically aligned physical parameters. Configure the physical parameters of the simulated robot after physical alignment.

18. The system according to claim 17, characterized in that, The physical alignment module is specifically used for: Obtain the real body state sequence of the real robot based on the second real action command sequence; The simulated physical parameters and the second real action command sequence are input into the differentiable neural network simulator to generate a simulation state sequence. The error between the real ontology state sequence and the simulated state sequence is obtained, and the simulated physical parameters are updated by backpropagating the gradient of the error through the differentiable neural network simulator to obtain the physically aligned physical parameters.

19. The system according to claim 18, characterized in that, The physical alignment module is specifically used for: The initial physical parameters of the real robot are input into a parameter mapping network to obtain the simulated physical parameters; The backpropagation of the gradient of the error through the differentiable neural network simulator to update the simulated physical parameters and obtain the physically aligned physical parameters includes: The gradient of the error is backpropagated using the differentiable neural network simulator to obtain the gradient of the weights of the parameter mapping network. Based on the gradient of the weights and the optimization objective of minimizing the error, the parameter mapping network is updated to obtain the updated parameter mapping network, and the physical parameters output by the updated parameter mapping network are determined as the physical parameters after physical alignment.

20. The system according to any one of claims 16 to 19, characterized in that, The visually aligned environment observation module is specifically used for: The first simulated object segmentation map, the simulated robot pose map, and the reference observation map are input into the visual alignment generation model to generate the visually aligned observation map at the next moment. The training data of the visual alignment generation model includes the sequence of real observation maps pre-collected by the real robot based on the first real action command sequence, as well as the sequence of simulated object segmentation maps and the sequence of simulated robot pose maps corresponding to the first real action command sequence.

21. The system according to claim 20, characterized in that, The visual alignment generation model adopts the form of a conditional video diffusion model; The visually aligned environment observation module is specifically used for: The first simulated object segmentation map and the simulated robot pose map are summed pixel by pixel, and the result of the pixel summation is used to extract features to obtain the simulation observation features; Feature extraction is performed on the reference observation map to obtain the true observation features; Based on the simulated observation features and the real observation features, the conditional signal of the visual alignment generation model is determined, and the conditional signal is input into the visual alignment generation model to generate the visually aligned observation map at the next time step.

22. A physical alignment device, characterized in that, The device includes: The real data acquisition module is used to acquire the real body state sequence of the real robot based on the real action command sequence; The simulation data acquisition module is used to input the simulation physical parameters and the real action command sequence into the differentiable neural network simulator to generate a simulation state sequence. The physical alignment module is used to update the simulated physical parameters based on the error between the real ontology state sequence and the simulated state sequence, through the backpropagation of the gradient of the error by the differentiable neural network simulator, so as to obtain the physically aligned physical parameters.

23. The apparatus according to claim 22, characterized in that, The physical alignment module is specifically used for: The initial physical parameters of the real robot are input into a parameter mapping network to obtain the simulated physical parameters; The gradient of the error is backpropagated using the differentiable neural network simulator to obtain the gradient of the weights of the parameter mapping network. Based on the gradient of the weights and the optimization objective of minimizing the error, the parameter mapping network is updated to obtain the updated parameter mapping network, and the physical parameters output by the updated parameter mapping network are determined as the physical parameters after physical alignment.

24. A visual alignment device, characterized in that, The device includes: The real-state acquisition module acquires the body state parameters of the simulated robot and the state parameters of the simulated object; The simulation observation image acquisition module is used to generate a simulation object segmentation map and a simulation robot pose map based on the body state parameters of the simulation robot and the state parameters of the simulation object. The visual alignment module is used to generate a visually aligned observation map based on the simulated object segmentation map, the simulated robot pose map, the reference observation map, and the visual alignment generation model. The reference observation map is used to provide the distribution of observation maps of the real robot's environment.

25. The apparatus according to claim 24, characterized in that, The visual alignment generation model adopts the form of a conditional video diffusion model; The visual alignment module is specifically used for: The simulated object segmentation map and the simulated robot pose map are summed pixel by pixel, and the result of the pixel summation is used to extract features to obtain the simulation observation features; Feature extraction is performed on the reference observation map to obtain the true observation features; Based on the simulated observation features and the real observation features, the conditional signal of the visual alignment generation model is determined, and the conditional signal is input into the visual alignment generation model to generate the visually aligned observation map.

26. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 15.

27. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster performs the method as described in any one of claims 1 to 15.

28. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 15.

Images