Robot simulation method and related apparatus

Abstract

This application provides a robot simulation method and related apparatus. The method includes: obtaining first state identifier information corresponding to a state structure description from a first server and storing the first state identifier information; the state structure description is used to define the data format of a state frame, and the first state identifier information is used to uniquely identify the version of the state structure description; obtaining a state frame including second state identifier information from the first server or a second server different from the first server; if the state identifier information passes the verification based on the second state identifier information obtained from the state frame and the first state identifier information, then applying the state frame to display the robot simulation effect based on the state frame. This application can reduce rendering and coding costs, obtain the semantic state required for reproducible analysis, and ensure the consistency and reproducibility of the semantic state.

Description

Technical Field

[0001] This application relates to the field of robot simulation technology, and more specifically, to a robot simulation method and related apparatus. Background Technology

[0002] With the development of robotics technology, robot simulation has become a core infrastructure in fields such as embodied intelligence and physical AI.

[0003] However, current robot simulation technology is not only costly in terms of rendering and encoding, but also difficult to anticipate or prevent problems such as packet loss and jitter. Furthermore, it is difficult to obtain the semantic state required for reproduction analysis. When anomalies occur, the lack of semantic state that can be used to reproduce experiments and perform difference analysis makes it difficult to locate anomalies, and the traceability cost required to investigate anomalies is high. Summary of the Invention

[0004] The embodiments of this application provide a robot simulation method and related apparatus, which can at least reduce rendering and encoding costs, obtain the semantic state required for reproducible analysis, ensure the consistency and reproducibility of the semantic state, and reduce the difficulty of anomaly localization.

[0005] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0006] According to one aspect of the embodiments of this application, a robot simulation method is provided. The method includes: obtaining first state identification information corresponding to a state structure description from a first server and storing the first state identification information; the state structure description is used to define the data format of a state frame, and the first state identification information is used to uniquely identify a version of the state structure description; obtaining a state frame including second state identification information from the first server or a second server different from the first server; if it is determined that the state identification information passes the verification based on the second state identification information obtained from the state frame and the first state identification information, then applying the state frame to display the robot simulation effect based on the state frame, wherein the verification of the state identification information is used to indicate whether the version of the state structure description identified by the first state identification information matches the version of the state structure description corresponding to the state frame.

[0007] According to one aspect of the embodiments of this application, a robot simulation apparatus is provided, the apparatus comprising: a state identifier acquisition and storage unit, configured to acquire first state identifier information corresponding to a state structure description from a first server and store the first state identifier information; the state structure description is used to define the data format of a state frame, and the first state identifier information is used to uniquely identify the version of the state structure description; a state frame acquisition unit, configured to acquire a state frame including second state identifier information from the first server or a second server different from the first server; and a state frame application unit, configured to apply the state frame to display the robot simulation effect based on the state frame if it is determined that the state identifier information passes the verification according to the second state identifier information acquired from the state frame and the first state identifier information, wherein the verification of the state identifier information is used to indicate whether the version of the state structure description identified by the first state identifier information matches the version of the state structure description corresponding to the state frame.

[0008] In some embodiments of this application, based on the foregoing scheme, the state identifier acquisition and storage unit is configured to: acquire a scene release package from the first server and store the first state identifier information in the scene release package, wherein the scene release package includes visual assets, the state structure description and the first state identifier information.

[0009] In some embodiments of this application, based on the foregoing scheme, the status identifier acquisition and storage unit is configured to: acquire third status identifier information corresponding to the status structure description of the current version of the scenario release package used by the first server from the first server; if the third status identifier information is inconsistent with the status identifier information corresponding to the status structure description stored locally, then acquire the scenario release package from the first server;

[0010] or, The scene release package also includes first resource identifier information corresponding to the visual asset; the state identifier acquisition and storage unit is configured to: acquire second resource identifier information corresponding to the visual asset of the current version of the scene release package used by the first server from the first server; acquire third state identifier information corresponding to the state structure description of the current version of the scene release package used by the first server from the first server; if the third state identifier information is inconsistent with the state identifier information corresponding to the state structure description stored locally, or if the second resource identifier information is inconsistent with the resource identifier information corresponding to the visual asset stored locally, then acquire the scene release package from the first server.

[0011] In some embodiments of this application, based on the foregoing scheme, the scene release package further includes a binding manifest file, the binding manifest file includes binding mapping information, the binding mapping information includes the mapping relationship between simulation entity identifiers and client visual node identifiers; the first resource identifier information corresponding to the visual asset is generated based on the hash value of the visual asset and the hash value of the binding manifest file.

[0012] In some embodiments of this application, based on the foregoing scheme, the device is applied to a client; the status frame acquisition unit is configured to: acquire a key frame including complete status field data from the first server or a second server different from the first server; or, acquire an incremental frame including only status field data that has changed relative to the previously acquired frame or reference frame from the first server or a second server different from the first server.

[0013] In some embodiments of this application, based on the foregoing scheme, the device further includes a rejection and recovery unit; after obtaining a status frame including second status identification information from the first server or a second server different from the first server, the rejection and recovery unit is configured to: if it is determined that the status identification information has failed verification based on the second status identification information obtained from the status frame and the first status identification information, then reject the application of the status frame and trigger a target recovery strategy to restore the robot simulation process.

[0014] In some embodiments of this application, based on the foregoing scheme, the target recovery strategy includes at least one of the following: re-obtaining the scenario release package currently used by the first server from the first server, re-obtaining the state structure description and binding manifest file currently used by the first server from the first server, requesting to obtain the keyframe at the current recovery time, clearing the cached scenario release package corresponding to the current session and loading the scenario release package currently used by the first server re-obtained from the first server.

[0015] In some embodiments of this application, based on the foregoing scheme, the state structure description includes the state field order; the increment frame includes a binary sequence marker, the binary sequence marker includes multiple binary bits arranged in order, each binary bit corresponds to a state field at a corresponding position in the state field order, and the value of each binary bit represents whether the data of the corresponding state field has changed; the increment frame includes state field data where the corresponding state field has changed.

[0016] In some embodiments of this application, based on the aforementioned scheme, the state frame is generated by the server according to the simulation state code generated after the simulation engine executes control logic and advances the simulation; the device further includes a comparison and evaluation unit; the comparison and evaluation unit is used to: simulate the same scene asset using the same control logic based on at least two simulation engines, and obtain comparison index data corresponding to each simulation engine, the comparison index data being used to measure whether the running results of at least two simulation engines are consistent; determine the comparison result of the running results of the simulation engines and the reason information for the inconsistency of the running results of different simulation engines based on the comparison index data corresponding to each simulation engine, and display the comparison result and the reason information.

[0017] In some embodiments of this application, based on the foregoing scheme, the apparatus further includes a rendering capability reporting unit; before obtaining a status frame including second status identification information from the first server or a second server different from the first server, the rendering capability reporting unit is used to: report rendering capability information to the first server or a second server different from the first server, so that the first server or the second server can distribute rendering configuration or select resources based on the rendering capability information.

[0018] According to one aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the robot simulation method as described in the above embodiments.

[0019] According to one aspect of the embodiments of this application, an electronic device is provided, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the robot simulation method as described in the above embodiments.

[0020] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium, a processor of a computer device reading the computer instructions from the computer-readable storage medium, and the processor executing the computer instructions to cause the computer device to perform the robot simulation method as described in the above embodiments.

[0021] In some embodiments of this application, the technical solutions provided achieve on-client rendering and encoding by obtaining state frames from a first server or a second server different from the first server and applying the state frames, without performing rendering and encoding in the cloud. This not only reduces rendering and encoding costs but also eliminates the need to transmit video streams, thus preventing packet loss, jitter, and other issues, and reducing bandwidth costs. Since the first state identifier information corresponding to the state structure description is obtained in advance from the first server, when a state frame including the second state identifier information is obtained, it can be determined whether the state identifier information passes verification based on the second state identifier information obtained from the state frame and the first state identifier information. Only when the verification passes is the state frame applied. Therefore, it ensures that the state frame is applied only when the version of the state structure description used by the client is consistent with the version of the state structure description corresponding to the state frame. This not only obtains the semantic state required for reproducing experiments and difference analysis but also ensures the consistency and reproducibility of the client's interpretation of the state with the server's simulation semantics. This not only facilitates anomaly localization when anomalies occur, reducing the traceability costs required for troubleshooting anomalies, but also ensures the effectiveness of operations such as robot strategy evaluation.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 A flowchart of a robot simulation method according to an embodiment of this application is shown.

[0025] Figure 2 An embodiment according to this application is shown. Figure 1 A flowchart detailing step 110 in the embodiment.

[0026] Figure 3 A schematic diagram illustrating the overall architecture and process of an embodiment of this application is shown.

[0027] Figure 4 A schematic diagram illustrating the contents of a scenario release package according to an embodiment of this application is shown.

[0028] Figure 5 A timing diagram is shown illustrating how a client obtains a scene publishing package from a server according to one embodiment of this application.

[0029] Figure 6 A schematic diagram of a processing flow for verifying status identification information according to an embodiment of this application is shown.

[0030] Figure 7 A timing diagram is shown for a server synchronizing keyframes and incremental frames to a client according to an embodiment of this application.

[0031] Figure 8 A block diagram of a robot simulation apparatus according to an embodiment of this application is shown.

[0032] Figure 9 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0033] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0034] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0035] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0036] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0037] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0038] Cloud-based robot simulation is widely used in robot research and development, algorithm verification, and other scenarios. Related technologies for cloud-based robot simulation mainly include two solutions: "the combination of cloud rendering and client-side video streaming" and "the combination of client-side asset storage and state-driven rendering."

[0039] In the "cloud-based rendering and client-side video streaming" solution, video feeds are typically pushed to the client via WebRTC, RTMP, etc., with the client primarily responsible for decoding, playback, and input feedback. However, this solution has the following drawbacks: cloud GPU rendering and encoding are costly and subject to high concurrent scaling pressure; it is sensitive to bandwidth, jitter, and packet loss; the client struggles to obtain the semantic states required for reproducing experiments and differential analysis; the cost of locating and tracing anomalies is high; and when simulated entities and visual nodes are misaligned, or joint order, coordinate axes, or unit scales are inconsistent, the client's interpretation of the state will differ from the server-side simulation semantics, potentially causing playback comparison metrics to deviate from reality and affecting the validity of strategy evaluation conclusions.

[0040] In the "client-side asset storage and state-driven rendering" solution, the client caches 3D assets and robot structure descriptions, while the server pushes semantic states such as joint angles, poses, and torques based on the calculation results of the cloud simulation engine for client rendering and interaction. However, this solution has the following drawbacks: when the relationships and state structures between assets, simulation entities, and visual nodes are inconsistent with the actual session state, problems such as misbinding, misalignment, display anomalies, and even crashes can easily occur; binding rules rely on implicit conventions such as naming and hierarchy, which can easily become invalid after export changes; and the update of the state structure description lacks a unified rejection and recovery mechanism, which can easily lead to problems such as "updates during simulation causing rendering contamination and making it difficult to locate problems."

[0041] Furthermore, cloud-based robot simulation in related technologies also involves a cross-engine aligned Sim2Sim verification process. Specifically, in the cross-engine aligned Sim2Sim verification process, the training platform and the verification platform are independent of each other. After the verification process begins, assets and strategies are exported from the training platform and reproduced and compared in different simulation engines on the verification platform. Because the data statistics, coordinate axes, unit scales, joint sequences, control frequencies, and observation and motion normalization parameters of different simulation engines and platforms are not uniform, it is necessary to write import, mapping, synchronization, and evaluation code separately for different simulation engines and maintain multiple sets of data conversion links, resulting in high engineering complexity and maintenance costs. In addition, comparative evaluation of simulations from different simulation engines is prone to problems such as drift, non-reproducibility, and non-auditability, often requiring offline log playback to reduce the cost of online reproduction experiments.

[0042] Therefore, this application first provides a robot simulation method. The robot simulation method provided based on the embodiments of this application can overcome the above-mentioned deficiencies and can bring the following technical effects: By delivering visual assets, state structure descriptions, and a verification list file containing the first state identifier information corresponding to the state structure descriptions to the client in a unified delivery package, the client can perform consistency verification on the state identifier information based on the first state identifier information during simulation. Only when the state identifier information passes the consistency verification is the client allowed to parse and apply the received state frame, thus avoiding problems such as misbinding between simulated entities and visual nodes from a mechanism perspective.

[0043] By rejecting the application of the state frame when the state identifier information fails the verification and triggering recovery strategies such as re-requesting keyframes, incompatible data can be prevented from entering the rendering and interaction chain, thereby improving the stability and observability of the system and ensuring its reliable operation.

[0044] By first obtaining the second resource identifier information corresponding to the visual assets of the current version of the scene release package used by the first server, and only obtaining the scene release package from the first server when the third resource identifier information is inconsistent with the resource identifier information corresponding to the visual assets stored locally, the scene release package can be omitted when the third resource identifier information is consistent with the resource identifier information corresponding to the visual assets stored locally. This allows for the reuse of already cached scene release packages and reduces resource consumption.

[0045] Since the client can obtain keyframes containing complete state field data from the server, or incremental frames containing only state field data that has changed relative to the previous or baseline frame, bandwidth consumption and parsing overhead can be reduced. In scenarios of frame loss, out-of-order delivery, and disconnection and reconnection, the baseline state can be reconstructed by requesting keyframes and a consistency recovery process, thereby improving recoverability.

[0046] By incorporating interface contract parameters such as coordinate axes, unit scale, joint sequence, control frequency, and normalization parameters into the binding manifest file, and describing the output state frame sequence with a unified state structure, cross-engine alignment playback and comparative evaluation can be facilitated, thereby improving reproducibility, traceability, and auditability.

[0047] The core product of this application combines a cloud-based robot simulation service with a browser-based visual client, providing cloud simulation capabilities and multi-terminal collaborative verification capabilities. Typical application scenarios include ordered scenarios, unordered scenarios, and weak network scenarios.

[0048] The ordered scenarios can include standard scenarios for robot experimental teaching, benchmark testing, strategy playback and demonstration, etc., which can be used through a browser, achieving low-threshold access.

[0049] Disordered scenarios can include cloud-based simulation debugging, log playback, and problem reproduction scenarios where R&D teams collaborate in multiple locations, or scenarios where different simulation engines use the same control logic to simulate the same scenario asset or the same standardized scenario representation, and then compare and evaluate the indicator data.

[0050] Weak network scenarios can specifically include mobile office, external network access, cross-regional collaboration, etc. These scenarios require low bandwidth consumption and controllable degradation strategies. The semantic state synchronization of this application can meet the requirement of low bandwidth consumption.

[0051] In this application, cloud-based robot simulation means: performing simulation calculations and policy reasoning for robot kinematics and / or dynamics and interaction environment on the cloud or a remote server to generate robot state flow. The client obtains robot scene release packages through the network and receives robot state flow (such as data including joint angles, poses, torques, sensor data, etc.) for interaction and visualization.

[0052] The implementation details of the technical solutions in the embodiments of this application are described in detail below: Figure 1A flowchart of a robot simulation method according to an embodiment of this application is shown. This robot simulation method can be executed by various devices with processing, computing, and display functions. Specifically, it can be executed by a target device, such as a user terminal or a cloud server. User terminals include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, smartwatches, etc. Please refer to... Figure 1 As shown, this robot simulation method can be executed by the client, and it can include at least the following steps: In step 110, the first state identifier information corresponding to the state structure description is obtained from the first server and stored; the state structure description is used to define the data format of the state frame, and the first state identifier information is used to uniquely identify the version of the state structure description.

[0053] The first state identifier information may or may not be located in the release packet.

[0054] Figure 2 An embodiment according to this application is shown. Figure 1 A flowchart detailing step 110 in the embodiment is provided. Please refer to [link / reference]. Figure 2 As shown, obtaining the first state identifier information corresponding to the state structure description from the first server and storing the first state identifier information may specifically include: In step 110', a scene release package is obtained from the first server, and the first state identification information in the scene release package is stored. The scene release package includes visual assets, the state structure description, and the first state identification information.

[0055] A ScenePackage, often simply called a deployment package, is a collection of files or objects generated by the server and directly distributed to the client.

[0056] The scene release package can include visual assets (scene.glb), state structure description (state.schema.json), binding manifest file (scene.manifest.json), and checksums.json. The first state identification information can be located in the checksums.json file.

[0057] In other words, a release package can be encapsulated from visual assets, state structure descriptions, binding manifest files, and validation manifest files.

[0058] Figure 3 A schematic diagram illustrating the overall architecture and process of an embodiment according to this application is shown. Please refer to [link / reference]. Figure 3As shown, this application addresses the practical needs of cloud-based simulation computing and browser-based visualization by providing a deliverable, verifiable, and recoverable state-driven rendering closed loop. The system includes a server and a client. The server can be a cloud-based robot simulation service, responsible for building deployment packages, generating verification information, collecting, encoding, and pushing states, and managing sessions through capability negotiation with the client. The client can be a browser or a visualization terminal, responsible for deployment package management, explicit binding establishment, consistency verification gates, state application, and rendering.

[0059] Figure 4 A schematic diagram illustrating the contents of a scenario distribution package according to an embodiment of this application is shown. Please refer to... Figure 4 As shown, the release package includes visual assets, which are the resources required for simulation visualization. Visual assets are used for the mesh, materials, textures, and hierarchical structure of the client's 3D display. Visual assets can be packaged in glTF, GLB, USD, or other equivalent formats, which are not limited here; the structure and joint topology of the robot body can be described using URDF.

[0060] The binding manifest file (scene.manifest.json) is used to stably establish the mapping relationship between "simulation entities and visual nodes" and carries cross-platform contract constraints.

[0061] The binding manifest file (scene.manifest.json) can include binding manifest (binding mapping information), cross-platform contract constraints, validation fields, and policy-related metadata.

[0062] Binding mapping information is a data structure used to explicitly declare the mapping relationship between simulation entity identifiers (entityId) and client visual node identifiers (such as visualNodeId, visualPath, visualUUID). It is used to explain how server-side simulation entities correspond to nodes, bones, or paths in client-side 3D assets.

[0063] Among them, the simulation entity identifier (entityId) is the ID on the simulation semantic side, used to identify robots, robot links, joints, end effectors, sensors, obstacles, objects, etc.

[0064] visualNodeId or visualPath represents the "visual node identifier" or "visual node path" in the client's 3D visual assets, respectively. It is the ID on the rendering side and is used to locate specific nodes, bones, and meshes in assets such as glTF and GLB / USD.

[0065] Binding mapping information can also include mode, jointMap, etc.

[0066] Here, "mode" is the "binding or driving mode", which is used to characterize how server-side state data is applied to client-side visual nodes.

[0067] An example of binding mapping information could be: { "entityId":"robot_001.arm_joint_1", "visualPath":" / Scene / Robot / arm / joint_1", "mode":"joint" }

[0068] The meaning of the above binding mapping information is: the simulation joint robot_001.arm_joint_1 in the server-side state stream corresponds to the / Scene / Robot / arm / joint_1 node in the client's 3D model. After the client receives the joint angle, it drives the visual node to rotate according to the joint mode.

[0069] A jointMap, or joint mapping table, is a data structure that describes the mapping relationship between joint identifiers on the simulation side and bone or node identifiers in the visual assets. When necessary, it can also include mapping parameters such as joint order, axial direction, zero offset, and scaling factor. It is used to determine the driving relationship between joint state and visual bones or nodes in explicit binding, replacing the implicit inference mapping method based on naming or hierarchical relationship.

[0070] Cross-platform contract constraints are used to ensure consistency between import and playback. Cross-platform contract constraints can include fields such as upAxis, metersPerUnit, jointOrder, controlHz, observationNormVersion, and actionNormVersion.

[0071] Cross-platform contract constraints are used to ensure that coordinate axes, units, joint order, control frequency, normalization parameters, etc. remain consistent across different simulation engines and platforms.

[0072] Specifically, upAxis indicates which direction the scene uses as the "up direction". For example, upAxis="Z" means that the Z-axis is vertically upward, which is common in robot or simulation coordinate systems.

[0073] metersPerUnit represents how many meters in the real world one unit corresponds to in the scene asset. For example, metersPerUnit=1.0 means that one unit in the asset equals one meter.

[0074] `controlHz` represents the control frequency of the strategy or controller, that is, how many control commands are output per second. For example, `controlHz=50` means that the strategy outputs 50 action commands per second, or once every 20ms.

[0075] `jointOrder` represents the joint order in the robot's joint state array. For example: jointOrder=[ "shoulder_pan_joint", "shoulder_lift_joint", "elbow_joint", "wrist_1_joint", "wrist_2_joint", "wrist_3_joint ].

[0076] Suppose a state frame sends an array: qpos=[0.1,0.2,-0.3,0.4,0.5,0.6]. The client must know that the first value corresponds to `shoulder_pan_joint`, the second value to `shoulder_lift_joint`, and the third value to `elbow_joint`. Otherwise, if the joint order is incorrect, the problem of "applying the angle of the elbow to the shoulder" will occur, and the robot's movements will be severely erratic.

[0077] observationNormVersion represents the version of the observation normalization parameter.

[0078] Robot strategies typically do not use raw observations directly, but rather normalized observations. For example, joint angles, velocities, forces, image features, and end-effector poses may be scaled using mean or std, or min or max operations.

[0079] For example, `observationNormVersion="obs_norm_v3"` indicates that the current policy inference uses the third version of observation normalization parameters. If `obs_norm_v3` is used during training but `obs_norm_v2` is incorrectly used during validation, the policy input distribution will change, leading to abnormal action outputs.

[0080] `actionNormVersion` represents the version of the action output with inverse normalization or scaling parameters. Sometimes, the action output by the strategy is not the direct actual control quantity, but rather a normalized action, for example, within the range [-1, 1]. Before execution, it needs to be converted into the actual target joint angle, velocity, torque, or end effector displacement according to the action scaling parameters.

[0081] For example: actionNormVersion="act_norm_v2".

[0082] The state structure description (state.schema.json) is a data structure that describes the set of transmittable fields, field types, layout methods (such as field order, encoding method, quantization method), and the relationship between fields and entity identifiers in the state frame.

[0083] The state structure description is the format definition table of the state frames. Essentially, it is a mapping table. During the simulation, the simulation engine pushes state frames to the client every frame. After receiving a string of state frame data, the client should read it into the robot state according to the state structure description rules.

[0084] For example, in the state structure description, the following can be specified: frame: integer, indicating which frame; timestamp: decimal, indicating the simulation time; schemaHash: integer, indicating the state format version; entityId: integer, indicating which simulation entity; position: 3 decimals, indicating the position [x,y,z]; orientation: 4 decimals, indicating the orientation quaternion [qx,qy,qz,qw]; jointId: integer, indicating the joint number; jointPosition: decimal, indicating the joint angle.

[0085] During the simulation, the server can push the following status field data to the client: frame=120; timestamp=2.0; schemaHash=8888; entityId=1; position=[0.1,0.0,0.8]; orientation=[0,0,0,1]; jointId=0,jointPosition=0.32; jointId=1,jointPosition=-0.15.

[0086] The server sends real-time status data according to the format defined in the state structure description. Only when the client receives a matching state structure description can it correctly parse the robot's position, orientation, and joint angles.

[0087] Therefore, the server can include the state structure description and binding manifest file together in the release package.

[0088] The scene release package may also include the first resource identifier information corresponding to the visual assets.

[0089] The verification checklist includes consistency verification information, which may include first state identifier information and / or first resource identifier information corresponding to the visual asset. Of course, consistency verification information may also include other information such as signature, version number, and hash values ​​of multiple files. The version number is... Figure 4 The Version shown is used for release packages or scenario versions, facilitating version management and traceability.

[0090] Signature (e.g.) Figure 4 The digital signature shown may be generated based on the first state identifier information and / or the first resource identifier information corresponding to the visual asset. Multiple file hash values ​​may include hash values ​​corresponding to the visual asset, the binding manifest file, and the state structure description, respectively.

[0091] The first state identifier is a unique identifier that corresponds to the state structure description. It can be calculated based on the state structure description, for example, it can be a schemaHash obtained by performing a hash operation on the state structure description.

[0092] The first resource identifier is used to identify the version of the resource collection.

[0093] In one embodiment of this application, the scene release package further includes a binding manifest file, which includes binding mapping information, including the mapping relationship between simulation entity identifiers and client visual node identifiers; the first resource identifier information corresponding to the visual asset is generated based on the hash value of the visual asset and the hash value of the binding manifest file.

[0094] The first resource identifier information, i.e., resourceHash, corresponding to the visual asset can be obtained by aggregating the hash value of the visual asset and the hash value of the binding manifest file (e.g., concatenating the two and then performing a hash operation). The first resource identifier information can serve as a prerequisite for cache reuse.

[0095] After obtaining the scene release package, the client can store the verification list or directly cache the entire scene release package. The client can also verify each file based on the hash value of each file in the verification list.

[0096] In one embodiment of this application, before obtaining a status frame including second status identification information from the first server or a second server different from the first server, the method further includes: reporting rendering capability information to the first server or a second server different from the first server, so that the first server or the second server can distribute rendering configuration or select resources based on the rendering capability information.

[0097] Rendering capability information may include whether WebGPU is supported, whether HDR can be implemented, and whether floating-point rendering targets are available. Browsers can use either WebGPU or WebGL.

[0098] Regarding resource selection, when the client is an HDR device, an HDR texture or an ambient light texture can be sent; when the client is not an HDR device, an SDR version can be sent.

[0099] The server can issue rendering configurations such as tone mapping parameters and post-processing switches based on rendering capability information.

[0100] When the scenario release package does not include the first resource identifier information, obtaining the scenario release package from the first server may include: obtaining the third state identifier information corresponding to the state structure description of the current version of the scenario release package used by the first server; if the third state identifier information is inconsistent with the state identifier information corresponding to the state structure description stored locally, then the scenario release package is obtained from the first server.

[0101] The current version of the scenario release package used by the first server can be the latest version of the scenario release package. Alternatively, the current version of the scenario release package used by the first server can be the version configured on the first server, that is, the scenario release package currently used by the first server.

[0102] The state identifier information corresponding to the state structure description in local storage is the same as the state identifier information corresponding to the state structure description in the scene release package currently loaded on the client.

[0103] In one embodiment of this application, obtaining the scene release package from the first server may further include: obtaining second resource identification information corresponding to the visual assets of the current version of the scene release package used by the first server; obtaining third state identification information corresponding to the state structure description of the current version of the scene release package used by the first server; if the third state identification information is inconsistent with the state identification information corresponding to the state structure description stored locally, or if the second resource identification information is inconsistent with the resource identification information corresponding to the visual assets stored locally, then the scene release package is obtained from the first server.

[0104] The resource identification information corresponding to the visual assets stored locally is the same as the resource identification information corresponding to the visual assets in the scene release package that has already been loaded on the client.

[0105] If the third state identifier information is consistent with the state identifier information corresponding to the state structure description stored locally, or if the third state identifier information is consistent with the state identifier information corresponding to the state structure description stored locally, and the second resource identifier information is consistent with the resource identifier information corresponding to the visual asset stored locally, then the scene release package can be reused directly without obtaining it from the server, thereby realizing the reuse of cached data.

[0106] Figure 5 A timing diagram illustrating a client retrieving a scene publishing package from a server according to an embodiment of this application is shown. See also... Figure 5 As shown, the specific process may include the following: First, the client sends an initial handshake message to the server: ClientHello(known hashes, supports, maxFrameBytes), which tells the server "what cached versions I have locally, what transport capabilities I support, and how large a status frame I can handle".

[0107] Here, known hashes are the verification information of the published packages that have been cached locally on the client, such as schemaHash, resourceHash, version, manifestHash, glbHash, etc. The server uses known hashes to determine whether the cached published packages can still be used.

[0108] Supports is a set of client capabilities, such as whether it supports binary-v1 binary state frames, json-delta-v1 incremental JSON state frames, compression, incremental synchronization, WebGPU, HDR rendering, etc.

[0109] maxFrameBytes is the maximum number of bytes in a single status frame that the client can accept. The server can use this to choose the encoding, compression method, or reduce the load per frame.

[0110] Then, the server returns a handshake confirmation message to the client: ServerHello(currenthashes, encoding, compress, keyframeInterval), which tells the client "what is the current authoritative version of the server, and what transmission parameters will be used for the subsequent state stream".

[0111] Here, `current hashes` refers to the consistency verification information of the currently published package on the server. This includes, for example, the current schemaHash, resourceHash, and version. It is compared with the known hashes reported by the client.

[0112] Encoding refers to the encoding method for subsequent state frames. For example, binary-v1 or json-delta-v1.

[0113] The `compress` option indicates whether compression is enabled and the compression method. Examples include `none`, `gzip`, and `br`.

[0114] keyframeInterval is the keyframe transmission interval. For example, a complete keyframe is sent every 60 frames, with incremental frames sent in between to reduce bandwidth and support recovery from disconnection.

[0115] Then, when schemaHash, resourceHash, etc. fail the verification, the server sends an update command to the client: PackageUpdate(reason, current, download), which notifies the client that "the locally cached release package is no longer compatible and needs to be downloaded or updated again."

[0116] Here, "reason" indicates the reason for the update. For example, "schemaHashMismatch" indicates an inconsistent state structure, "resourceHashMismatch" indicates an inconsistent visual resource or Manifest, and "versionTooOld" indicates that the client version is too old.

[0117] Current contains information about the current correct version on the server. Current can include the current sceneId, version, schemaHash, resourceHash, etc.

[0118] Download information is the download URL. Download can include the full release package address, manifest address, GLB address, checksum file address, or differential package address. Examples include packageUrl, manifestUrl, glbUrl, checksumUrl, and deltaUrl.

[0119] Next, the client can download the complete release package from object storage or CDN, or update the existing partial scenario release package.

[0120] Then, object storage or CDN can return the entire package file to the client, or it can return a package difference file. For example, when the third state identifier information is inconsistent with the state identifier information corresponding to the state structure description in local storage, but the third resource identifier information is consistent with the resource identifier information corresponding to the visual asset in local storage, only the state structure description and checklist file in the release package can be downloaded; when the third resource identifier information is inconsistent with the resource identifier information corresponding to the visual asset in local storage, but the third state identifier information is consistent with the state identifier information corresponding to the state structure description in local storage, only other files besides the state structure description in the release package can be downloaded.

[0121] Next, after the client completes the download or update, it sends a confirmation handshake message to the server again: ClientHello (updated known hashes). Its function is to tell the server, "I have updated to the new release package, and these are the local hashes now. Please confirm again."

[0122] Among them, the updated known hashes are the consistency verification information in the release package after the client re-downloads and verifies them.

[0123] Subsequently, the server sends a ServerHello (consistency confirmation) to the client: when the server confirms that the consistency check information it is currently using is consistent with the latest consistency check information sent by the client, it will return an confirmation message to the client.

[0124] Next, we can proceed to state stream synchronization, where the server continuously sends state frames to the client in a state stream manner.

[0125] After the client obtains the release package, it will load the release package. When loading the release package, it will parse the binding manifest file, establish the mapping relationship between entities and visual nodes, and record schemaHash, resourceHash, and version.

[0126] In step 120, a status frame including second status identification information is obtained from the first server or a second server different from the first server.

[0127] A state frame may include only state field data, or no fields; the order of fields defined in the StateSchema can be used to determine which state field data belongs to which field.

[0128] After establishing the mapping relationship between entities and visual nodes, the client can start obtaining status frames from the server.

[0129] The client can obtain status frames from the first server or from other servers (such as the second server).

[0130] The simulation officially begins when the server sends a status frame to the client for the first time. Steps 120 and 130 can be repeated to allow the client to continuously receive and apply status frames from the server, thereby rendering the simulation effect.

[0131] A state frame is a data frame generated by the server and sent to the client during simulation. It contains a frame sequence number or timestamp, consistency verification information, and state data of one or more entities.

[0132] Status frames can be divided into key frames and incremental frames.

[0133] In step 130, if the state identifier information passes the verification based on the second state identifier information obtained from the state frame and the first state identifier information, then the state frame is applied to display the robot simulation effect based on the state frame. The verification of the state identifier information is used to indicate whether the version of the state structure description identified by the first state identifier information matches the version of the state structure description corresponding to the state frame.

[0134] The second state identifier information can be located in the header of the state frame. The state frame header must contain at least: frameIndex, timestamp, schemaHash, and isKeyframe. isKeyframe is used to indicate whether it is a keyframe.

[0135] Whenever a client receives a status frame, it can verify the status identifier information based on the second status identifier information in its frame header.

[0136] When the status identification information passes the verification, it means that the version of the status structure description identified by the first status identification information matches the version of the status structure description corresponding to the status frame; when the status identification information fails the verification, it means that the version of the status structure description identified by the first status identification information does not match the version of the status structure description corresponding to the status frame.

[0137] It can be determined whether the second state identifier information obtained from the state frame is consistent with the first state identifier information. If they are consistent, the state identifier information passes the verification; otherwise, the state identifier information fails the verification. That is, the state frame is only allowed to be parsed and applied if the second state identifier information is consistent with the first state identifier information.

[0138] Of course, it is also possible to determine whether the second state identifier information obtained from the state frame matches the first state identifier information. If they match, the verification passes; otherwise, the state identifier information fails the verification. For example, the second state identifier information and the first state identifier information can be two numerical values. A preset calculation rule (such as addition) can be used to perform a calculation based on the second state identifier information and the first state identifier information to obtain the calculation result. Then, it is determined whether the calculation result is consistent with the predetermined calculation result. If they are consistent, the state identifier information passes the verification; otherwise, the state identifier information fails the verification.

[0139] The client can render robot simulation effects through the HDR high-fidelity pipeline. The HDR pipeline can use different tone mapping algorithms, that is, use HDR rendering targets and linear color space processing, perform exposure control and tone mapping output to improve the consistency of highlight details and shadow levels, and achieve high-fidelity reproduction.

[0140] The client can also write incremental state to the GPU buffer to update a large number of entity transformation matrices or instantiation parameters in a GPU batch manner, thereby reducing CPU overhead and achieving batch updates. Employing a WebGPU-based rendering pipeline and GPU batch update mechanism can improve rendering efficiency.

[0141] In one embodiment of this application, after obtaining a status frame including second status identification information from the first server or a second server different from the first server, the method further includes: if it is determined that the status identification information has failed verification based on the second status identification information obtained from the status frame and the first status identification information, then the application of the status frame is rejected, and a target recovery strategy is triggered to restore the robot simulation process.

[0142] In one embodiment of this application, the target recovery strategy includes at least one of the following: re-obtaining the scenario release package currently used by the first server from the first server, re-obtaining the state structure description and binding manifest file currently used by the first server from the first server, requesting the keyframe at the current recovery time, clearing the cached scenario release package corresponding to the current session, and loading the scenario release package currently used by the first server re-obtained from the first server.

[0143] The scenario release package currently used by the first server is the current version of the scenario release package used by the first server; the state structure description and binding manifest file currently used by the first server is the state structure description and binding manifest file in the current version of the scenario release package used by the first server.

[0144] Figure 6A schematic diagram illustrating a processing flow for verifying status identification information according to an embodiment of this application is shown. Please refer to... Figure 6 As shown, after the client receives the status frame, it can perform a consistency check based on schemaHash. If the consistency check passes, the status frame can be parsed and applied to render and interactively update the status frame. If the consistency check fails, the application of the frame is rejected, and the corresponding diagnostic information is recorded. The client can also choose from three recovery strategies: lightweight update, full update, and rebuild binding. Then, consistency recovery can be performed based on the selected recovery strategy.

[0145] The aforementioned consistency check provides a gate mechanism that prevents incompatible state frames from entering the rendering and interaction chain, thereby reducing the risk of misbinding, display anomalies, and abnormal program termination, and improving fault location and traceability capabilities.

[0146] In one embodiment of this application, obtaining a status frame including second status identification information from the first server or a second server different from the first server includes: obtaining a key frame including complete status field data from the first server or a second server different from the first server; or, obtaining an incremental frame including only status field data that has changed relative to the previously obtained frame or reference frame from the first server or a second server different from the first server.

[0147] A keyframe is a state frame that contains data corresponding to all the complete state fields of an entity, or data corresponding to the minimum set of fields that, after lossy or lossless compression, are still sufficient to independently reconstruct the complete visualization state. After receiving a keyframe, the client can establish a complete visualization baseline for the current simulation entity without relying on any historical state frames.

[0148] The server can first determine whether a keyframe or an incremental frame needs to be generated, and then use the complete status field data of the current frame to generate the keyframe or incremental frame.

[0149] The server can send a key frame every predetermined period (e.g., 30 frames), and the remaining frames are incremental frames.

[0150] The reference frame can be any frame that has already been acquired; it can be a key frame or an incremental frame.

[0151] Since incremental frames only include status field data corresponding to fields that have changed, bandwidth and parsing overhead can be reduced.

[0152] Consistency verification can be performed based on the client's initial connection to the server, loading of the deployment package, and recovery after disconnection and reconnection.

[0153] When a client first connects to the server or when the client and server reconnect after a disconnection, the server can send a keyframe to the client. This allows for reconnection recovery in cases of frame loss, out-of-order frames, or the inability to find a reference frame.

[0154] When the client and server connection is interrupted and reconnected, the client first obtains the current consistency verification information from the server. If they are consistent, the client continues to connect; if they are inconsistent, the client triggers the re-retrieval of the release package. After the server recovers, it sends a key frame. The client uses the key frame to reconstruct the baseline state and then continues to receive incremental frames.

[0155] Please continue reading Figure 6 As shown, after the consistency recovery is complete, a key frame can be requested first, then the key frame can be applied to rebuild the baseline state, and then incremental frames can continue to be received.

[0156] In one embodiment of this application, the state structure description includes a state field order; the increment frame includes a binary sequence marker, which includes a plurality of binary bits arranged in sequence, each binary bit corresponding to a state field at a corresponding position in the state field order, and the value of each binary bit representing whether the data of the corresponding state field has changed; the increment frame includes state field data whose corresponding state field has changed.

[0157] The number of bits in the binary sequence marker is the same as the total number of fields in the StateSchema. The binary sequence marker maps bits according to the order in which the fields are declared in the StateSchema. A 1 in the nth bit (starting from 0) in the binary sequence marker indicates that the nth field in the frame has changed relative to the base frame, while a 0 indicates that it has not changed. The incremental state frame only carries the state field data with the corresponding bit set to 1. Unchanged fields are not transmitted and are completed by the client based on the cached base state. The client can determine which field the received state field data corresponds to based on the binary sequence marker.

[0158] Figure 7 A timing diagram illustrating the synchronization of keyframes and incremental frames between a server and a client according to an embodiment of this application is shown. See also... Figure 7As shown, when a client first connects to the server, it requests keyframes. After performing schemaHash on the keyframes, the client parses the status field data to establish a baseline state. Then, the server frequently synchronizes incremental frames to the client. After verifying the schemaHash of each incremental frame, the server also applies the incremental frame. In case of abnormal situations such as frame loss, out-of-order delivery, or failure to obtain the baseline frame, the client can send a request to the server containing the cause and the frame index of the most recently applied status frame to re-request the keyframes. The server can return the corresponding keyframe to the client based on the frame index of the most recently applied status frame to re-establish the baseline state on the client and then continue receiving incremental frames.

[0159] In one embodiment of this application, the state frame is generated by the server based on the simulation state code generated after the simulation engine executes control logic and advances the simulation; the method further includes: simulating the same scene asset using the same control logic based on at least two simulation engines, and obtaining comparison index data corresponding to each simulation engine, wherein the comparison index data is used to measure whether the running results of at least two simulation engines are consistent; determining the comparison result of the running results of the simulation engines and the reason information for the inconsistency of the running results of different simulation engines based on the comparison index data corresponding to each simulation engine, and displaying the comparison result and the reason information.

[0160] The same scene asset can be simulated using two or more simulation engines, each employing the same control logic.

[0161] Control logic can be a strategy model, i.e., a control decision model. Its role is to output the next action or control command based on the current observed state, such as robot joint angles, end-effector pose, sensor data, and environmental conditions. Examples of control commands include target joint position, velocity, torque, and end-effector motion commands. Scene assets can include visual assets, as well as collision models, sensors, and physical parameters.

[0162] The embodiment of this application imports the training platform product into the cloud simulation system and performs a Sim-to-Sim comparison. The purpose is to conduct a reproducible comparative evaluation of the same asset and the same strategy on different simulation engines or verification platforms.

[0163] First, the training platform can be exported: export assets (such as scenes, robots, joint structures, sensors, etc.) and policy-related products expressed in USD (Universal Scene Description) format; the policy-related products are used for policy reasoning and result reproduction, including but not limited to: policy model files (such as neural network weight files and their structures, inference graph descriptions), policy execution programs or inference logic files, observation or action normalization and scaling parameter tables, control frequency and controller parameter configurations; when it is necessary to reproduce experiments or conduct comparative evaluations, reward function configuration, environment randomization configuration and random seed settings can also be included.

[0164] Then, server-side import and standardization can be performed: the server parses USD, completes the standardization of units, axes, joint order, etc., generates an internally unified scene representation, and produces visual assets, Manifest, and StateSchema for the browser. The so-called standardization means that USD scene files from different sources may have different axes, units, and joint orders. The server needs to convert them into the platform's internally unified standard format before continuing to generate the release package, Manifest, and StateSchema.

[0165] Next, the same scenario and the same strategy can be run in at least two simulation engines, and the output will uniformly follow the same StateSchema to form a StateFrame. Next, a comparative evaluation can be performed, namely, aligning and replaying the metrics in the browser and comparing the metrics; when the metric data differs, the source of the difference can be traced by combining the version information related to the constraints and the strategy (such as asset normalization, control frequency, joint mapping, normalization parameters, etc.), and the source of the difference can be used as the cause information.

[0166] Comparison metrics, in a Sim2Sim scenario, refer to a set of indicators used to measure whether the results of two or more simulation engines are consistent. Examples may include: Robot state metrics: such as joint angles, joint velocities, joint torques, root node pose, link pose, end effector pose, etc.; trajectory metrics: such as robot end effector trajectory, chassis trajectory, object motion trajectory, key point trajectory, etc.; contact or interaction metrics: such as contact point, contact force, collision status, whether the object was successfully grasped, whether it slipped, etc.; sensor or observation metrics: such as camera images, depth maps, force sensors, IMU, joint encoders, policy input observation, etc.; policy and task metrics: such as policy output action, reward value, task success rate, completion time, number of collisions, path length, energy consumption, etc.; time synchronization metrics: such as frame index, timestamp, control cycle, playback alignment error, etc.

[0167] The following describes an embodiment of the apparatus described in this application, which can be used to execute the robot simulation method described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the robot simulation method described above.

[0168] Figure 8 A block diagram of a robot simulation apparatus according to an embodiment of this application is shown. (Refer to...) Figure 8 As shown, a robot simulation device 800 according to an embodiment of this application includes: a state identifier acquisition and storage unit 810, a state frame acquisition unit 820, and a state frame application unit 830. The state identifier acquisition and storage unit 810 is used to acquire and store first state identifier information corresponding to a state structure description from a first server; the state structure description is used to define the data format of the state frame, and the first state identifier information is used to uniquely identify the version of the state structure description. The state frame acquisition unit 820 is used to acquire a state frame including second state identifier information from the first server or a second server different from the first server. The state frame application unit 830 is used to apply the state frame to display the robot simulation effect based on the state frame if the state identifier information passes the verification based on the second state identifier information acquired from the state frame and the first state identifier information. The verification of the state identifier information is used to indicate whether the version of the state structure description identified by the first state identifier information matches the version of the state structure description corresponding to the state frame.

[0169] In some embodiments of this application, based on the aforementioned scheme, the state identifier acquisition and storage unit 810 is configured to: acquire a scene release package from the first server and store the first state identifier information in the scene release package, wherein the scene release package includes visual assets, the state structure description and the first state identifier information.

[0170] In some embodiments of this application, based on the foregoing scheme, the status identifier acquisition and storage unit 810 is configured to: acquire third status identifier information corresponding to the status structure description of the current version of the scenario release package used by the first server from the first server; if the third status identifier information is inconsistent with the status identifier information corresponding to the status structure description stored locally, then acquire the scenario release package from the first server; or, The scene release package also includes first resource identifier information corresponding to the visual asset; the status identifier acquisition and storage unit 810 is configured to: acquire second resource identifier information corresponding to the visual asset of the current version of the scene release package used by the first server from the first server; acquire third status identifier information corresponding to the status structure description of the current version of the scene release package used by the first server from the first server; if the third status identifier information is inconsistent with the status identifier information corresponding to the locally stored status structure description, or if the second resource identifier information is inconsistent with the resource identifier information corresponding to the locally stored visual asset, then acquire the scene release package from the first server.

[0171] In some embodiments of this application, based on the foregoing scheme, the scene release package further includes a binding manifest file, the binding manifest file includes binding mapping information, the binding mapping information includes the mapping relationship between simulation entity identifiers and client visual node identifiers; the first resource identifier information corresponding to the visual asset is generated based on the hash value of the visual asset and the hash value of the binding manifest file.

[0172] In some embodiments of this application, based on the foregoing scheme, the device is applied to a client; the status frame acquisition unit 820 is configured to: acquire a key frame including complete status field data from the first server or a second server different from the first server; or, acquire an incremental frame including only status field data that has changed relative to the previously acquired frame or reference frame from the first server or a second server different from the first server.

[0173] In some embodiments of this application, based on the foregoing scheme, the device further includes a rejection and recovery unit; after obtaining a status frame including second status identification information from the first server or a second server different from the first server, the rejection and recovery unit is configured to: if it is determined that the status identification information has failed verification based on the second status identification information obtained from the status frame and the first status identification information, then reject the application of the status frame and trigger a target recovery strategy to restore the robot simulation process.

[0174] In some embodiments of this application, based on the foregoing scheme, the target recovery strategy includes at least one of the following: re-obtaining the scenario release package currently used by the first server from the first server, re-obtaining the state structure description and binding manifest file currently used by the first server from the first server, requesting to obtain the keyframe at the current recovery time, clearing the cached scenario release package corresponding to the current session and loading the scenario release package currently used by the first server re-obtained from the first server.

[0175] In some embodiments of this application, based on the foregoing scheme, the state structure description includes the state field order; the increment frame includes a binary sequence marker, the binary sequence marker includes multiple binary bits arranged in order, each binary bit corresponds to a state field at a corresponding position in the state field order, and the value of each binary bit represents whether the data of the corresponding state field has changed; the increment frame includes state field data where the corresponding state field has changed.

[0176] In some embodiments of this application, based on the aforementioned scheme, the state frame is generated by the server according to the simulation state code generated after the simulation engine executes control logic and advances the simulation; the device further includes a comparison and evaluation unit; the comparison and evaluation unit is used to: simulate the same scene asset using the same control logic based on at least two simulation engines, and obtain comparison index data corresponding to each simulation engine, the comparison index data being used to measure whether the running results of at least two simulation engines are consistent; determine the comparison result of the running results of the simulation engines and the reason information for the inconsistency of the running results of different simulation engines based on the comparison index data corresponding to each simulation engine, and display the comparison result and the reason information.

[0177] In some embodiments of this application, based on the foregoing scheme, the apparatus further includes a rendering capability reporting unit; before obtaining a status frame including second status identification information from the first server or a second server different from the first server, the rendering capability reporting unit is used to: report rendering capability information to the first server or a second server different from the first server, so that the first server or the second server can distribute rendering configuration or select resources based on the rendering capability information.

[0178] Figure 9 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0179] It should be noted that, Figure 9 The computer system 900 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0180] like Figure 9 As shown, the computer system 900 includes a CPU 901, which can perform various appropriate actions and processes according to a program stored in ROM 902 or a program loaded into RAM 903 from storage section 908, such as performing the methods described in the above embodiments. The RAM 903 also stores various programs and data required for system operation. The CPU 901, ROM 902, and RAM 903 are interconnected via bus 904. An I / O interface 905 is also connected to bus 904.

[0181] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. Removable media 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 910 as needed so that computer programs read from them can be installed into storage section 908 as needed.

[0182] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by CPU 901, it performs various functions defined in the system of this application.

[0183] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0184] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0185] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0186] In one aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0187] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0188] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0189] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0190] The data collection and processing plan outlined in this application must be implemented in strict accordance with the requirements of relevant national laws and regulations, obtaining the informed consent or separate consent of the data subject (or having a legal basis as stipulated by the relevant national laws and regulations), and conducting subsequent data use and processing within the scope authorized by laws and regulations and the data subject.

[0191] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A robot simulation method, characterized in that, The method includes: The first state identifier information corresponding to the state structure description is obtained from the first server and stored; the state structure description is used to define the data format of the state frame, and the first state identifier information is used to uniquely identify the version of the state structure description. Obtain a status frame including the second status identifier information from the first server or a second server different from the first server; If the state identifier information passes the verification based on the second state identifier information obtained from the state frame and the first state identifier information, then the state frame is applied to display the robot simulation effect based on the state frame. The verification of the state identifier information is used to indicate whether the version of the state structure description identified by the first state identifier information matches the version of the state structure description corresponding to the state frame.

2. The robot simulation method according to claim 1, characterized in that, The step of obtaining the first state identifier information corresponding to the state structure description from the first server and storing the first state identifier information includes: The scene release package is obtained from the first server, and the first state identification information in the scene release package is stored. The scene release package includes visual assets, the state structure description and the first state identification information.

3. The robot simulation method according to claim 2, characterized in that, The step of obtaining the scenario release package from the first server includes: Obtain the third state identifier information corresponding to the state structure description of the current version of the scenario release package used by the first server from the first server; If the third state identifier information is inconsistent with the state identifier information corresponding to the state structure description stored locally, then the scene release package is obtained from the first server. or, The scene release package also includes first resource identification information corresponding to the visual asset; obtaining the scene release package from the first server includes: Obtain the second resource identifier information corresponding to the visual assets of the current version of the scene release package used by the first server; Obtain the third state identifier information corresponding to the state structure description of the current version of the scenario release package used by the first server from the first server; If the third state identifier information is inconsistent with the state identifier information corresponding to the state structure description stored locally, or if the second resource identifier information is inconsistent with the resource identifier information corresponding to the visual assets stored locally, then the scene release package is obtained from the first server.

4. The robot simulation method according to claim 3, characterized in that, The scenario release package also includes a binding manifest file, which includes binding mapping information, including the mapping relationship between simulation entity identifiers and client visual node identifiers; the first resource identifier information corresponding to the visual asset is generated based on the hash value of the visual asset and the hash value of the binding manifest file.

5. The robot simulation method according to claim 3, characterized in that, The method is executed by the client; obtaining the status frame including the second status identifier information from the first server or a second server different from the first server includes: Obtain a keyframe including complete status field data from the first server or a second server different from the first server; or Obtain incremental frames from the first server or a second server different from the first server, which only include state field data that has changed relative to the previously obtained frame or reference frame.

6. The robot simulation method according to claim 5, characterized in that, After obtaining a status frame including second status identification information from the first server or a second server different from the first server, the method further includes: If it is determined that the status identifier information has failed verification based on the second status identifier information obtained from the status frame and the first status identifier information, then the status frame is rejected and the target recovery strategy is triggered to restore the robot simulation process.

7. The robot simulation method according to claim 6, characterized in that, The target recovery strategy includes at least one of the following: re-obtaining the scenario release package currently used by the first server from the first server, re-obtaining the state structure description and binding manifest file currently used by the first server from the first server, requesting the keyframe at the current recovery time, clearing the cached scenario release package corresponding to the current session and loading the scenario release package currently used by the first server re-obtained from the first server.

8. The robot simulation method according to claim 5, characterized in that, The state structure description includes the state field order; the increment frame includes a binary sequence marker, which includes multiple binary bits arranged in sequence, each binary bit corresponding to a state field at a corresponding position in the state field order, and the value of each binary bit represents whether the data of the corresponding state field has changed; the increment frame includes the state field data of the state field where the corresponding state field has changed.

9. The robot simulation method according to claim 1, characterized in that, The state frame is generated by the server based on the simulation state encoding produced by the simulation engine after executing control logic and advancing the simulation; the method further includes: Simulation of the same scene asset is performed using at least two simulation engines with the same control logic, and comparison index data corresponding to each simulation engine is obtained. The comparison index data is used to measure whether the running results of at least two simulation engines are consistent. Based on the comparative index data corresponding to each simulation engine, the comparison results of the simulation engine's running results and the reasons for the inconsistency in the running results of different simulation engines are determined, and the comparison results and the reasons are displayed.

10. The robot simulation method according to claim 1, characterized in that, Before obtaining a status frame including the second status identifier information from the first server or a second server different from the first server, the method further includes: The rendering capability information is reported to the first server or a second server different from the first server, so that the first server or the second server can distribute rendering configurations or select resources based on the rendering capability information.

11. A robot simulation device, characterized in that, The device includes: A status identifier acquisition and storage unit is used to acquire first status identifier information corresponding to a status structure description from a first server and store the first status identifier information; the status structure description is used to define the data format of a status frame, and the first status identifier information is used to uniquely identify the version of the status structure description. A status frame acquisition unit is used to acquire a status frame including second status identification information from the first server or a second server different from the first server. A state frame application unit is configured to apply the state frame to display the robot simulation effect based on the state frame if the state identifier information passes the verification based on the second state identifier information obtained from the state frame and the first state identifier information. The verification of the state identifier information is used to indicate whether the version of the state structure description identified by the first state identifier information matches the version of the state structure description corresponding to the state frame.

Images