Simulation verification method and device of body robot, computer equipment and medium

Abstract

The application relates to the technical field of embodied robot training, and discloses a simulation verification method and device for an embodied robot, computer equipment and a medium, the method comprising the following steps: loading two WebAssembly modules generated by compiling a physical simulation engine and an inference engine in a browser; preallocating a memory pool for an inference module after acquiring strategy network data, realizing intermediate tensor memory multiplexing based on computation graph data dependency analysis implementation, and exposing a buffer address of the inference module to a physical module. In a simulation cycle, the physical module directly writes state data into an input buffer, the inference module reads the state and executes forward inference by using a multiplexed memory segment and weights, and writes actions into an output buffer for the physical module to read. The application realizes real-time simulation with zero installation on the browser side, can eliminate delay jitter caused by dynamic allocation through static memory management, helps to ensure the determinacy of inference time consumption, and realizes zero-copy data exchange between modules through address exposure.

Description

Technical Field

[0001] This application relates to the field of embodied robot training, and more particularly to a simulation verification method, apparatus, computer equipment, and medium for embodied robots. Background Technology

[0002] Strategy simulation verification for embodied robots typically requires installing physical simulation software and inference frameworks on a local computer, which is cumbersome to configure and difficult to reproduce across platforms. Some solutions attempt to run physical simulations in a browser, but the strategy inference stage still relies on a remote server or local inference environment, failing to achieve a fully browser-based closed-loop simulation.

[0003] Another approach runs both physics simulation and policy inference within the browser. However, this approach uses a general neural network inference framework, which involves dynamic memory management. Under millisecond-level real-time simulation step sizes, this can easily introduce fluctuations in inference time, affecting the determinism of the simulation loop. Approaches relying on edge nodes to interact with the browser require external computing resources and are affected by network latency.

[0004] Therefore, how to achieve high-performance closed-loop operation of physical simulation and policy reasoning within a browser without relying on local software installation and external computing resources, while ensuring the consistency of simulation step size and the determinism of reasoning time, is an urgent problem to be solved. Summary of the Invention

[0005] Therefore, it is necessary to address the technical problem of inconsistent simulation cycle step size in existing embodied robot simulation technologies.

[0006] Firstly, a simulation verification method for an embodied robot is provided, the method comprising: Load the first WASM module and the second WASM module in the browser. The first WASM module is compiled and generated by the robot physics simulation engine, and the second WASM module is compiled and generated by the neural network inference engine. Obtain the structural description data and weight parameters of the robot's reinforcement learning policy network; Based on the structure description data, an input buffer and an output buffer are allocated to the second WASM module, the addresses of the input buffer and the output buffer are exposed to the first WASM module, a memory pool is pre-allocated for the second WASM module, and the weight parameters are written into a designated memory area of ​​the second WASM module. Data dependency analysis is performed on the computation graph of the robot reinforcement learning policy network to determine the lifecycle of each intermediate tensor, and control multiple intermediate tensors with non-overlapping lifecycles to reuse the same memory segment in the memory pool. The simulation loop is executed, and in each step: the first WASM module drives the physical simulation of the robot model based on the action data of the previous step, generates simulation state data, and writes the simulation state data into the input buffer; the second WASM module reads the simulation state data from the input buffer as input, uses the reused memory segment in the memory pool and the weight parameters in the specified memory region to perform forward inference of the robot reinforcement learning policy network, generates the current step action data, and writes the current step action data into the output buffer; the first WASM module reads the current step action data from the output buffer as the action input for the next simulation step.

[0007] Secondly, a simulation verification device for an embodied robot is provided, the device comprising: A loading module is used to load a first WASM module and a second WASM module in the browser. The first WASM module is compiled and generated by the robot physics simulation engine, and the second WASM module is compiled and generated by the neural network inference engine. The acquisition module is used to acquire the structural description data and weight parameters of the robot's reinforcement learning policy network; The allocation module is used to allocate an input buffer and an output buffer to the second WASM module according to the structure description data, expose the addresses of the input buffer and the output buffer to the first WASM module, pre-allocate a memory pool for the second WASM module, and write the weight parameters into a specified memory area of ​​the second WASM module. The reuse module is used to perform data dependency analysis on the computation graph of the robot reinforcement learning policy network, determine the life cycle of each intermediate tensor, and control multiple intermediate tensors with non-overlapping life cycles to reuse the same memory segment in the memory pool. The simulation module is used to execute a simulation loop. In each step: the first WASM module drives the physical simulation of the robot model based on the action data of the previous step, generates simulation state data, and writes the simulation state data into the input buffer; the second WASM module reads the simulation state data from the input buffer as input, uses the reused memory segment in the memory pool and the weight parameters in the specified memory region to perform forward inference of the robot reinforcement learning policy network, generates the current step action data, and writes the current step action data into the output buffer; the first WASM module reads the current step action data from the output buffer as the action input for the next simulation step.

[0008] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described simulation verification method for an embodied robot.

[0009] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described simulation verification method for an embodied robot.

[0010] The beneficial effects of this application are: This application compiles the robot physics simulation engine and the neural network inference engine into a first WASM module and a second WASM module, respectively, and loads and runs them in a browser. This allows users to complete the entire process of policy simulation verification without installing any physics simulation software, inference framework, or related dependency libraries on their local computer, eliminating the problems of cumbersome environment configuration and difficulty in cross-platform reproduction in traditional solutions.

[0011] In the initialization phase of the second WASM module, this application pre-allocates a fixed-size memory pool based on the structural description data of the robot reinforcement learning policy network, and performs data dependency analysis on the computation graph to determine the lifecycle of each intermediate tensor. Intermediate tensors with non-overlapping lifecycles reuse the same memory segment in the memory pool. This technique ensures that every forward inference in the entire simulation loop is completed within the pre-allocated memory pool, without triggering any dynamic memory allocation operations. This effectively avoids fluctuations in inference time caused by dynamic memory management, ensuring the consistency of the simulation loop step size and the determinism of inference time. It also helps solve the step size jitter problem caused by dynamic memory operations in real-time simulation scenarios for general inference frameworks.

[0012] This application allocates input and output buffers in the second WASM module and exposes their addresses to the first WASM module. This allows the simulation state data generated by the first WASM module in each simulation step to be directly written to the input buffer. The second WASM module can directly read data from the input buffer as input tensors to perform forward inference and directly write the generated current step action data to the output buffer for the first WASM module to read. This address-sharing-based data exchange mechanism eliminates the overhead of data copying and format conversion between the physics simulation module and the policy inference module, improves the data exchange efficiency of each simulation step, and provides a technical foundation for achieving high-performance closed-loop operation of physics simulation and policy inference within a browser. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] in: Figure 1 This is a schematic diagram of an application scenario provided in one embodiment; Figure 2 This is a flowchart of a simulation verification method for an embodied robot in one embodiment; Figure 3 This is a structural block diagram of a simulation verification device for an embodied robot in one embodiment; Figure 4 This is a schematic diagram of a computer storage medium in one embodiment; Figure 5 This is a structural block diagram of a computer device in one embodiment. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] The scenario diagram provided in this application embodiment shows that the embodied robot training system includes: a computer device 11 and a server 12. The two devices are interconnected via a network, and the communication connection can be wireless or wired, together forming the hardware environment for executing the embodied robot training system.

[0017] Computer device 11 is a user-side terminal device running a browser. Server 12 is a remotely deployed data storage and computing node, and the two communicate via a network connection.

[0018] Before the simulation verification begins, computer device 11 sends a resource request to server 12 via a browser. In response to this request, server 12 transmits the first WASM module file (compiled using Emscripten for the robot physics simulation engine), the second WASM module file (compiled using Emscripten for the neural network inference engine), and the structural description data and weight parameter files of the robot reinforcement learning policy network to be verified to computer device 11 via the network. The processor of computer device 11 loads the received first and second WASM modules into the browser, obtains the structural description data and weight parameters, and then performs initialization operations such as memory pool pre-allocation, buffer address exposure, computation graph data dependency analysis, and tensor memory reuse configuration based on this data.

[0019] During the simulation loop, when the user needs to switch to verify another robot reinforcement learning policy network, computer device 11 sends a policy switching request to server 12. In response to this request, server 12 sends the structural description data and weight parameters corresponding to the policy network to be switched to computer device 11. Upon receiving this data, the processor of computer device 11, without interrupting the simulation loop, writes the new weight parameters to a designated memory area to replace the original weight parameters, re-executes data dependency analysis on the computation graph of the new policy network, updates the memory segment reuse relationships in the memory pool, and completes the online switching of the policy network.

[0020] During the simulation loop execution, the processor of computer device 11 independently completes each step of physical simulation calculation, strategy reasoning calculation and rendering display within the browser. The entire process does not rely on the computing resources of server 12. Server 12 only provides module file and strategy data transmission services during initial loading and strategy switching.

[0021] The present application will now be described in detail through specific embodiments.

[0022] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the simulation verification method for an embodied robot provided in this application embodiment includes the following steps: S1. Load the first WASM module and the second WASM module in the browser.

[0023] The robot physics simulation engine is a software library for simulating the interaction between robots and their physical environment. Internally, it implements multi-rigid-body dynamics algorithms, capable of calculating and updating the robot's joint positions, joint velocities, body postures, and other motion states based on physical quantities such as the driving torques of each joint, external contact forces, and gravity. This physics simulation engine performs numerical integration at fixed time steps, calculating and outputting the latest simulation state of the robot model at each step.

[0024] The neural network inference engine is a software framework for performing forward propagation computations in artificial neural networks. Internally, it implements the forward propagation logic for various operators in the neural network computation graph, including matrix multiplication, convolution operations, activation function calculations, and tensor shape transformations. The inference engine receives an input tensor, executes operators layer by layer according to a predefined computation graph, and finally generates an output tensor. In this scheme, the inference engine is specifically used to load and execute the forward inference of a robot reinforcement learning policy network.

[0025] The browser environment loads a first WASM (WebAssembly, a portable, small, fast-loading, and web-compatible binary format that can serve as a virtual machine target for executing compiled code in high-level languages) module and a second WASM module. The first WASM module is generated by compiling the source code of a robot physics simulation engine written in C or C++ using the Emscripten compiler, outputting a WebAssembly bytecode file executable by the browser. The second WASM module is generated by compiling the source code of a neural network inference engine written in C or C++ into a WebAssembly bytecode file using the Emscripten compiler. During loading, the browser allocates independent WASM memory spaces for each module and completes the instantiation and initialization of the modules.

[0026] S2. Obtain the structural description data and weight parameters of the robot's reinforcement learning policy network.

[0027] The robot reinforcement learning policy network is a pre-trained artificial neural network model used to define the mapping function from the observed state to the action to be performed by the robot in the simulation environment. The policy network receives the robot's current simulation state data as input, performs layer-by-layer calculations, and outputs the action data that the robot should perform in the next moment. The policy network is described by two parts of data: structural description data and weight parameters.

[0028] Obtain the structural description data and weight parameters of the aforementioned robot reinforcement learning policy network. The structural description data records the complete topological information of the policy network, specifically including the number of layers, the type of each layer, the dimensions of the input and output tensors of each layer, the kernel size and stride of convolutional layers, and the connection order between layers. The weight parameters are a large number of floating-point values ​​obtained after reinforcement learning training, corresponding to the weight matrix and bias vector of each layer in the policy network. The file containing the structural description data and weight parameters can be downloaded from a remote server via a network request, or the corresponding data can be read from the browser's local storage. The data exchange format for the structural description data can use structured text formats such as JSON, while the weight parameters can be in binary format to reduce transmission size.

[0029] S3. Based on the structural description data, allocate input and output buffers for the second WASM module, expose the addresses of the input and output buffers to the first WASM module, pre-allocate a memory pool for the second WASM module, and write the weight parameters into the specified memory area of ​​the second WASM module.

[0030] Specifically, based on the input and output tensor dimensions recorded in the structural description data, input and output buffers are allocated in the memory space of the second WASM module. The size of the input buffer is equal to the dimension of the input tensor multiplied by the number of bytes of a single floating-point number, and the size of the output buffer is equal to the dimension of the output tensor multiplied by the number of bytes of a single floating-point number. The starting addresses of the input and output buffers in the memory space of the second WASM module are obtained, and these two starting addresses are written to a preset storage location accessible by the first WASM module, thereby exposing the buffer addresses to the first WASM module.

[0031] This method also pre-allocates a fixed-size memory pool for the second WASM module based on the structural description data. Specifically, it iterates through the input and output tensors of each layer in the structural description data, calculating the number of intermediate tensors generated by the policy network during a single forward inference process and the size of each intermediate tensor. Intermediate tensors refer to the temporary results generated by each layer's computation during forward inference, excluding the input and output tensors. The sizes of all intermediate tensors are summed to obtain the total memory space requirement. A contiguous memory region of the same size as the total memory space requirement is reserved in the memory space of the second WASM module as a memory pool. This memory pool is reused during inference and is not dynamically expanded or reduced.

[0032] After the memory pool pre-allocation is completed, the weight parameters obtained in step S2 are written layer by layer into a designated memory region in the memory space of the second WASM module, according to the order of each layer of the policy network. This designated memory region is physically independent of the memory pool.

[0033] S4. Perform data dependency analysis on the computation graph of the robot reinforcement learning policy network, determine the lifecycle of each intermediate tensor, and control the reuse of the same memory segment in the memory pool for multiple intermediate tensors with non-overlapping lifecycles.

[0034] After pre-allocating the memory pool, data dependency analysis is performed on the computation graph of the robot's reinforcement learning policy network. The computation graph is a directed acyclic graph composed of operator nodes and data edges. Each operator node represents the forward computation of a layer of the network, and each data edge represents the flow of a tensor between nodes. For each tensor in the computation graph, the operator node that generated the tensor is marked as the starting node, and the operator node that last consumed the tensor is marked as the ending node. The lifecycle of each tensor is determined based on the positions of the starting and ending nodes in the topological sorting execution sequence. The lifecycle is defined as the continuous interval from the time point when the starting node begins execution to the time point when the ending node completes execution.

[0035] The lifetimes of all tensors are compared pairwise to identify multiple intermediate tensors whose lifetimes do not overlap temporally. For these intermediate tensors with non-overlapping lifetimes, a memory segment equal in size to the largest intermediate tensor is allocated in the memory pool, ensuring that these intermediate tensors use the same memory segment sequentially within their respective lifetimes. The offset of each reused intermediate tensor within this memory segment is recorded, and a mapping table between intermediate tensors and memory segment offsets is established for lookup during inference execution.

[0036] S5. Execute the simulation loop.

[0037] In each step of the simulation loop: the first WASM module drives the physical simulation of the robot model based on the action data of the previous step, generates simulation state data, and writes the simulation state data into the input buffer; the second WASM module reads the simulation state data from the input buffer as input, uses the reused memory segment in the memory pool and the weight parameters in the specified memory region to perform forward inference of the robot reinforcement learning policy network, generates the current step action data, and writes the current step action data into the output buffer; the first WASM module reads the current step action data from the output buffer as the action input for the next simulation step.

[0038] Specifically, after entering the simulation loop, the following operations are performed sequentially in each simulation step.

[0039] By executing the physical simulation code in the first WASM module, the motion data generated in the previous step is used as input instructions for the joint actuators in the robot model, driving the robot model to perform kinematic and dynamic solutions within one physical step. After the solution is completed, the first WASM module outputs simulation state data, including at least the position and velocity values ​​of each joint in the robot model. This simulation state data is then continuously written into the input buffer according to the dimensional order of the input tensors of the policy network in the second WASM module. During writing, the position and velocity values ​​of each joint are sequentially placed into the corresponding offset positions in the input buffer according to the feature arrangement order agreed upon during the training of the policy network.

[0040] Subsequently, by executing the inference code in the second WASM module, simulation state data is continuously read as input tensors for the policy network, starting from the beginning address of the input buffer. No data copying operations occur during the reading process; the inference code directly uses the memory address of the input buffer as the data source pointer for the input tensors. Utilizing the reused memory segments in the memory pool, and according to the offset mapping table established in step S4, the corresponding memory segment in the memory pool is allocated for each intermediate tensor, and the forward inference calculation of the policy network is performed layer by layer. The weight parameters required for the calculation of each layer are read from the specified memory region at preset offsets. After the calculation is completed, the memory segment occupied by the intermediate tensor is immediately released after its lifetime, for use by subsequent non-overlapping intermediate tensors. The entire forward inference process is always completed within the pre-allocated memory pool, without triggering dynamic memory allocation during operating system or browser runtime, nor triggering garbage collection mechanisms. The final generated current step action data is written to the output buffer.

[0041] Finally, by executing the first WASM module, the current step motion data is read from the output buffer and used as the motion command to drive the robot model in the next physical simulation, thus completing a complete simulation and reasoning loop.

[0042] In this embodiment, real-time simulation verification of embodied robot policies was implemented in a browser environment. The dual WASM module architecture enables physical simulation and policy inference to run collaboratively within the browser without requiring the user to install any local software. The memory pool pre-allocation mechanism and tensor lifecycle reuse strategy determine the memory layout of all intermediate tensors during the inference engine initialization phase, ensuring that each forward inference in the entire simulation loop does not trigger dynamic memory allocation and garbage collection operations. This helps eliminate inference latency jitter and guarantees deterministic real-time execution of the simulation loop. The address exposure mechanism for input and output buffers, combined with the direct writing of simulation state data according to the input tensor dimension order, achieves zero-copy data exchange between the physical engine and the inference engine, further reducing the data transport overhead for each simulation step. The combination of these techniques enables the physical simulation frequency to reach 1000 Hz in the browser, with a single policy inference time of less than 1 millisecond and a simulation rendering frame rate stable at over 60 frames per second, meeting the performance requirements for real-time simulation verification of robot reinforcement learning policies.

[0043] In an optional implementation, a memory pool is pre-allocated for the second WASM module, including: S21. Based on the structural description data, determine the number of intermediate tensors generated by each layer of the robot reinforcement learning policy network and the size of each intermediate tensor.

[0044] Based on the structural description data obtained in step S2, the configuration of each layer of the robot reinforcement learning policy network is analyzed one by one to determine the number of intermediate tensors generated by each layer during the network's forward inference process and the size of each intermediate tensor. An intermediate tensor refers to the temporary result data generated by each layer's computation during the forward inference process, excluding the input and output tensors. The structural description data records the type and output dimension of each layer. For a fully connected layer, its output activation values ​​constitute an intermediate tensor, and the size of this intermediate tensor is equal to the layer's output dimension multiplied by the number of bytes of a single floating-point number. For a convolutional layer, its output feature map constitutes an intermediate tensor, and its size is the number of output channels multiplied by the height of the output feature map multiplied by the width of the output feature map multiplied by the number of bytes of a single floating-point number. For multi-input layers such as concatenation layers or addition layers, each input tensor has already been counted in previous layers, and the new intermediate tensors generated in this layer are counted separately according to their output dimensions. After traversing each layer, the total number of intermediate tensors and the size of each intermediate tensor are summarized.

[0045] For example, following the aforementioned four-layer fully connected policy network, its structure is as follows: input 37 dimensions, first layer output 256 dimensions, second layer output 256 dimensions, third layer output 128 dimensions, and fourth layer output 8 dimensions. The first fully connected layer produces a 256-dimensional intermediate tensor with a size of 256 x 4 bytes = 1024 bytes. The second fully connected layer produces a 256-dimensional intermediate tensor, also with a size of 1024 bytes. The third fully connected layer produces a 128-dimensional intermediate tensor with a size of 128 x 4 bytes = 512 bytes. The fourth fully connected layer directly produces the output tensor, without considering the intermediate tensors. Therefore, this policy network has a total of three intermediate tensors with sizes of 1024 bytes, 1024 bytes, and 512 bytes, respectively.

[0046] S22. Based on the size of each intermediate tensor, calculate the total memory space requirement of all intermediate tensors.

[0047] The total memory space requirement is obtained by summing the sizes of all intermediate tensors determined in step S21. During summation, the intermediate tensors are added one by one in the order they were generated in the policy network, and the size of each intermediate tensor is calculated in bytes corresponding to its data type. For network parameters and activation values ​​represented using 32-bit floating-point numbers, each value occupies 4 bytes. The calculated total memory space requirement is the minimum capacity of the memory pool, which can simultaneously accommodate the total space required for all intermediate tensors to be stored independently. It should be noted that this calculation is the total space required for all intermediate tensors to be stored independently, not the actual space occupied after tensor reuse optimization in step S4. The actual space occupied after reuse optimization will be significantly smaller than this sum, but pre-allocation prioritizes ensuring the maximum requirement.

[0048] For example, summing the sizes of the three intermediate tensors determined in step S21 (1024 bytes + 1024 bytes + 512 bytes) results in a total memory requirement of 2560 bytes. If the policy network is larger, for example, containing 10 fully connected layers with each layer outputting 512 dimensions, then each layer generates a 2048-byte intermediate tensor, resulting in 10 intermediate tensors across 10 layers, with a total requirement of 20480 bytes. This calculation process involves only simple accumulation operations and can be completed in microseconds.

[0049] S23. In the memory space of the second WASM module, a contiguous memory region of the same size as the total memory space requirement is reserved as a memory pool.

[0050] Within the memory space of the second WASM module, a contiguous memory region with a fixed starting address and a size equal to the total memory space requirement calculated in step S22 is allocated and reserved as a memory pool. Specifically, during the initialization phase of the second WASM module, the WASM memory management mechanism allocates a contiguous space of a specified size from the module's linear free memory region in one go, and records the starting address and size of this memory region. Subsequently, throughout the entire lifecycle of the simulation loop, the boundaries of this memory pool remain fixed, and no release or reallocation operations are performed on this region. All intermediate tensor data storage during forward inference is allocated and reclaimed from this memory pool. Allocation and reclamation are achieved solely by updating the usage status of memory segments, without involving operating system-level memory management calls.

[0051] For example, when compiling the second WASM module, Emscripten typically allocates 16MB or more of linear memory for it. After module instantiation, a contiguous 2560-byte region is allocated from the free area of ​​linear memory as a memory pool, with its starting address recorded as 0x00100000 and size as 2560 bytes. This memory pool is used consistently in all subsequent simulation steps. Internally, the inference engine allocates temporary storage space for intermediate tensors within this memory pool using offsets, without calling dynamic memory allocation functions such as malloc or new.

[0052] This embodiment, based on the structural description data of the policy network, accurately calculates the total size of all intermediate tensors during the forward inference process and reserves a contiguous memory region of the corresponding size all at once. This pre-allocation mechanism ensures that the capacity of the memory pool is determined before inference begins, eliminating the need for any dynamic memory allocation operations during inference. This eliminates the unpredictability of time overhead caused by dynamic allocation and avoids the risk of allocation failure due to memory fragmentation. The contiguous memory region layout improves CPU cache hit rate, and the compact arrangement of intermediate tensor data in memory reduces memory access latency caused by cache misses, further improving the overall performance of inference computation.

[0053] In an optional embodiment, S3, controlling the same memory segment in multiple intermediate tensor reuse memory pools with non-overlapping lifecycles, includes: S31. Traverse the computation graph of the robot reinforcement learning policy network. For each intermediate tensor, mark the node that generated the intermediate tensor in the computation graph as the starting node and the node that consumed the intermediate tensor for the last time as the ending node.

[0054] The computation graph of the robot's reinforcement learning policy network is traversed. The computation graph is a directed acyclic graph composed of operator nodes and data edges. Each operator node represents a layer of computation in the network, and each data edge represents a tensor passed from one operator node to another. For each intermediate tensor, the operator node in the computation graph that first generated the intermediate tensor during execution is identified, and that operator node is marked as the starting node of the intermediate tensor. Simultaneously, the operator nodes in the computation graph that need to read the intermediate tensor as input during execution are identified, and the operator node that is last in the topological sorting execution order is found, and that operator node is marked as the ending node of the intermediate tensor, i.e., the node that last consumed the intermediate tensor. Thus, each intermediate tensor has obtained a pair of labels, pointing to its producer and its last consumer, respectively.

[0055] For example: Suppose there is a branching structure in the policy network. The first fully connected layer produces an intermediate tensor A, the second fully connected layer consumes A and produces an intermediate tensor B, and the third fully connected layer consumes B and produces an intermediate tensor C. Then, the starting node of A is the first fully connected layer and the ending node is the second fully connected layer. The starting node of B is the second fully connected layer and the ending node is the third fully connected layer. The starting node of C is the third fully connected layer and the ending node is the fourth fully connected layer. If there is also a skip connection structure in the policy network, where an intermediate tensor is consumed by multiple subsequent layers simultaneously, then the ending node is the one with the latest topological order among these consuming nodes.

[0056] S32. Determine the lifecycle of each intermediate tensor based on its start node and end node, where the lifecycle represents the time range from the start of execution at the start node to the completion of execution at the end node.

[0057] Specifically, based on the start and end nodes marked for each intermediate tensor in step S31, the lifecycle of the intermediate tensor is determined. First, the linear execution order of all operator nodes is determined according to the topological dependencies of the computation graph. For each intermediate tensor, the start point of its lifecycle is defined as the position where the start node begins execution in this execution order, and the end point of its lifecycle is defined as the position where the end node finishes execution in this execution order. The entire lifecycle represents a continuous time range from the start of execution by the start node to the completion of execution by the end node. Within this time range, the intermediate tensor needs to continuously occupy memory space because subsequent consumer nodes that have not yet executed may need to read the tensor's data at any time. Once the end node completes execution, the lifecycle of the intermediate tensor terminates, and the memory space it occupies can be released.

[0058] For example, following the aforementioned four-layer network, the linear execution order of the operator nodes is layer 1, layer 2, layer 3, and layer 4. The starting node of intermediate tensor A is in layer 1, and the ending node is in layer 2; its lifecycle begins execution in layer 1 and continues until layer 2 is completed. The starting node of intermediate tensor B is in layer 2, and the ending node is in layer 3; its lifecycle begins execution in layer 2 and continues until layer 3 is completed. The lifecycles of A and B overlap because A is still being consumed during layer 2 execution, while B has already begun to be generated during layer 2 execution. Both intermediate tensors A and B require memory at the same time, therefore they cannot share the same memory segment.

[0059] S33. Identify multiple intermediate tensors whose lifecycles do not overlap, allocate a memory segment in the memory pool, use the memory segment sequentially within their respective lifecycles, and record the offset of each intermediate tensor within the memory segment.

[0060] In this process, the lifecycles of all intermediate tensors identified in step S32 are compared one by one to identify combinations of intermediate tensors whose lifecycles do not overlap in time. Non-overlapping means that the lifecycle intervals of two intermediate tensors do not coincide at any given time; the lifecycle of one tensor ends completely before the lifecycle of another tensor begins. For multiple identified intermediate tensors with non-overlapping lifecycles, a memory segment of appropriate size is allocated in the memory pool. The size of this memory segment is at least equal to the size of the largest tensor in the group. This group of intermediate tensors is mapped to this memory segment, allowing them to use the same physical memory region sequentially within their respective lifecycles. The offset of each reused intermediate tensor within this memory segment is recorded; this offset represents the starting position of the intermediate tensor's data within the memory segment. Since multiple intermediate tensors in a reuse group do not exist simultaneously in time, their offsets can all be set to the starting offset of the memory segment (zero offset), or different offset values ​​can be set according to actual alignment requirements. This offset information is recorded in a mapping table for querying during forward inference in the simulation loop.

[0061] For example, suppose there are two parallel branches in the policy network. Branch 1 processes the following order: the first convolutional layer generates an intermediate tensor X, the first activation layer consumes X and generates an intermediate tensor Y, and then Y is consumed by the second convolutional layer. At this point, the lifetimes of X and Y have ended. Branch 2 executes after Branch 1, and its processing order is: the third convolutional layer generates an intermediate tensor P, the second activation layer consumes P and generates an intermediate tensor Q. The lifetimes of X and Y in Branch 1 do not overlap with the lifetimes of P and Q in Branch 2. Allocate a memory segment in the memory pool, with the largest size among the four intermediate tensors X, Y, P, and Q. When Branch 1 executes, X and Y are stored sequentially in this memory segment. After Branch 1 finishes, Branch 2 begins execution, and P and Q are stored sequentially in the same memory segment. Record the offsets of X, Y, P, and Q within this memory segment; since they are used sequentially, the offsets can all be zero.

[0062] In this embodiment, a memory reuse strategy based on data dependency analysis enables multiple intermediate tensors that originally required independent memory to share the same physical memory in a time-sharing manner. This significantly reduces the actual memory footprint of the inference engine's memory pool while ensuring computational correctness. Compared to a pre-allocation scheme without reuse optimization, this optimization can compress the memory pool size by 30% to 60%. The reduced memory space can be used to further expand the model size or load more policy networks, or run policy network models with larger parameter sets within the same memory capacity, thus improving the model carrying capacity of the browser-side simulation verification system.

[0063] In one possible implementation, exposing the addresses of the input buffer and the output buffer to the first WASM module includes: A1. Obtain the first starting address of the input buffer in the memory space of the second WASM module, and the second starting address of the output buffer in the memory space of the second WASM module.

[0064] After allocating input and output buffers for the second WASM module in step S3, the starting addresses of these two buffers within the second WASM module's memory space are obtained. The input buffer occupies a contiguous byte sequence within the second WASM module's memory space, with its starting address being the offset of the first byte of this byte sequence within the WASM linear memory, i.e., the first starting address. Similarly, the output buffer occupies a contiguous byte sequence, with its starting address being the offset of the first byte of this byte sequence within the WASM linear memory, i.e., the second starting address. These two address values ​​are obtained through the memory management interface of the second WASM module. This interface can be a function exported by the WASM module, which returns the memory offset of the buffer within the module. The obtained first and second starting addresses are integer values, representing the byte offset starting from the first address of the WASM linear memory.

[0065] For example: The total linear memory space of the second WASM module is 16MB, starting at address 0x00000000. A contiguous 148-byte space is allocated for the input buffer, starting at offset 0x00100000 in the linear memory, thus the first starting address is 0x00100000. A contiguous 32-byte space is allocated for the output buffer, immediately following the memory pool, starting at offset 0x00100A00, thus the second starting address is 0x00100A00. Calling the `get_input_buffer_address` and `get_output_buffer_address` functions exported by the second WASM module yields return values ​​of 0x00100000 and 0x00100A00, respectively.

[0066] A2. Write the first starting address and the second starting address into a preset storage location accessible to the first WASM module, so that the first WASM module can read the first starting address and the second starting address from the preset storage location during execution, thereby performing data reading and writing on the input buffer and the output buffer.

[0067] Specifically, the first and second starting addresses obtained in step A1 are written to a preset storage location accessible to the first WASM module. This preset storage location is a shared memory region allocated by the browser when loading the two WASM modules. This region does not belong to the independent linear memory space of either WASM module, but is a memory region supported by a shared ArrayBuffer provided by the browser. Both WASM modules can read and write to this region through their respective memory access interfaces. The first and second starting addresses are written to the preset storage location at agreed offsets. After writing, the first WASM module reads the first and second starting addresses from the agreed offsets at this preset storage location at each step of the simulation loop. After obtaining these two addresses, the first WASM module uses them as internal data read / write target pointers. Through the direct access capability of WASM linear memory, it writes simulation state data to the memory location corresponding to the first starting address and reads the current step action data from the memory location corresponding to the second starting address.

[0068] For example, when a browser loads two WASM modules, JavaScript glue code creates an 8-byte shared ArrayBuffer. The first 4 bytes of this ArrayBuffer store a first starting address, and the last 4 bytes store a second starting address. The JavaScript code writes the first starting address (0x00100000) to the first 4 bytes of the ArrayBuffer and the second starting address (0x00100A00) to the last 4 bytes. The first WASM module internally has a reference to this shared ArrayBuffer. After the physical simulation calculations in the simulation loop are completed, the first WASM module reads 0x00100000 from the first 4 bytes of the ArrayBuffer and writes the simulation state data directly to a contiguous memory region starting at that address. After inference is completed, the first WASM module reads 0x00100A00 from the last 4 bytes of the ArrayBuffer and reads the action data from the contiguous memory region starting at that address. The entire read / write process does not involve any data copying; the first WASM module directly manipulates the buffer data in the memory space of the second WASM module through addresses.

[0069] A zero-copy data channel based on address sharing was established between the two WASM modules. The first WASM module gained direct read and write capabilities to the input and output buffers in the memory space of the second WASM module by reading the buffer start address stored in a preset storage location. In each step of the simulation loop, simulation state data and action data do not need to undergo data copying or serialization / deserialization operations through intermediate buffers. The physics engine directly writes to the input buffer of the inference engine, and the inference engine directly writes to the output buffer for the physics engine to read. This zero-copy mechanism eliminates the additional overhead of cross-module data transmission, reducing the data exchange time of a single-step simulation to below microseconds, providing crucial support for the real-time requirements of a physics simulation frequency of 1000 Hz and a rendering frame rate stable at over 60 frames per second.

[0070] Furthermore, when executing the simulation loop, the MuJoCo (Multi-Joint dynamics with Contact, an open-source physics engine for multi-joint rigid body dynamics simulation, widely used in robotics and biomechanics research) physics engine is selected as the robot physics simulation engine. This engine implements a rigid body dynamics solution algorithm based on the Newton-Euler equations, which can calculate and output the motion state of each joint of the robot model at a fixed time step.

[0071] First, determine the dimensional order of the input tensors for the robot's reinforcement learning policy network. During the training phase of the policy network, the arrangement of features in the input tensors is fixed. For example, the first few dimensions of the observation vector correspond to the position values ​​of each joint, the subsequent few dimensions correspond to the velocity values ​​of each joint, and the remaining dimensions correspond to other features such as body posture angular velocity. Obtain this dimensional order convention from the structural description data or the policy network's metadata.

[0072] In each step of the simulation loop, the robot model completes a physical simulation calculation for one step by executing the first WASM module, which is generated by the MuJoCo physics engine. The MuJoCo physics engine outputs simulation state data, which includes at least the joint position vector and joint velocity vector for each active joint of the robot model. The joint position vector consists of the rotational angle or translational displacement of each joint, and the joint velocity vector consists of the rotational angular velocity or translational linear velocity of each joint.

[0073] After obtaining the joint position and velocity vectors, they are not directly written to the input buffer according to the data structure order within MuJoCo. Instead, they are arranged and written sequentially according to the input tensor dimension order agreed upon during policy network training. Specifically, starting from the first address of the input buffer, each component of the joint position vector is written sequentially to the first few storage units of the buffer, followed by the sequential writing of each component of the joint velocity vector. If the input tensor of the policy network contains other features such as body pose angles or sensor readings, they are continued to be written sequentially according to the agreed order. The writing process uses direct write operations of WASM linear memory, with each floating-point number occupying 4 bytes. The components in the vector are stored closely adjacently in the buffer without any padding bytes or separators.

[0074] Since the data is arranged in an order completely consistent with the dimensions of the policy network input tensor during writing, the second WASM module only needs to use the starting address of the input buffer as the data source pointer for the input tensor in subsequent inference steps, and directly read the continuously stored numerical sequence in tensor form. The second WASM module does not need to perform any data rearrangement, dimension conversion, or format adaptation operations, nor does it need to create a copy of the input tensor, thus achieving zero-copy and conversion-free connection between the physical simulation output and the policy network input.

[0075] For example, consider the walking simulation of a bipedal humanoid robot. The robot's joint configuration includes 12 movable joints, each with 6 degrees of freedom. The input tensor of the policy network has 37 dimensions. The agreed-upon feature arrangement is as follows: the first 12 dimensions correspond to the position values ​​of the 12 joints; dimensions 13 to 24 correspond to the velocity values ​​of the 12 joints; dimensions 25 to 27 correspond to the body's three-axis attitude angles; dimensions 28 to 30 correspond to the body's three-axis angular velocities; and dimensions 31 to 37 correspond to other sensor features.

[0076] In a certain step of the simulation loop, the MuJoCo physics engine calculates the current rotation angle values ​​of the 12 joints as follows: 0.523, -0.261, 0.785, -1.047, 0.349, 0.174, -0.523, 0.261, -0.785, 1.047, -0.349, and 0.000, in radians. Simultaneously, the current rotational angular velocity values ​​of the 12 joints are also calculated. Starting from the beginning address of the input buffer, these 12 position values ​​are written consecutively, followed by 12 velocity values, and then attitude angles and other feature values, totaling 37 32-bit floating-point numbers, occupying 148 consecutive bytes. During inference, the second WASM module directly reads these 37 floating-point numbers from the beginning address of the input buffer as a complete input tensor and performs forward computation on the policy network.

[0077] By defining the robot physics simulation engine as the MuJoCo physics engine and writing the joint position vectors and joint velocity vectors from the simulation state data continuously into the input buffer according to the dimensional order of the policy network input tensor, data layout coordination between the physics engine output and the inference engine input is achieved. The joint state data generated by the MuJoCo physics engine is aligned with the policy network input format when written to the input buffer, eliminating the need for any data format conversion or memory rearrangement operations when the inference engine reads it. This design decouples the physics simulation module and the policy inference module from their internal data structure differences through a unified data layout convention, while achieving tightly coupled direct data transfer at the memory level. This eliminates the additional computational and memory copy overhead caused by inserting a data adaptation layer between the two modules in traditional solutions, further reducing the end-to-end latency of each simulation loop and ensuring the real-time performance and determinism of the simulation system during high-frequency operation.

[0078] In an optional implementation, the method of this application further includes: B1. During the simulation loop, acquire the structural description data and weight parameters of the robot reinforcement learning policy network to be switched.

[0079] During the continuous operation of the simulation loop, a policy switching command is received from an external source, specifying the target policy network to which the switch should take place. Based on this identifier, the structural description data and weight parameters of the robot reinforcement learning policy network to be switched to are downloaded from a remote server via a network request, or the corresponding data is read from the browser's local storage. The format of the downloaded or read data is completely consistent with the format of the initial policy network data obtained in step S2. The structural description data uses a structured text format such as JSON (JavaScript Object Notation, a lightweight structured data exchange format) to record the complete topology information of the network, and the weight parameters use a binary format to store the floating-point values ​​of the weight matrix and bias vector of each layer. While receiving data, the current simulation loop does not stop. The physical simulation and policy inference continue to run using the original policy network until steps B2 and B3 are completed, at which point the new policy network is activated in subsequent steps.

[0080] For example: A user is observing a real-time simulation of a bipedal robot's walking strategy in a browser. They then select a jumping strategy from the strategy list on the webpage and click the switch button. Upon receiving the switch command, an HTTP request is sent to the server to download the JSON file describing the jumping strategy network's structure and the binary file containing the weight parameters. The jumping strategy network's structure consists of a 37-dimensional observation vector as input, a 512-dimensional output from the first fully connected layer, a 256-dimensional output from the second fully connected layer, and a 12-dimensional action vector from the third fully connected layer. The weight parameter file is approximately 220KB in size. The download process takes about 0.8 seconds. During this time, the simulation continues to run the original walking strategy, and the robot model's walking animation in the browser remains unaffected.

[0081] B2. Write the weight parameters of the robot reinforcement learning policy network to be switched to the specified memory area to replace the original weight parameters.

[0082] After obtaining the weight parameters of the policy network to be switched, the weight parameters of the original policy network within a specified memory region in the second WASM module are replaced. The specified memory region is a contiguous memory space reserved in step S3 for storing the weight parameters; its size is reserved during initialization according to the weight size of the original policy network. If the total size of the weight parameters of the policy network to be switched is less than or equal to the capacity of the specified memory region, the new weight parameters are directly written byte by byte into the memory region, sequentially overwriting the original data from the region's starting address in layer order. If the total size of the weight parameters of the policy network to be switched is greater than the capacity of the original specified memory region, the boundaries of the specified memory region need to be adjusted first, expanding the region from the free memory of the second WASM module. However, the size and address of the memory pool remain unaffected, as they are independent of each other in memory space. After writing is complete, the original weight parameters are completely replaced, and the specified memory region stores the complete weight parameters of the policy network to be switched.

[0083] For example: The weight parameters of the original walking strategy network occupy approximately 520KB of space, and 600KB of capacity is reserved in the second WASM module's memory space in the designated memory region. The weight parameters of the jump strategy network to be switched occupy approximately 220KB, which is less than the 600KB reserved capacity. The 220KB of jump strategy weight data is written continuously starting from address 0x00200000 in the designated memory region, overwriting the original walking strategy weight data. The write operation is completed in approximately 1 millisecond. The remaining 380KB of space in the designated memory region is not released temporarily for use in subsequent switches.

[0084] B3. Perform data dependency analysis on the computation graph of the robot reinforcement learning policy network to be switched, redetermine the lifecycle of each intermediate tensor, and control the reuse of the same memory segment in the memory pool for multiple intermediate tensors with non-overlapping lifecycles to replace the original memory segment reuse relationship.

[0085] In this process, the computation graph of the robot reinforcement learning policy network to be switched undergoes a complete data dependency analysis process to replace the original memory segment reuse relationship. First, based on the structural description data of the policy network to be switched, all operator nodes and data edges in the computation graph are traversed, and the start and end nodes of each intermediate tensor are marked. Second, the lifecycle of each intermediate tensor is determined according to the execution order of the topology sort, i.e., the time range from the start of execution at the start node to the completion of execution at the end node. Third, all combinations of intermediate tensors with non-overlapping lifecycles are identified, and a memory segment with a size equal to the largest tensor size in the group is allocated in the memory pool for each group of non-overlapping intermediate tensors, and the offset of each intermediate tensor within that memory segment is recorded. Finally, the newly established memory segment reuse relationship table replaces the original reuse relationship table established in step S4. This replacement action briefly locks the inference engine's inference operations during execution; it is immediately unlocked after the replacement is completed, and subsequent steps in the simulation loop use the new reuse relationship table for memory allocation.

[0086] It should be noted that if the total size of the intermediate tensors of the network to be switched exceeds the memory pool capacity pre-allocated in step S3, the original memory pool needs to be released and a larger memory pool reallocated. However, this situation is usually avoided by reserving sufficient redundant memory pool capacity during initialization to ensure that the policy switch can be completed within the time interval of one simulation step.

[0087] For example, the jump policy network to be switched has three intermediate tensors: intermediate tensor D (512 dimensions, 2048 bytes), intermediate tensor E (256 dimensions, 1024 bytes), and intermediate tensor F (12 dimensions, 48 ​​bytes). The lifecycle of D starts from the first layer and ends at the second layer; the lifecycle of E starts from the second layer and ends at the third layer; and the lifecycle of F starts from the third layer and ends at the output layer. The lifecycles of D and E overlap and cannot be reused. The lifecycles of D and F also overlap and cannot be reused. Therefore, this policy network requires three independent memory segments, totaling 3120 bytes. The memory pool has a reserved space of 2560 bytes, which is insufficient to accommodate this policy network. Upon detecting insufficient memory pool capacity, the original 2560 bytes of memory pool are released, and a new 4096-byte contiguous region is reallocated from the free memory of the second WASM module. A reuse relationship table is then established based on the new memory pool. The entire release and redistribution operation is completed in approximately 0.5 milliseconds.

[0088] Furthermore, parallel acceleration techniques are employed when performing inference computations.

[0089] Single Instruction Multiple Data (SIMD) is an instruction set-level parallel computing technique that allows the same computational operation to be performed on multiple data points simultaneously under the control of a single instruction. During the compilation phase of the second WASM module, developers enabled the SIMD compilation option provided by the Emscripten toolchain, enabling the code in the neural network inference engine involving large-scale matrix multiplication and vector operations to be compiled into bytecode containing WASM SIMD instructions. WASM SIMD instructions support 128-bit wide vector registers and can process four 32-bit floating-point numbers or eight 16-bit integers simultaneously in a single instruction.

[0090] In each inference step of the simulation loop, the second WASM module is executed to read simulation state data from the input buffer as the input tensor, and the forward inference of the policy network is performed layer by layer on the memory segments reused in the memory pool. When the inference reaches the matrix multiplication operation of the fully connected layer, a row vector of the weight matrix is ​​loaded into the WASM SIMD vector register, and the corresponding column vector of the input tensor is loaded into another vector register. By fusing multiply-accumulate instructions, the multiplication and accumulation of four floating-point numbers are completed in a single instruction. For input dimensions that are not multiples of 4, scalar instructions are used to supplement the calculations on the last four elements, without affecting the overall SIMD acceleration effect.

[0091] When inference reaches element-wise operations such as activation function calculation or vector addition, the input vector is loaded into the vector register in groups of four floating-point numbers. The corresponding SIMD instructions then perform parallel calculations of ReLU (Rectified Linear Unit) activation values ​​or addition of bias vectors. Intermediate tensor data generated during computation is written to the specified offset position in the corresponding memory segment of the memory pool according to the offset mapping table established in step S4 or step B3. In the next layer of computation, intermediate tensor data is read from the memory segment, and parallel computation continues using SIMD instructions until the final layer generates the current step action data and writes it to the output buffer.

[0092] Throughout the forward inference process, all matrix multiplications and vector operations are performed within the memory pool. Intermediate and final results are stored in pre-allocated memory segments without triggering any dynamic memory allocation operations.

[0093] For example, taking a four-layer fully connected policy network as an example, the input tensor is a 37-dimensional vector, the first-layer weight matrix is ​​256 rows multiplied by 37 columns, and the output is a 256-dimensional intermediate tensor. When calculating the first element of the first-layer output, the 37 elements of the first row of the weight matrix need to be multiplied by the 37 elements of the input tensor. These 37 floating-point numbers are divided into 9 groups of 4 elements each, with the last element processed separately. For each group of 4 elements, the `wasm_f32x4_mul` instruction is used to multiply the weight row vector fragment with the input vector fragment element-wise, and the `wasm_f32x4_add` instruction is used to accumulate the multiplication results into an accumulator register. After the 9 groups of calculations are completed, the results in the 4 accumulator registers are horizontally summed, and the scalar product of the last 37th element is added to obtain the final value of the output element. This process is repeated row-by-row to calculate all 256 rows of the weight matrix, generating a complete 256-dimensional output intermediate tensor, which is stored in the corresponding memory segment in the memory pool.

[0094] For the ReLU activation operation of the 256-dimensional intermediate tensor, it is divided into 64 groups, each with 4 floating-point numbers. The wasm_f32x4_max instruction is used to compare with the zero vector, and the non-negative truncation operation of the 4 elements is completed in one instruction.

[0095] This embodiment fully leverages the vectorized parallel processing capabilities of modern CPUs by invoking SIMD instructions for matrix multiplication and vector operations during the forward inference of the policy network, achieving near-native machine code computational throughput in the WebAssembly environment. Compared to scalar computation without SIMD optimization, theoretically, with a 128-bit SIMD instruction width, the throughput of floating-point operations can be increased to approximately four times that of the scalar method. Actual inference time is thus significantly reduced, with a single inference computation time of less than 1 millisecond, ensuring that the forward inference of the policy network does not become a performance bottleneck in the simulation loop. The combination of SIMD acceleration and a memory pool pre-allocation mechanism jointly guarantees the real-time performance and determinism of browser-side robot reinforcement learning policy simulation verification from both computational and memory efficiency perspectives.

[0096] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0097] Please see Figure 3 This document illustrates a schematic diagram of a simulation verification device for an embodied robot provided in an exemplary embodiment of this application, hereinafter referred to as device 3. Device 3 can be implemented as all or part of a simulation verification device for an embodied robot through software, hardware, or a combination of both. Device 3 includes: Loading module 301 is used to load a first WASM module and a second WASM module in a browser. The first WASM module is compiled and generated by a robot physics simulation engine, and the second WASM module is compiled and generated by a neural network inference engine. The acquisition module 302 is used to acquire the structural description data and weight parameters of the robot reinforcement learning policy network; The allocation module 303 is used to allocate an input buffer and an output buffer to the second WASM module according to the structure description data, expose the addresses of the input buffer and the output buffer to the first WASM module, pre-allocate a memory pool for the second WASM module, and write the weight parameters into a specified memory area of ​​the second WASM module. The reuse module 304 is used to perform data dependency analysis on the computation graph of the robot reinforcement learning policy network, determine the life cycle of each intermediate tensor, and control multiple intermediate tensors with non-overlapping life cycles to reuse the same memory segment in the memory pool. The simulation module 305 is used to execute a simulation loop. In each step: the first WASM module drives the physical simulation of the robot model based on the action data of the previous step, generates simulation state data, and writes the simulation state data into the input buffer; the second WASM module reads the simulation state data from the input buffer as input, uses the reused memory segment in the memory pool and the weight parameters in the specified memory region to perform forward inference of the robot reinforcement learning policy network, generates the current step action data, and writes the current step action data into the output buffer; the first WASM module reads the current step action data from the output buffer as the action input for the next simulation step.

[0098] For further details regarding the implementation of the above-mentioned technical solutions by the modules in the control device of the simulation verification device for the embodied robot, please refer to the description in the simulation verification method for the embodied robot provided in the above-mentioned embodiments of the invention, which will not be repeated here.

[0099] It should be noted that the device 3 provided in the above embodiments, when executing the simulation verification method for embodied robots, is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the above functions. In addition, the simulation verification device for embodied robots and the simulation verification method embodiment for embodied robots provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiment, which will not be repeated here.

[0100] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0101] Figure 4 The diagram shown is a schematic of a computer storage medium provided in an embodiment of this application. The computer storage medium can store multiple instructions (i.e., ... Figure 4 (a computer program in the above), the instructions are adapted to be loaded and executed by a processor as described above. Figure 2 The method steps of the illustrated embodiment can be found in the following documentation for detailed execution. Figure 2 The specific details of the illustrated embodiments will not be elaborated here.

[0102] This application also provides a computer program product that stores at least one instruction, which is loaded and executed by the processor to implement the simulation verification method of the embodied robot as described in the above embodiments.

[0103] Please see Figure 5 This document provides a schematic diagram of the structure of a computer device according to an embodiment of this application. Figure 5 As shown, the computer device 500 may include: at least one processor 501, at least one network interface 504, user interface 503, memory 505, and at least one communication bus 502.

[0104] The communication bus 502 is used to enable communication between these components.

[0105] The user interface 503 may include input / output units such as a display screen, a camera, a keyboard, and a touch screen.

[0106] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface) for communication and data transmission with external devices.

[0107] The processor 501 may include one or more processing cores. The processor 501 connects to various parts within the computer device 500 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 505, and by calling data stored in the memory 505. Optionally, the processor 501 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 501 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip, without being integrated into the processor 501.

[0108] The memory 505 may include random access memory (RAM) or read-only memory. Optionally, the memory 505 may include a non-transitory computer-readable storage medium. The memory 505 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 505 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 505 may also be at least one storage device located remotely from the aforementioned processor 501. Figure 5 As shown, the memory 505, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and application programs.

[0109] exist Figure 5In the computer device 500 shown, the user interface 503 is mainly used to provide an input interface for the user and to obtain the user's input data; while the processor 501 can be used to call the application program stored in the memory 505 and specifically execute, such as Figure 1 The method shown can be referred to for details. Figure 1 As shown, it will not be elaborated further here.

[0110] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0111] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A simulation verification method for an embodied robot, characterized in that, include: Load the first WASM module and the second WASM module in the browser. The first WASM module is compiled and generated by the robot physics simulation engine, and the second WASM module is compiled and generated by the neural network inference engine. Obtain the structural description data and weight parameters of the robot's reinforcement learning policy network; Based on the structure description data, an input buffer and an output buffer are allocated to the second WASM module, the addresses of the input buffer and the output buffer are exposed to the first WASM module, a memory pool is pre-allocated for the second WASM module, and the weight parameters are written into a designated memory area of ​​the second WASM module. Data dependency analysis is performed on the computation graph of the robot reinforcement learning policy network to determine the lifecycle of each intermediate tensor, and control multiple intermediate tensors with non-overlapping lifecycles to reuse the same memory segment in the memory pool. The simulation loop is executed. In each step: the physical simulation of the robot model is driven by the first WASM module based on the action data of the previous step, simulation state data is generated, and the simulation state data is written into the input buffer. The second WASM module reads the simulation state data from the input buffer as input, uses the reused memory segment in the memory pool and the weight parameters in the specified memory region to perform forward inference of the robot reinforcement learning policy network, generates the current step action data, and writes the current step action data into the output buffer; the first WASM module reads the current step action data from the output buffer as the action input for the next simulation step.

2. The method according to claim 1, characterized in that, The pre-allocation of the memory pool for the second WASM module includes: Based on the structural description data, determine the number of intermediate tensors generated by each layer of the robot reinforcement learning policy network and the size of each intermediate tensor; Calculate the total memory space requirement for all intermediate tensors based on their dimensions. In the memory space of the second WASM module, a contiguous memory region of the same size as the total memory space requirement is reserved as the memory pool.

3. The method according to claim 1, characterized in that, The multiple intermediate tensors with non-overlapping lifecycles that reuse the same memory segment in the memory pool include: Traverse the computation graph of the robot reinforcement learning policy network. For each intermediate tensor, mark the node that generated the intermediate tensor in the computation graph as the starting node and the node that consumed the intermediate tensor for the last time as the ending node. The lifecycle of each intermediate tensor is determined based on its start node and end node, wherein the lifecycle represents the time range from the start of execution at the start node to the completion of execution at the end node. Identify multiple intermediate tensors whose lifecycles do not overlap, allocate a memory segment in the memory pool, use the memory segment sequentially within the lifecycles of the multiple intermediate tensors, and record the offset of each intermediate tensor within the memory segment.

4. The method according to claim 1, characterized in that, Exposing the addresses of the input buffer and the output buffer to the first WASM module includes: Obtain the first starting address of the input buffer in the memory space of the second WASM module, and the second starting address of the output buffer in the memory space of the second WASM module; The first starting address and the second starting address are written to a preset storage location accessible to the first WASM module, so that the first WASM module can read the first starting address and the second starting address from the preset storage location during execution, thereby performing data read and write operations on the input buffer and the output buffer.

5. The method according to claim 1, characterized in that, The robot physics simulation engine is the MuJoCo physics engine, and the simulation state data includes at least the joint position vector and joint velocity vector of the robot model. By executing the first WASM module, the joint position vector and the joint velocity vector are continuously written into the input buffer according to the dimensional order of the input tensor of the robot reinforcement learning policy network, so that the second WASM module can directly read the joint position vector and joint velocity vector continuously stored in the input buffer as the input tensor.

6. The method according to claim 1, characterized in that, Also includes: During the simulation loop, the structural description data and weight parameters of the robot reinforcement learning policy network to be switched are acquired; Write the weight parameters of the robot reinforcement learning policy network to be switched into the designated memory area to replace the original weight parameters. Data dependency analysis is performed on the computation graph of the robot reinforcement learning policy network to be switched, the life cycle of each intermediate tensor is redefined, and multiple intermediate tensors with non-overlapping life cycles are controlled to reuse the same memory segment in the memory pool to replace the original memory segment reuse relationship. In the subsequent steps of the simulation loop, after the writing and replacement of the reuse relationship are completed, the forward inference of the robot reinforcement learning policy network to be switched is performed by the second WASM module using the redefined reuse relationship in the memory pool and the replaced weight parameters in the specified memory region.

7. The method according to claim 1, characterized in that, The step of performing forward inference of the robot reinforcement learning policy network through the second WASM module, using the reused memory segments in the memory pool and the weight parameters in the specified memory region, includes: The second WASM module utilizes the reused memory segments in the memory pool to perform matrix multiplication and / or vector operations in the robot reinforcement learning policy network, calling Single Instruction Multiple Data (SID) to generate the current step action data.

8. A simulation verification device for an embodied robot, characterized in that, The device includes: A loading module is used to load a first WASM module and a second WASM module in the browser. The first WASM module is compiled and generated by the robot physics simulation engine, and the second WASM module is compiled and generated by the neural network inference engine. The acquisition module is used to acquire the structural description data and weight parameters of the robot's reinforcement learning policy network; The allocation module is used to allocate an input buffer and an output buffer to the second WASM module according to the structure description data, expose the addresses of the input buffer and the output buffer to the first WASM module, pre-allocate a memory pool for the second WASM module, and write the weight parameters into a specified memory area of ​​the second WASM module. The reuse module is used to perform data dependency analysis on the computation graph of the robot reinforcement learning policy network, determine the life cycle of each intermediate tensor, and control multiple intermediate tensors with non-overlapping life cycles to reuse the same memory segment in the memory pool. The simulation module is used to execute a simulation loop. In each step: the first WASM module drives the physical simulation of the robot model based on the action data of the previous step, generates simulation state data, and writes the simulation state data into the input buffer; the second WASM module reads the simulation state data from the input buffer as input, uses the reused memory segment in the memory pool and the weight parameters in the specified memory region to perform forward inference of the robot reinforcement learning policy network, generates the current step action data, and writes the current step action data into the output buffer; the first WASM module reads the current step action data from the output buffer as the action input for the next simulation step.

9. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the simulation verification method for the embodied robot as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the simulation verification method for the embodied robot as described in any one of claims 1 to 7.

Images