A method for simulating robot motion, electronic device and storage medium

Abstract

This application relates to the field of computer technology, providing a robot motion simulation method, electronic device, and storage medium. It reconstructs a 3D scene model, including precise geometric and semantic information, based on real-world scene data; generates a physical collision mesh using the geometric information; and determines differentiated physical simulation parameters for different regions of the physical collision mesh based on semantic labels. By integrating the physical collision mesh and physical simulation parameters, a high-fidelity simulation environment is constructed within a simulation platform. This overcomes the technical bottlenecks of traditional methods, which rely on cumbersome manual modeling and overly simplified and uniform physical parameter settings. It achieves the generation of a digital environment with precisely aligned physical properties, directly usable for robot motion simulation testing, from raw scene data. This lays a reliable foundation for training robot control strategies under highly realistic conditions, improves the success rate of transferring simulation to reality, and reduces the risk of strategy deployment failure due to simulation environment distortion.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a robot motion simulation method, electronic device, and storage medium. Background Technology

[0002] In the research and development of robotics, especially legged robots (such as quadruped robots), physics-based simulators are key tools for algorithm development, control strategy training, and system testing. High-fidelity simulation environments can significantly reduce the cost and risk of physical experiments and accelerate the iteration process.

[0003] The creation of current simulation environments relies heavily on manual labor. Technicians manually construct the geometric model of the scene in 3D modeling software based on the target scene, and set physical property parameters (such as friction coefficient, elastic coefficient, etc.) for different parts of the model (such as ground, walls, obstacles) one by one based on experience.

[0004] This manual modeling and parameter tuning is inefficient and cannot quickly reproduce diverse real-world scenarios. Furthermore, the manually set physical parameters are generally simplified and uniform, failing to accurately reflect the complexity and spatial non-uniformity of physical properties in real-world scenarios due to differences in materials, structures, and states (such as wetness, dryness, and age). For example, a real courtyard containing concrete, grass, and sand is often simplified in simulation to a plane with a single physical property. This leads to a fundamental difference between the interactive dynamics learned by the robot control strategy in simulation and the real world.

[0005] Therefore, this difference between simulation and reality makes it easy for control strategies trained in simulation to slip, gait incoordination, obstacle avoidance failure, or even body tipping over when transferred to real robots. Summary of the Invention

[0006] This application provides a robot motion simulation method, electronic device, and storage medium, aiming to solve the problems in the prior art where, due to systematic deviations between the simulation environment and the real world in terms of visual, geometric, and physical properties, the control strategies trained in the simulation are prone to slippage, uncoordinated gait, obstacle avoidance failure, or even tipping over when transferred to the real robot.

[0007] This application provides a robot motion simulation method, including: Based on real-world scene data, a 3D scene model including geometric information and semantic labels is generated. Based on the geometric information of the 3D scene model, generate the physical collision mesh of the 3D scene model; Based on the semantic labels of the 3D scene model, corresponding physical simulation parameters are determined for different semantic regions of the physical collision mesh. The physical simulation parameters are used to characterize the physical contact properties of each semantic region. Based on the physical collision mesh and physical simulation parameters, a simulation environment corresponding to the real scene is constructed. Simulate the robot's motion in a simulation environment and generate simulation results of the robot's motion in the simulation environment.

[0008] In one possible implementation, the steps of constructing a simulation environment corresponding to the real scene based on the physical collision mesh and physical simulation parameters include: Import the geometric data of the physical collision mesh into the physics simulation engine; By associating and binding physical simulation parameters with the corresponding semantic regions of the physical collision mesh, the physical simulation engine can perform simulation calculations using the bound physical simulation parameters when it detects contact with a specified semantic region.

[0009] In one possible implementation, the steps of simulating robot motion in a simulation environment and generating simulation results of robot motion in the simulation environment include: In the physics simulation engine, the robot control strategy is run to generate the original motion instructions. The robot control strategy is trained in the physics simulation engine with the physics simulation parameters configured. The robot's real-time state in the physical simulation engine is obtained and input into a pre-trained residual network to obtain the motion compensation amount output by the residual network. The robot's real-time state includes at least one of joint position, joint velocity, joint torque, or body posture. The motion compensation amount is superimposed on the original motion command to generate the execution action used to drive the robot's motion simulation.

[0010] In one possible implementation, the residual network is obtained in the following way: Acquire the actual motion state data of the robot when executing a preset action sequence; Reproduce preset action sequences in the physics simulation engine to obtain simulated motion state data; The residual network is trained using the difference between real motion state data and simulated motion state data as a supervision signal.

[0011] In one possible implementation, the steps of generating a 3D scene model including geometric information and semantic labels based on real-world scene data include: Based on image data from real-world scenes, a 3D Gaussian sputtering method is used to reconstruct the 3D scene and generate a 3D Gaussian representation model of the real scene. In the process of reconstructing a 3D scene using the 3D Gaussian sputtering method, at least one of the following geometric prior constraints is introduced: 2D plane regularization, unbiased depth rendering, multi-view photometric consistency, and normal prior.

[0012] In one possible implementation, the steps of reconstructing a 3D scene based on image data from a real-world scene using a 3D Gaussian sputtering method to generate a 3D Gaussian representation model of the real-world scene include: Perform two-dimensional semantic segmentation on image data to obtain two-dimensional semantic labels; Two-dimensional semantic labels are mapped to a three-dimensional Gaussian representation model, thus endowing spatial points in the three-dimensional Gaussian representation model with semantic information.

[0013] In one possible implementation, based on the semantic labels in the 3D scene model, corresponding physical simulation parameters are determined for different semantic regions of the physical collision mesh, including: Obtain prompts including the robot's physical attributes, target task description, and semantic tags; The prompt words are input into a pre-trained large language model to obtain the physical simulation parameters of spatial changes corresponding to the semantic labels generated by the large language model. The physical contact properties represented by the physical simulation parameters include at least one of the following: friction coefficient, elastic coefficient, stiffness or damping.

[0014] This application also provides an electronic device, including: Memory, used to store instructions; The processor is used to execute the aforementioned robot motion simulation method by calling instructions stored in memory.

[0015] This application also provides a computer-readable storage medium storing computer-executable instructions. When the processor executes the computer-executable instructions, it implements the above-described robot motion simulation method.

[0016] This application also provides a computer program product, which, when read and executed by a computer, causes the aforementioned robot motion simulation method to be executed.

[0017] This application provides a robot motion simulation method, electronic device, and storage medium. The motion simulation method first reconstructs a 3D scene model, including precise geometric and semantic information, based on real-world scene data. Then, it uses the geometric information of the 3D scene model to generate a physical collision mesh for physical interaction, and determines differentiated physical simulation parameters for different regions of the physical collision mesh based on semantic labels. Finally, it integrates the physical collision mesh with the spatially varying physical simulation parameters to construct a high-fidelity simulation environment in a simulation platform. This solves the technical bottlenecks of traditional methods, such as reliance on cumbersome manual modeling and overly simplified and uniform physical parameter settings, enabling the generation of a digital environment with precisely aligned physical properties from raw scene data, directly usable for robot motion simulation testing. This lays a reliable foundation for training robot control strategies under highly realistic conditions, improves the success rate of simulation-to-real-world transfer, and reduces the risk of strategy deployment failure due to simulation environment distortion. Attached Figure Description

[0018] To more clearly illustrate the implementation methods in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0019] Figure 1 A flowchart illustrating a robot motion simulation method provided in this application embodiment. Figure 1 .

[0020] Figure 2 A flowchart illustrating a robot motion simulation method provided in this application embodiment. Figure 2 .

[0021] Figure 3 A flowchart illustrating a robot motion simulation method provided in this application embodiment. Figure 3 .

[0022] Figure 4 A flowchart illustrating a robot motion simulation method provided in this application embodiment. Figure 4 .

[0023] Figure 5 A flowchart illustrating a robot motion simulation method provided in this application embodiment. Figure 5 .

[0024] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of this application.

[0026] It should be noted that many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of this application is not limited to the specific embodiments disclosed below.

[0027] In this application, the use of terms such as "first," "second," etc., is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features.

[0028] The various embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0029] This application provides a robot motion simulation method, electronic device, and storage medium, aiming to solve the problems in the prior art where, due to systematic deviations between the simulation environment and the real world in terms of visual, geometric, and physical properties, the control strategies trained in the simulation are prone to slippage, uncoordinated gait, obstacle avoidance failure, or even tipping over when transferred to the real robot.

[0030] like Figure 1 As shown in the embodiment of this application, a robot motion simulation method includes steps S110 to S150.

[0031] Step S110: Based on real scene data, generate a 3D scene model including geometric information and semantic labels.

[0032] Among them, real scene data is data collected from the robot's real motion environment. For example, real scene data is an RGB video sequence from a first-person perspective. For example, by using a handheld RGB camera to record video around the target training scene (such as a courtyard containing grass, cement, and sand pits), the RGB video sequence of the real scene from a first-person perspective can be obtained. In this case, the first-person perspective is the viewpoint of the RGB camera.

[0033] After acquiring the RGB video sequence of the real scene, the RGB video sequence is processed to generate a 3D scene model that can be rendered from the robot's perspective, including geometric information and semantic labels. The 3D scene model not only contains the color and geometric information of the scene, but also assigns semantic labels to each spatial point or region in the 3D scene model, such as grass, cement, and sand. The 3D scene model is used to provide the robot with visual observation and rendering from its perspective.

[0034] Step S120: Generate the physical collision mesh of the 3D scene model based on the geometric information of the 3D scene model.

[0035] After obtaining the 3D scene model, precise geometric information is extracted from it. Based on this geometric information, a watertight triangular mesh, or physical collision mesh, is generated using a meshing algorithm. For example, the traveling cube method is used. The physical collision mesh is a simplified, facet-free representation of the real scene geometry, specifically designed for high-speed collision detection in physical simulations.

[0036] Among them, watertightness specifically refers to the generated triangular mesh model being a closed, hole-free manifold surface, so that each edge in the physical collision mesh is shared by only two triangular faces, allowing the physical collision mesh to clearly distinguish between the inside and outside, like a sealed container.

[0037] Step S130: Based on the semantic labels of the 3D scene model, determine the corresponding physical simulation parameters for different semantic regions of the physical collision mesh.

[0038] Based on semantic tags, the physical simulation parameters that each semantic region should possess are inferred through automated means. For example, a region with the semantic tag "sand" is assigned a lower coefficient of friction and certain plasticity parameters. Conversely, a region with the semantic tag "cement" is assigned a higher coefficient of friction and stiffness.

[0039] Physical simulation parameters are used to characterize the physical contact properties of each semantic region, so as to accurately characterize the interaction properties of each semantic region when it comes into physical contact with the robot in motion simulation.

[0040] Step S140: Based on the physical collision mesh and physical simulation parameters, construct a simulation environment corresponding to the real scene.

[0041] Specifically, the physical collision mesh is used as the geometric skeleton of the simulation environment, and the physical simulation parameters are used as the surface properties of different regions of the geometric skeleton. The physical collision mesh and the physical simulation parameters are configured together to construct a digital simulation environment that corresponds to the real scene in terms of geometric shape and physical interaction characteristics.

[0042] Step S150: Simulate the robot's motion in the simulation environment and generate the motion simulation results of the robot in the simulation environment.

[0043] In a high-fidelity simulation environment, a digital model of the robot is loaded. By driving the robot to move within the simulation environment, and based on the physical collision mesh and bound physical simulation parameters, all forces and motions during the robot's interaction with the environment are calculated. Motion simulation results are generated, including a series of data such as robot pose, joint states, and contact forces. These results are used to analyze the robot's potential performance in corresponding real-world scenarios or to train robot control strategies.

[0044] This approach first reconstructs a 3D scene model based on real-world scene data, incorporating precise geometric and semantic information. Then, it utilizes the geometric information from the 3D scene model to generate a physical collision mesh for physical interaction, and determines differentiated physical simulation parameters for different regions of the physical collision mesh based on semantic labels. Finally, it integrates the physical collision mesh with the spatially varying physical simulation parameters to construct a high-fidelity simulation environment within the simulation platform. This overcomes the technical bottlenecks of traditional methods, which rely on cumbersome manual modeling and overly simplified and uniform physical parameter settings for simulation environment construction. It achieves the generation of a digital environment with precisely aligned physical properties, directly usable for robot motion simulation testing, from raw scene data. This lays a reliable foundation for training robot control strategies under highly realistic conditions, improves the success rate of transferring simulation to reality, and reduces the risk of strategy deployment failure due to simulation environment distortion.

[0045] To instantiate abstract physical collision meshes and physical simulation parameters into a computable simulation environment. In some embodiments, such as Figure 2 As shown, the steps for constructing a simulation environment corresponding to the real scene based on the physical collision mesh and physical simulation parameters include steps S210 to S220.

[0046] Step S210: Import the geometric data of the physical collision mesh into the physics simulation engine.

[0047] The application programming interface (API) of the physics simulation engine is invoked. For example, the simulation engine could be MuJoCo (Multi-Jointdynamics with Contact), PyBullet (Python Bullet, a Python wrapper for the Bullet physics engine), or the RaiSim simulator. The vertex coordinate data and triangle patch index data of the physics collision mesh are encapsulated according to the format required by the physics simulation engine (such as coordinate system and data arrangement), and passed to the physics simulation engine through the API function. The physics simulation engine creates and maintains a static collider representing the environmental terrain in memory; the shape of the collider is defined by the passed-in physics collision mesh.

[0048] Step S220: Associate and bind the physical simulation parameters with the corresponding semantic regions of the physical collision mesh.

[0049] First, within the physics simulation engine, multiple material objects with different properties are created. Each material object contains a set of physical parameters, such as static friction coefficient, dynamic friction coefficient, elastic coefficient, stiffness, or damping. These physical parameter values ​​are derived from the physical simulation parameters determined for different semantic regions.

[0050] Then, establish the binding relationship between a specific set of faces of the physical collision mesh (corresponding to a certain semantic region, such as all faces marked as ice) and the corresponding material object (such as low friction material). For example, the binding relationship can be achieved by attaching a material ID to the mesh face, or by creating sub-collision bodies for different regions in the physical simulation engine and specifying materials for each.

[0051] This allows the physics simulation engine to perform simulation calculations using bound physical simulation parameters when it detects contact with a specified semantic region. Specifically, the contact solver of the physics simulation engine is configured to automatically perform simulation calculations of contact forces, friction forces, etc., using the physical simulation parameters bound to the semantic region when contact between a robot (or other object) and a specified semantic region of the physical collision mesh is detected. This creates a simulation environment within the physics simulation engine that corresponds geometrically and physically to the real-world scene.

[0052] This application integrates physical collision meshes and physical simulation parameters into a physical simulation engine to ensure the engineering feasibility and repeatability of the simulation environment, providing a reliable environmental foundation for subsequent high-precision dynamic simulations.

[0053] In the established simulation environment, to further compensate for the inherent dynamic model errors of the physics simulation engine and generate simulation results that are highly consistent with the motion trajectory of the real robot, in some embodiments, such as Figure 3As shown, the steps for simulating robot motion in a simulation environment include steps S310 to S330.

[0054] Step S310: In the physics simulation engine, run the robot control strategy to generate original motion commands.

[0055] The robot control strategy is trained in a physical simulation engine configured with physical simulation parameters.

[0056] In the simulation environment of the physics simulation engine, a pre-trained robot control strategy is run. The robot control strategy takes the robot's real-time observed state (such as joint encoder feedback and body inertial measurement unit data) as input and outputs a set of raw motion commands.

[0057] For example, the original motion command is the desired joint position, velocity, or torque. The robot control strategy is trained in the same physical simulation engine with configured physical simulation parameters through reinforcement learning. Therefore, the robot control strategy can adapt to the physical characteristics of the simulation environment.

[0058] Step S320: Obtain the real-time state of the robot in the physical simulation engine, and input the real-time state into the pre-trained residual network to obtain the motion compensation amount output by the residual network.

[0059] During each motion simulation step run by the physics simulation engine, the real-time state of the robot model is acquired synchronously. The real-time state includes at least one of joint position, joint velocity, joint torque, or body posture. The acquired real-time state is input into a pre-trained residual network, which outputs a motion compensation amount based on the input real-time state.

[0060] Motion compensation is designed to correct the inherent discrepancies between the simplified rigid body dynamics model of the physics simulation engine and the actuator dynamics of the real robot (such as motor friction, gear backlash, and signal delay).

[0061] Step S330: The motion compensation amount is superimposed with the original motion command to generate the execution action used to drive the robot motion simulation.

[0062] The original motion commands are vector-added to obtain the final executed motion. The physics simulation engine receives the executed motion and applies it to the robot model, driving the robot to perform the next movement. In this way, the robot's dynamic response in the simulation environment is dynamically corrected by a residual network based on the original model, making the dynamic response of the simulated robot closer to that of a real robot.

[0063] The pre-trained residual network is obtained as follows: First, a series of preset action sequences (such as stepping, walking, and trotting) are executed on a real robot, and real motion state data such as high-frequency joint encoders, motor currents, and body poses are recorded simultaneously. Then, in the physics simulation engine, the simulated robot model is controlled to reproduce the exact same preset action sequence, and corresponding simulated motion state data is collected. Finally, using minimizing the difference between the real motion state data and the simulated motion state data (such as the mean square error of joint angle trajectories) as a supervision signal, a reinforcement learning algorithm is used to train the residual network. After training, the weights of the residual network are frozen, thus obtaining the pre-trained residual network.

[0064] For example, reinforcement learning algorithms include proximal policy optimization algorithms. Proximal policy optimization algorithms employ an asymmetric Actor-Critic architecture. The Actor network only receives observations directly obtainable within the simulation, while the Critic network receives augmented states containing privileged information such as the global true position to guide value assessment. Furthermore, the reward function incorporates joint space tracking error penalties, gait contact mimicry incentives, and regularization terms to limit excessively large residual amplitudes.

[0065] In this way, by introducing a data-driven dynamic residual compensation mechanism, compared with traditional system identification or global randomization methods, it is possible to fit nonlinear dynamic errors that are difficult to model more precisely, so that the overall response of the physical simulation engine is aligned with the real scene.

[0066] In some embodiments, in order to reconstruct a geometrically consistent, high-precision, and physically interactive 3D scene model from an RGB video sequence, a 3D Gaussian sputtering method is used to reconstruct the 3D scene based on image data from real scene data, generating a 3D Gaussian representation model of the real scene.

[0067] The specific process is as follows: First, the structure-of-motion reconstruction algorithm is used on the input RGB video sequence to recover the camera pose of each frame and generate a sparse point cloud. Then, a 3D Gaussian primitive is used to perform differentiable parametric modeling of the scene. Each Gaussian primitive is defined by parameters such as center position, covariance matrix (controlling the shape of the ellipsoid), opacity, and spherical harmonic coefficients (controlling color). By optimizing the parameters of the Gaussian primitives, the color error between the image rendered from any viewpoint (especially the robot's viewpoint) and the original image in the video is minimized.

[0068] In the optimization process of 3D Gaussian sputtering, geometric prior constraints are introduced as part of the loss function to force the reconstructed 3D geometry to conform to the structural priors of the physical world. The geometric prior constraints include at least one of the following: 2D plane regularization, unbiased depth rendering, multi-view photometric consistency, and normal prior.

[0069] Two-dimensional planar regularization applies constraints to near-planar regions (such as ground and walls) in a three-dimensional scene model, causing Gaussian elements within the region to flatten and conform to the real surface.

[0070] Unbiased depth rendering uses an improved volume rendering formula to calculate depth, avoiding the depth value deviation that may be caused by standard volume rendering, thus obtaining a more accurate depth map of the geometry.

[0071] Multi-view luminosity consistency is achieved by introducing multi-view normalized cross-correlation loss during optimization, which forces the local image patches rendered by the 3D Gaussian model to maintain consistent luminosity under different viewpoints, thereby improving multi-view geometric consistency.

[0072] Normal prior uses the surface normal map predicted by the single image normal prediction model on the input video frame as supervision to constrain the normal map rendered from the 3D Gaussian model, thereby improving the geometric realism of the reconstructed surface.

[0073] By integrating one or more of the aforementioned geometric prior constraints for joint optimization, the resulting 3D Gaussian representation model is not only visually realistic but also possesses a high-precision geometric structure. This lays the foundation for subsequent extraction of high-quality physical collision meshes and effectively overcomes the geometric noise, floating artifacts, and structural fragmentation problems caused by traditional 3D modeling which relies solely on photometric loss. Simultaneously, it can extract coherent, watertight surfaces and preserve clear terrain boundaries, achieving a balance between visual fidelity and geometric accuracy.

[0074] In some embodiments, such as Figure 4 As shown, the steps for reconstructing a 3D scene and generating a 3D Gaussian representation model of the real scene based on image data from real scene data include steps S410 to S420.

[0075] Step S410: Perform two-dimensional semantic segmentation on the image data to obtain two-dimensional semantic labels.

[0076] Using a two-dimensional semantic segmentation model pre-trained on a large image dataset, forward inference is performed on each frame of the input real-world RGB video to predict an initial semantic category (such as sky, road, vegetation) for each pixel in the image, thereby generating a two-dimensional semantic segmentation pseudo-label map for each frame. The pseudo-label map provides preliminary, multi-view supervision information on scene semantics.

[0077] Step S420: Map the two-dimensional semantic labels to the three-dimensional Gaussian representation model, and assign semantic information to the spatial points in the three-dimensional Gaussian representation model.

[0078] In the initialization stage of geometric and appearance reconstruction of the scene using 3D Gaussian sputtering technology, a learnable semantic feature vector is assigned to each 3D Gaussian primitive, and the semantic feature vector serves as the carrier of the semantic information of the 3D Gaussian primitive.

[0079] In optimizing the 3D Gaussian representation model, a 2D semantic segmentation pseudo-label map is introduced as a supervision signal. Through optimization algorithms, the semantic information rendered by the 3D model from different perspectives is forced to remain consistent with the corresponding 2D pseudo-labels. After multi-view joint optimization, the semantic information is successfully and consistently embedded into each spatial point of the 3D Gaussian representation model. Ultimately, the system obtains a 3D Gaussian representation model containing semantic information. When it is necessary to generate 2D semantic labels from a specific perspective (such as a robot's perspective), these labels can be directly rendered and decoded from this semantically encoded 3D model.

[0080] In this way, this embodiment realizes the transfer of two-dimensional visual semantics to three-dimensional geometric space, providing a reliable data foundation for the subsequent accurate transmission of semantic information to the physical collision mesh and the realization of semantic-based differentiated physical parameter allocation.

[0081] In some embodiments, such as Figure 5 As shown, based on the semantic labels in the 3D scene model, the corresponding physical simulation parameters are determined for different semantic regions of the physical collision mesh, including steps S510 to S520.

[0082] Step S510: Obtain prompt words including the robot's physical attributes, target task description, and semantic tags.

[0083] A cue word is a structured text that typically includes robot context, task context, query request, and a list of semantic tags.

[0084] Robot context describes the basic information and physical properties of the target robot. For example, robot context could be: a quadruped robot weighing 20 kg, with its legs made of rubber.

[0085] The task context describes the task that the robot is scheduled to perform. For example, the task context could be: to run dynamically and turn quickly in a simulation environment.

[0086] A query request explicitly requests the large language model to infer physical parameters based on common sense. For example, a query request could be: Based on the above information, determine the reasonable range of static friction coefficient, dynamic friction coefficient, elastic coefficient, and damping values ​​for the following materials when they come into contact with the robot's foot.

[0087] Semantic labels are given in the form of a list, which lists all semantic labels extracted from the physical collision mesh. For example, the material list includes: dry asphalt pavement, short-pile artificial turf, smooth indoor tiles, compacted dirt road, loose sand, and ice surface.

[0088] Step S520: Input the prompt word into the pre-trained large language model to obtain the physical simulation parameters of spatial changes corresponding to the semantic labels generated by the large language model.

[0089] The constructed prompts are input into a pre-trained large language model. Based on the physical common sense and world knowledge contained in the massive training corpus, the large language model infers the meaning of each semantic label. For example, if the semantic label is "ice surface," the large language model might output: "static friction coefficient 0.01-0.1, kinetic friction coefficient 0.005-0.05, elasticity coefficient close to 0." Another example: if the semantic label is "short-pile artificial turf," the large language model might output: "static friction coefficient 0.8-1.2, kinetic friction coefficient 0.6-1.0, with some elasticity."

[0090] The output text of the large language model is parsed, and the physical parameter values ​​or ranges corresponding to each semantic tag are extracted. These physical parameter values ​​or ranges are then associated with the corresponding semantic regions on the physical collision mesh, thus completing the automatic allocation of physical simulation parameters. The physical contact properties represented by the physical simulation parameters include at least one of the following: friction coefficient, elastic coefficient, stiffness, or damping.

[0091] In this way, by leveraging the common-sense reasoning capabilities of large language models, the limitations of traditional methods that rely on manual experience to set physical parameters or simple table lookups are overcome. This allows for the automatic generation of spatially varying simulation parameters that conform to physical laws for complex and heterogeneous scenarios, thereby improving the accuracy of robot motion simulation in simulation environments.

[0092] This application also provides an electronic device for implementing the above-described robot motion simulation method. For example... Figure 6 As shown, the electronic device 600 includes a memory 610 and a processor 620.

[0093] The memory 610 is used to store computation instructions, such as program code for implementing the above-mentioned robot motion simulation method, trained robot control strategy model, pre-trained residual network model, pre-trained large language model interface, and intermediate processing data.

[0094] The processor 620 is coupled to the memory 610. The processor 620 is used to call and execute instructions stored in the memory 610 to perform the following operations: generating a 3D scene model including geometric information and semantic labels based on real scene data; generating a physical collision mesh of the 3D scene model based on the geometric information of the 3D scene model; determining corresponding physical simulation parameters for different semantic regions of the physical collision mesh based on the semantic labels of the 3D scene model, the physical simulation parameters being used to characterize the physical contact properties of each semantic region; constructing a simulation environment corresponding to the real scene based on the physical collision mesh and physical simulation parameters; performing simulated robot motion in the simulation environment, and generating simulation results of robot motion in the simulation environment.

[0095] This application also provides a computer-readable storage medium, which can be a USB flash drive, a portable hard drive, a read-only memory, a random access memory, a magnetic disk, or an optical disk, or any other medium capable of storing program code.

[0096] The computer-readable storage medium stores computer-executable instructions. When the processor of the electronic device executes the computer-executable instructions, the processor implements the above-mentioned robot motion simulation method.

[0097] This application also provides a computer program product that, when read and executed by a computer, causes the aforementioned robot motion simulation method to be executed.

[0098] The computer program product can be a software package or SDK (Software Development Kit), or it can be a computer program stored on a computer-readable storage medium (such as a USB flash drive, solid-state drive, optical disc, or cloud storage space).

Claims

1. A method for simulating the motion of a robot, characterized in that, include: Based on real-world scene data, a 3D scene model including geometric information and semantic labels is generated. Based on the geometric information of the 3D scene model, a physical collision mesh for the 3D scene model is generated; Based on the semantic labels of the 3D scene model, corresponding physical simulation parameters are determined for different semantic regions of the physical collision mesh. The physical simulation parameters are used to characterize the physical contact properties of each semantic region. Based on the physical collision mesh and the physical simulation parameters, a simulation environment corresponding to the real scene is constructed. The robot's motion is simulated in a simulation environment, generating motion simulation results of the robot in that environment.

2. The method according to claim 1, characterized in that, The step of constructing a simulation environment corresponding to the real scene based on the physical collision mesh and the physical simulation parameters includes: Import the geometric data of the physical collision mesh into the physics simulation engine; The physical simulation parameters are associated and bound to the corresponding semantic regions of the physical collision mesh, so that when the physical simulation engine detects contact with the specified semantic region, it uses the bound physical simulation parameters to perform simulation calculations.

3. The method according to claim 2, characterized in that, The steps of simulating the robot's motion in a simulation environment and generating the motion simulation results of the robot in the simulation environment include: In the physical simulation engine, the robot control strategy is run to generate original motion commands. The robot control strategy is trained in the physical simulation engine configured with the physical simulation parameters. The robot's real-time state in the physics simulation engine is obtained, and the real-time state is input into a pre-trained residual network to obtain the motion compensation amount output by the residual network. The robot's real-time state includes at least one of joint position, joint velocity, joint torque, or body posture. The motion compensation amount is superimposed on the original motion command to generate an execution action for driving the robot's motion simulation.

4. The method according to claim 3, characterized in that, The residual network is obtained in the following way: Obtain the actual motion state data of the robot executing the preset action sequence; The preset action sequence is reproduced in the physical simulation engine to obtain simulation motion state data; The residual network is trained using the difference between the real motion state data and the simulated motion state data as a supervision signal.

5. The method according to claim 1, characterized in that, The step of generating a 3D scene model including geometric information and semantic labels based on real scene data includes: Based on the image data in the real scene data, the three-dimensional scene is reconstructed using the three-dimensional Gaussian sputtering method to generate a three-dimensional Gaussian representation model of the real scene; In the process of reconstructing a 3D scene using the 3D Gaussian sputtering method, at least one of the following geometric prior constraints is introduced: 2D plane regularization, unbiased depth rendering, multi-view photometric consistency, and normal prior.

6. The method according to claim 5, characterized in that, The steps of reconstructing a 3D scene and generating a 3D Gaussian representation model of the real scene based on image data from the real scene data include: Two-dimensional semantic segmentation is performed on the image data to obtain two-dimensional semantic labels; The two-dimensional semantic labels are mapped to the three-dimensional Gaussian representation model, thereby assigning semantic information to the spatial points in the three-dimensional Gaussian representation model.

7. The method according to any one of claims 1-6, characterized in that, The step of determining the corresponding physical simulation parameters for different semantic regions of the physical collision mesh based on the semantic tags in the 3D scene model includes: Obtain prompt words including the robot's physical attributes, target task description, and semantic tags; The prompt word is input into a pre-trained large language model to obtain the physical simulation parameters of spatial variation corresponding to the semantic label generated by the large language model. The physical contact properties characterized by the physical simulation parameters include at least one of the following: friction coefficient, elastic coefficient, stiffness or damping.

8. An electronic device, characterized in that, include: Memory, used to store instructions; A processor for invoking instructions stored in the memory to execute the method of any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, When the computer reads and executes the computer program product, the method as described in any one of claims 1 to 7 is performed.

Images