Robot simulation control method, simulation system, device and storage medium
By generating a real-life three-dimensional model and binding it to the twin mathematical model, combining dynamics and vision simulators to train the robot control model, the problem of high video memory in complex scenarios is solved, and efficient simulation data generation and real robot control are achieved.
Patent Information
- Application Number
- CN202510518785.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-06-07
- Filing Date
- 2025-04-24
- Publication Date
- 2025-08-05
AI Technical Summary
In the prior art, the end-to-end deep learning model has high memory requirements in complex scenarios and is difficult to directly apply to actual scenarios. The simulation data training model requires additional data acquisition during real-life deployment.
By obtaining environmental information at the simulation robot application site, a real-life three-dimensional model is generated and bound to the twin mathematical model, a dynamic simulator is used to simulate motion, combined with a visual simulator to synchronize rendering, and training the robot control model.
It realizes that a large amount of simulation data can be generated in arbitrary scenes and objects only need to be modeled once, which is used for real-life robot control, reduces memory requirements and improves efficiency, and completes training in a simulation environment to reduce security risks.
Smart Images
Figure CN120428593A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a robot simulation control method, a simulation system, a device and a storage medium, and belongs to the technical field of artificial intelligence. Background Art
[0002] Robot control models (for example, the ACT (Action Chunking With Transformer) algorithm) are end-to-end deep learning models that can perform real-time robot control based on visual information, joint encoders, and torque sensor information as observations. This means they can map images to actions, allowing the robot to learn and imitate specific actions from visual input. Currently, end-to-end deep learning models require simulation environments (such as Mujoco and Gazebo) for robot training, followed by verification of model performance within the simulation environment. However, models trained with simulation data are difficult to directly apply to real-world scenarios; data collection and training are still required.
[0003] The existing method for training in simulation and deploying in reality is ProgressiveNetwork, which often runs three models in parallel (a simulation model, a transition model from simulation to reality, and a real-world model), using the output of the previous model's current feature map as the input of the next model. Because this method requires running three identical models in parallel, it places high demands on video memory and may not be applicable to networks with relatively complex model structures. Summary of the Invention
[0004] The object of the present invention is to provide a robot simulation control method, simulation system, device and storage medium for solving the problems existing in the prior art.
[0005] In order to achieve the above object, the present invention provides the following technical solutions:
[0006] According to a first aspect, an embodiment of the present invention provides a robot simulation control method, the method comprising:
[0007] Acquire environmental information of the application site of the simulation robot, and generate a real-scene three-dimensional model based on the environmental information;
[0008] Binding the real-scene three-dimensional model and the twin mathematical model;
[0009] Training a robot control model based on the real-scene three-dimensional model in different postures and the twin mathematical model, wherein the trained robot control model is used for controlling the robot;
[0010] The steps of training the robot control model include:
[0011] simulating different movements of the simulated robot in the twin mathematical model by a dynamics simulator;
[0012] During the process of simulating the movement of the simulated robot, the dynamic simulator transmits the motion data to the visual simulator in real time, and the movement of the real-scene three-dimensional model is synchronously rendered by the visual simulator; the twin mathematical model is trained based on the image data rendered by the real-scene three-dimensional model in different postures and the dynamic data generated by the twin digital model.
[0013] In a second aspect, a simulation system is provided, comprising:
[0014] A visual simulator for generating realistic 3D models,
[0015] Dynamics simulator for generating twin mathematical models,
[0016] A controller is used to make judgments and decisions based on the image output generated by the visual simulator and / or the contact force information generated by the dynamic simulator and input the decision results to the dynamic simulator; the dynamic simulator transmits motion data to the visual simulator in real time, and the visual simulator synchronously renders the motion of the real-scene three-dimensional model.
[0017] In a third aspect, a model binding method is provided for binding a real-scene 3D model and a twin mathematical model, comprising the following steps:
[0018] Bind the size information and coordinate information in the real-scene three-dimensional model and the twin mathematical model.
[0019] Optionally, binding the size information in the real-scene three-dimensional model and the twin mathematical model includes:
[0020] Obtaining a first coordinate set of the environmental information in a calibration plate coordinate system;
[0021] When constructing the Gaussian model, obtaining a second coordinate set including the environmental information of the calibration plate;
[0022] Converting the coordinates in the second coordinate set to a third coordinate set that has the same size as the coordinates in the first coordinate set;
[0023] Obtaining a rotation and translation matrix of the first coordinate set and the third coordinate set;
[0024] The real-scene three-dimensional model and the twin mathematical model are size-bound according to the rotation and translation matrix.
[0025] Optionally, converting the coordinates in the second coordinate set into a third coordinate set having a size consistent with the coordinates in the first coordinate set includes:
[0026] Calculating a first distance between adjacent coordinate points in the first coordinate system;
[0027] Calculating a second distance between adjacent coordinate points in the second coordinate system;
[0028] Calculating a distance proportional coefficient k according to the first distance and the second distance;
[0029] Each coordinate point in the second coordinate set is scaled k times to form the third coordinate set.
[0030] Optionally, obtaining the rotational and translational moments of the first coordinate set and the third coordinate set includes:
[0031] Calculating the centroid coordinates of the third coordinate set and the first coordinate set respectively;
[0032] Obtaining coordinates of the third coordinate set and the first coordinate set after decentralization;
[0033] Calculating a covariance matrix based on the decentralized third coordinate set and the first coordinate set;
[0034] Obtaining a rotation matrix according to the covariance matrix;
[0035] A translation vector is obtained according to the center of mass of the first coordinate set, the center of mass of the third coordinate set, and the rotation matrix.
[0036] Optionally, the environmental information includes posture information and contact force information of the simulated robot at the application site, object position and contact force information of an object contacted by the simulated robot, and scene information around the simulated robot.
[0037] Optionally, the controller includes at least one of a keyboard, VR, mouse and an analog object.
[0038] In a fourth aspect, a robot simulation control device is provided, which includes a memory and a processor, wherein the memory stores at least one program instruction, and the processor implements the method described above by loading and executing the at least one program instruction.
[0039] In a fifth aspect, a computer storage medium is provided, wherein at least one program instruction is stored in the computer storage medium, and the at least one program instruction is loaded and executed by a processor to implement the method described above.
[0040] By acquiring environmental information of the application site of the simulated robot, a real-scene 3D model is generated based on the environmental information; the real-scene 3D model and the twin mathematical model are bound; a robot control model is trained based on the real-scene 3D model and the twin mathematical model in different postures, and the trained robot control model is used to control the robot; wherein the step of training the robot control model includes: simulating different movements of the simulated robot in the twin mathematical model through a dynamic simulator; during the simulation of the movement of the simulated robot, the dynamic simulator transmits the movement data to the visual simulator in real time, and the visual simulator synchronously renders the movement of the real-scene 3D model; and the twin mathematical model is trained based on the image data rendered by the real-scene 3D model in different postures and the dynamic data generated by the twin digital model. This solves the problem of the existing technology that requires high video memory and may not be applicable in complex scenes. For any scene and object, only one model is needed to set the object placement arbitrarily. Through object placement and scene design, combined with simulated robot control or planning, a large amount of simulation data can be generated for real-world robot control, which is more efficient than previous methods.
[0041] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 A schematic diagram of a possible application scenario involved in various embodiments of the present invention;
[0043] Figure 2 A flowchart of a robot simulation control method provided by one embodiment of the present invention;
[0044] Figure 3 A schematic diagram of a principle for verifying a trained robot control model provided by one embodiment of the present invention. DETAILED DESCRIPTION
[0045] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.
[0046] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0047] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.
[0048] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0049] Please refer to Figure 1 , which shows a possible system structure diagram of a simulation system involved in various embodiments of the present application, such as Figure 1 As shown, the simulation system may include:
[0050] The visual simulator 110 is used to generate a real-scene 3D model; optionally, the visual simulator 110 can be a real-scene 3D model real-time renderer, and in actual implementation, it can be a Gaussian renderer, and Figure 1 Taking the visual simulator 110 as a Gaussian renderer as an example, in one possible implementation scenario, a simulated robot can use a camera to capture environmental information about the simulated robot at the application site, and generate a real-scene 3D model based on the environmental information. The environmental information includes the simulated robot's posture information at the application site, information about objects the simulated robot is in contact with, and information about the scene surrounding the simulated robot.
[0051] For example, if a simulated robot is on a table and is used to pick up a cup of water, the environmental information includes the posture information of the simulated robot, the posture information of the cup, the position information of the table, the relative position information of the three, and the contact force information between the robot terminal and the cup. In actual implementation, the environmental information can completely correspond to the information of the actual application site.
[0052] In one possible embodiment, to implement subsequent steps, after obtaining the real-scene 3D model, the visual simulator 110 may synchronize the data of the real-scene 3D model with the dynamics simulator 120. Alternatively, the image data may be synchronized with the dynamics simulator 120 so that the dynamics simulator 120 can perform subsequent steps.
[0053] The dynamics simulator 120 is used to generate a twin mathematical model. As a possible embodiment, Figure 1 As shown, the dynamics simulator 120 can be Mujoco. When the visual simulator 110 synchronizes image data with the dynamics simulator 120, the dynamics simulator 120 can obtain a mathematical model of the simulated robot at the application site and a mathematical model of the object the simulated robot touches, thereby generating dynamics data. The mathematical model of the simulated robot and the mathematical model of the object are twin mathematical models of the 3D real-world model. The twin mathematical model is bound to the 3D real-world model, and the two bound models have the same shape, size, and posture. The dynamics simulator 120 can transmit the dynamics data to the controller 130 in real time.
[0054] Controller 130 is configured to make decisions based on the image output from visual simulator 110 and input the decision results to dynamics simulator 120. For example, an operator can control the dynamics simulator via a keyboard, VR, or other means based on the image output from the visual simulator. Dynamics simulator 120 transmits motion data to visual simulator 110 in real time, and visual simulator 110 simultaneously renders the motion of the real-world 3D model.
[0055] In addition, when there is no need for dynamic simulation, the operator can also directly generate motion data through a keyboard, VR, etc. and transmit it to the visual simulator 110 independent of the dynamic simulator. The visual simulator 110 synchronously renders the motion of the real-scene three-dimensional model.
[0056] Alternatively, the controller 130 may be a keyboard, VR, mouse, or analog object. The analog object may be a simulated robot, simulated arm, or the like, and is used to simulate the movement of objects in the control application. For example, in the above scenario, the simulated robot may be controlled to pick up a water cup, or may be controlled to travel a certain distance.
[0057] In actual implementation, after the dynamics simulator 120 transmits the image data of the twin mathematical model to the controller 130, the controller 130 sends the end pose after simulated movement to the dynamics simulator, and the dynamics simulator transmits the camera pose, robot pose and object pose of the twin mathematical model after movement to the visual simulator 110, and the visual simulator 110 renders the movement of the real-scene three-dimensional model.
[0058] In addition, the dynamics simulator 120 can also simulate environmental forces (such as gravity, friction, magnetism, etc.) and / or contact forces after receiving instructions from the controller 130. For example, in the above scenario, the simulation control robot picks up a water cup, simulating the friction required to pick up the cup and the weight of the cup.
[0059] Please refer to Figure 2 , which shows a possible method flow chart of a robot simulation control method provided by an embodiment of the present application, the robot simulation control method can be used Figure 1 In the real-time scenario shown in Figure 2 As shown, the method includes:
[0060] Step 210: Acquire environmental information of the application site of the simulation robot, and generate a real-scene three-dimensional model based on the environmental information;
[0061] The environmental information includes one or more of the following: posture information and contact force information of the simulated robot at the application site, object position and contact force information of objects contacted by the simulated robot, and scene information surrounding the simulated robot. The posture information includes posture information of the simulated robot's body and limbs at the application site. In actual implementation, when acquiring environmental information, a calibration plate can be placed in the application scene, and environmental information including the calibration plate can be collected. Subsequently, the translation and rotation matrix of the current camera coordinate system can be obtained by identifying the calibration marks on the calibration plate.
[0062] After obtaining the environmental information in real time, a real-scene three-dimensional model corresponding to each piece of environmental information can be generated in real time.
[0063] In this embodiment, the real-time renderer of the real-scene 3D model can be used as a visual simulator to provide image samples for simulation training.
[0064] The real-scene 3D model generated by the visual simulator includes a panoramic real-scene 3D model and a non-panoramic real-scene 3D model that only retains the operated objects. The controller performs mixed training by rendering image data generated by the panoramic real-scene 3D model and the non-panoramic real-scene 3D model that only retains the operated objects.
[0065] When training the model, part of the data can be generated using the panoramic 3D model, while the other part of the training data can be generated using the non-panoramic 3D model. For example, 30% of the data can be generated using the panoramic 3D model, while 70% of the training data can be generated using the non-panoramic 3D model.
[0066] During training, the controller simulates contact between the simulated robot and the operating object, and the contact information generated is expressed in the form of sound, image, and / or force.
[0067] Step 220: Binding the real-scene 3D model and the twin mathematical model;
[0068] Binding the real-scene three-dimensional model and the twin mathematical model refers to binding the size information and coordinate information in the real-scene three-dimensional model and the twin mathematical model.
[0069] In actual implementation, this step may include:
[0070] First, obtaining a first coordinate set of the environmental information in a calibration plate coordinate system;
[0071] The environmental information may include multiple sets of images, each of which contains a calibration plate. This step can obtain the coordinate sets A{(x1, y1, z1), (x2, y2, z2), (x3, y3, z3)....} of the multiple camera groups in the calibration plate coordinate system.
[0072] Second, when constructing the Gaussian model, obtaining a second coordinate set including the environmental information of the calibration plate;
[0073] When building the Gaussian model, the camera pose of each frame is calculated through SFM, and the image coordinates including the calibration plate are obtained from each image to form the second coordinate set S{(x1, y1, z1), (x2, y2, z2), (x3, y3, z3)....}.
[0074] Third, converting the coordinates in the second coordinate set into a third coordinate set having the same size as the coordinates in the first coordinate set;
[0075] (1) calculating a first distance between adjacent coordinate points in the first coordinate system;
[0076] Calculate the distances La{l1,l2,l3....} between adjacent points in the coordinate set A.
[0077] (2) calculating a second distance between adjacent coordinate points in the second coordinate system;
[0078] The distance Ls{l1,l2,l3....} between adjacent coordinate points in the set S.
[0079] (3) calculating a distance proportional coefficient k based on the first distance and the second distance;
[0080] Calculate the distance ratios {k1, k2, k3, ....} of the corresponding distances in the set A and the distances in the set S, and calculate the proportional coefficient k of each distance ratio.
[0081] (4) Scale each coordinate point in the second coordinate set by k times to form the third coordinate set.
[0082] That is, after scaling the coordinates in the set S by k times, the result is Sk, which is the third coordinate set.
[0083] Fourth, obtaining the rotation and translation matrices of the first coordinate set and the third coordinate set;
[0084] (1) Calculate the centroid coordinates of the third coordinate set and the first coordinate set respectively;
[0085] The centroid coordinates are the average values of the coordinates on each coordinate axis.
[0086] (2) obtaining the coordinates of the third coordinate set and the first coordinate set after decentralization;
[0087] For the first coordinate set and the third coordinate set, calculate the coordinates after decentralization. That is, for the first coordinate set A and the third coordinate set Sk, the coordinates after decentralization are
[0088] (3) calculating a covariance matrix based on the decentralized third coordinate set and the first coordinate set;
[0089] The covariance matrix H is:
[0090] (4) Obtaining a rotation matrix according to the covariance matrix;
[0091] Perform singular value decomposition on the covariance matrix to obtain the rotation matrix R, H = U ∑ V T Where U and V are orthogonal matrices, Σ is a diagonal matrix. The rotation matrix R is composed of U and V.
[0092] (5) Obtain a translation vector based on the center of mass of the first coordinate set, the center of mass of the third coordinate set, and the rotation matrix.
[0093] The translation vector t is:
[0094] The rotation matrix and translation vector are combined to form the rotation and translation matrix Mk.
[0095] After that, the Gaussian model is bound to the mathematical model. The mathematical model is the model in the dynamics simulator. The mathematical model sets the model's mass, friction coefficient and other parameters. After the binding is completed, the Gaussian model in the Gaussian renderer and the mathematical model in the dynamics simulator are unified in the coordinate system.
[0096] The above is enough to build a complete simulation environment. Later, the simulation of the simulation robot can be realized through the interaction between the visual simulator and the dynamic simulator in the simulation environment.
[0097] Step 230: training a robot control model based on the real-scene three-dimensional model in different postures and the twin mathematical model.
[0098] Among them, the trained robot control model is used to control the real robot.
[0099] Optionally, the step of training the robot control model includes:
[0100] First, simulating different motions of the simulated robot in the twin mathematical model using a dynamics simulator;
[0101] Optionally, as described in the implementation scenario, combined with Figure 1 , through a controller such as VR / mouse, etc., the change of the end posture of the application site is controlled, and the end posture is sent to the dynamic simulator, which simulates the movement of the simulation robot.
[0102] Second, in the process of simulating the movement of the simulated robot, the dynamic simulator transmits the motion data to the visual simulator in real time, and the movement of the real-scene three-dimensional model is synchronously rendered by the visual simulator; training is performed according to the real-scene three-dimensional model in different postures and the twin mathematical model.
[0103] The dynamics simulator sends the camera pose, the simulated robot pose, and the object pose to the visual simulator in real time. The visual simulator then simultaneously renders the motion of the real-world 3D model. Of course, in actual implementation, the rendered results can be displayed in real time.
[0104] The steps of real-time data transmission between the dynamics simulator and the visual simulator include:
[0105] The position and pose of the mathematical model in the dynamics simulator is transmitted to the real-world 3D renderer via the ROS2 interface in real time. When objects in the dynamics simulator move, the corresponding objects in the real-world 3D renderer also move accordingly in real time. The position and pose of the camera on the robot in the dynamics simulator is then transmitted to the real-world 3D renderer via the ROS2 interface in real time. A rendering camera is added to the corresponding position in the real-world 3D renderer, and the camera image at the current position is published via ROS2 to achieve visual simulation.
[0106] Among them, real-time data transmission includes the robot body and limb motion data in the dynamic simulator and the motion data of the contact object transmitted to the display screen, with a time delay of no more than m milliseconds, and also includes real-time three-dimensional rendering of the real scene.
[0107] After obtaining the real-life 3D model and the twin mathematical model in different postures, training can be performed based on the two. In actual implementation, the simulation system generates environmental pattern templates corresponding to the limb movement by controlling the robot's limbs for training. Figure 1 The robot's limbs can be controlled using VR or a mouse, meaning that tasks can be completed using VR or a mouse. The camera image on the simulated robot and the robot's joint data in Mujoco, including motor rotation angles and torques, are saved as training datasets. In practice, n datasets can be collected for the same pose. n is an integer greater than or equal to 50.
[0108] After training the robot control model, the trained robot control model is tested and verified through simulation. After verification, it can be used to control a real robot in a real scene.
[0109] Please refer to Figure 3 , which shows a possible verification schematic, such as Figure 3 As shown, the Gaussian renderer sends image data to the trained robot control model. The trained robot control model can control the robot to move through the controller and simulate the movement in Mujoco. Mujoco sends the camera pose, robot pose and object pose to the Gaussian renderer, and the Gaussian renderer can render in real time.
[0110] In summary, by acquiring environmental information of the application site of the simulated robot, a real-scene three-dimensional model is generated based on the environmental information; the real-scene three-dimensional model and the twin mathematical model are bound; a robot control model is trained based on the real-scene three-dimensional model and the twin mathematical model in different postures, and the trained robot control model is used to control the robot; wherein the step of training the robot control model includes: simulating different movements of the simulated robot in the twin mathematical model through a dynamic simulator; during the process of simulating the movement of the simulated robot, the dynamic simulator transmits motion data in real time to a visual simulator, and the visual simulator synchronously renders the movement of the real-scene three-dimensional model; and the twin mathematical model is trained based on the image data rendered by the real-scene three-dimensional model in different postures and the dynamic data generated by the twin digital model. This solves the problem of high video memory requirements in the prior art and possible inapplicability in complex scenes. For any scene and object, only one model is needed to set the object placement arbitrarily. Through object placement and scene design, combined with simulated robot control or planning, a large amount of simulation data can be generated for real-world robot control, which is more efficient than previous methods.
[0111] Furthermore, in this application, the robot control model training process is completed in a simulated environment, reducing safety risks. Furthermore, the Gaussian model has a high degree of visual fidelity to real-world scenes, including small local features and texture characteristics, effectively resolving the discrepancy between real-world and simulated environments. This plays a key role in enabling the robot to perform delicate tasks.
[0112] An embodiment of the present application also provides a model binding method for binding a real-scene three-dimensional model and a twin mathematical model. The specific method is as described in step 220 of the above embodiment and will not be repeated here.
[0113] The present application also provides a robot simulation control device, which includes a memory and a processor. The memory stores at least one program instruction, and the processor implements the above-mentioned method by loading and executing the at least one program instruction.
[0114] The present application also provides a computer storage medium, in which at least one program instruction is stored. The at least one program instruction is loaded and executed by a processor to implement the method described above.
[0115] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0116] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.
Claims
1. A robot simulation control method, characterized in that: The method comprises: Acquire environmental information of the application site of the simulation robot, and generate a real-scene three-dimensional model based on the environmental information; Binding the real-scene three-dimensional model and the twin mathematical model; Training a robot control model based on the real-scene three-dimensional model in different postures and the twin mathematical model, wherein the trained robot control model is used for controlling the robot; The steps of training the robot control model include: simulating different motions of the simulated robot in the twin mathematical model by a dynamics simulator; During the process of simulating the movement of the simulated robot, the dynamic simulator transmits the motion data to the visual simulator in real time, and the movement of the real-scene three-dimensional model is synchronously rendered by the visual simulator; the twin mathematical model is trained based on the image data rendered by the real-scene three-dimensional model in different postures and the dynamic data generated by the twin digital model.
2. A simulation system, characterized in that: include: A visual simulator for generating realistic 3D models, Dynamics simulator for generating twin mathematical models, A controller is used to make judgments and decisions based on the image output generated by the visual simulator and / or the contact force information generated by the dynamic simulator and input the decision results to the dynamic simulator; the dynamic simulator transmits motion data to the visual simulator in real time, and the visual simulator synchronously renders the motion of the real-life three-dimensional model.
3. The simulation system according to claim 2, wherein: The real-scene 3D model generated by the visual simulator includes a panoramic real-scene 3D model and a non-panoramic real-scene 3D model that only retains the operated objects. The controller performs mixed training by rendering image data generated by the panoramic real-scene 3D model and the non-panoramic real-scene 3D model that only retains the operated objects.
4. The simulation system according to claim 2, wherein: During training, the controller simulates contact between the simulated robot and the operating object, and the contact information generated is expressed in the form of sound, image, and / or force.
5. A model binding method for binding a real-scene three-dimensional model and a twin mathematical model, characterized in that: The following steps are involved: Bind the size information and coordinate information in the real-scene three-dimensional model and the twin mathematical model.
6. The method according to claim 5, characterized in that Binding the size information and coordinate information in the real-scene three-dimensional model and the twin mathematical model includes: Obtaining a first coordinate set of the environmental information in a calibration plate coordinate system; When constructing the Gaussian model, obtaining a second coordinate set including the environmental information of the calibration plate; Converting the coordinates in the second coordinate set to a third coordinate set that has the same size as the coordinates in the first coordinate set; Obtaining a rotation and translation matrix of the first coordinate set and the third coordinate set; The real-scene three-dimensional model and the twin mathematical model are size-bound according to the rotation and translation matrix.
7. The method according to claim 6, characterized in that The converting the coordinates in the second coordinate set into a third coordinate set having the same size as the coordinates in the first coordinate set comprises: Calculating a first distance between adjacent coordinate points in the first coordinate system; Calculating a second distance between adjacent coordinate points in the second coordinate system; Calculating a distance proportional coefficient k according to the first distance and the second distance; Each coordinate point in the second coordinate set is scaled k times to form the third coordinate set.
8. The method according to claim 7, characterized in that The obtaining of the rotational and translational moments of the first coordinate set and the third coordinate set includes: Calculating the centroid coordinates of the third coordinate set and the first coordinate set respectively; Obtaining coordinates of the third coordinate set and the first coordinate set after decentralization; Calculating a covariance matrix based on the decentralized third coordinate set and the first coordinate set; Obtaining a rotation matrix according to the covariance matrix; A translation vector is obtained according to the center of mass of the first coordinate set, the center of mass of the third coordinate set, and the rotation matrix.
9. The method according to claim 6, characterized in that The environmental information includes posture information and / or contact force information of the simulated robot at the application site, object position and / or contact force information of an object contacted by the simulated robot, and scene information around the simulated robot.
10. The system according to claim 2, wherein: The controller includes at least one of a keyboard, VR, mouse, and an analog object.