Method, device, equipment, medium and program product for simulating robot motion

Abstract

This application provides a method, apparatus, device, medium, and program product for simulating robot motion, relating to the field of robot simulation technology. The method includes: constructing a simulated robot based on an editable joint information tree; utilizing the simulated robot's multimodal sensors to extract features from the simulation environment and obtain task state data of the simulated robot's task to be performed, thereby determining the simulated robot's execution action information. This application, by importing the editable joint information tree into a game engine, enables customized modification and creation of the simulated robot within the simulation environment, improving the flexibility of robot motion simulation.

Description

Technical Field

[0001] This application relates to the field of robot simulation technology, and in particular to a method, apparatus, equipment, medium and program product for simulating robot motion. Background Technology

[0002] With the rapid development of robotics technology, its application areas have expanded from traditional industrial manufacturing to various complex scenarios such as inspection and testing, medical care, and services. Before deploying humanoid robots in these unstructured and dynamically changing environments, thorough testing and verification are necessary. Direct debugging and testing in real-world environments is not only costly and time-consuming, but also carries safety risks due to immature algorithms or unknown environmental factors that could damage the robot or equipment. Therefore, it is necessary to develop high-fidelity simulation environments to simulate robot behavior, test control algorithms, and verify task feasibility in a virtual world.

[0003] Existing robot motion simulations are mostly based on graphics engines or dedicated simulation platforms. For example, professional robot simulation software uses precise 3D modeling to construct high-precision scenes such as industrial laboratories and integrates control systems such as the Robot Operating System (ROS) and MoveIt (a robotic arm motion planning framework) to achieve high-precision physical simulation and task verification of humanoid robots in simulated environments. In addition, some solutions utilize commercial game engines such as Unreal Engine to build robot simulation models and achieve remote control and status visualization of the physical robot through network communication. Other solutions are based on the Unity engine, combining mathematical modeling and scene modeling to build comprehensive mobile robot testing systems that include dynamics, sensors, and scene simulation for rapid testing and iteration of intelligent algorithms.

[0004] Existing robot motion simulations rely on specific commercial software platforms. While these platforms offer high levels of functional integration, users have limited customization and expansion capabilities for core functions. Furthermore, the use of closed-source commercial game engines, despite their rich plugin ecosystems, restricts the modification of their core code. This presents significant obstacles for researchers seeking deep customization of the underlying physics engine or rendering pipeline to meet specific simulation needs (such as simulation of special sensors or extreme physical environments), thus limiting the flexibility and scalability of the simulation system. Summary of the Invention

[0005] This application provides a method, apparatus, device, medium, and program product for simulating robot motion, in order to solve the shortcomings of inflexible robot motion simulation in the prior art and improve the flexibility of robot motion simulation.

[0006] Firstly, this application provides a method for simulating robot motion, comprising: A simulated robot is constructed based on an editable joint information tree. The joint information tree includes a root node, at least one joint and its motion parameters. Each joint is connected to the root node through a link. Each joint is connected to a parent link and a child link. The multimodal sensors of the simulated robot are used to extract features from the simulation environment and obtain the task status data of the simulated robot's task to be executed. Based on task status data, the execution action information of the simulated robot is determined in order to simulate the robot's motion.

[0007] In one embodiment, constructing a simulated robot based on an editable joint information tree of the robot includes: Based on the robot's joint information tree, connection information, visual information, and motion information of multiple child nodes are created. Child nodes include joints, parent links of joints, and child links of joints. The connection information of a child node includes the connection information between the parent link and child links of the joint within the child node, as well as the connection information between the child node and other child nodes. The visual information of a child node includes its appearance structure and position. The motion information of a child node includes its position attributes and collision response. Position attributes are used to adjust the rotation or position of the joint to adjust its initial posture. The collision response is used to determine the response when a child node collides with a simulated collider. Based on the parameter information, visual information, and motion information of each sub-node, the state of each sub-node is simulated to obtain a simulated robot.

[0008] In one embodiment, the multimodal sensor includes a colored camera sensor and a lidar sensor. The colored camera sensor is created based on the vision node of the Godot engine, and the lidar sensor is created based on the ray projection node of the Godot engine. The multimodal sensor of the simulated robot is invoked to extract features from the simulation environment and obtain the task status data of the simulated robot's task to be executed, including: Based on the viewport node of the colored camera sensor, feature extraction is performed on the simulation environment to obtain the colored image associated with the task to be performed; At least one RayCast3D node based on a lidar sensor is used to simulate at least one laser beam; the code based on the lidar sensor controls the rotation of the RayCast3D node to achieve scanning within a set angle range, and obtains scanning data in the simulation environment. The scanning data includes the distance of the RayCast3D node to each simulated collider in the simulation environment. Task status data is obtained based on colored images and scanned data.

[0009] In one embodiment, the simulation environment includes simulated colliders and a simulated scene, and the simulation environment is determined based on the following: In the 3D creation software of the game engine, each simulated collision body is obtained based on the 3D shape data, material data, surface texture data and structural information of each object; the structural information of the object includes the category label of the combined structure object or the category label of the single structure object, as well as the component connection relationship of the combined structure object. In 3D creation software, the scene in which the robot is located is modeled, and the positions and orientations of objects are arranged to obtain the simulation scene and its root scene file; Based on the names, positions, and orientations of the objects in the root scene file, each simulated collider is imported into the simulation scene to obtain the simulation environment.

[0010] In one embodiment, determining the execution action information of the simulated robot based on task state data includes: The motion of the simulated robot is simulated based on the execution action information at the current moment, so as to obtain the completion degree of the task to be executed and the task status data at the current moment; Based on the completion status of the tasks to be performed, evaluate the post-execution status of the execution action information at the current moment, and obtain the reward score at the current moment; The driving strategy decision-making agent calculates the task state data and reward score at the current moment to determine the next action information of the simulated robot; The next moment's execution action information is used as the current moment's execution action information. The simulation robot's motion is performed based on the current moment's execution action information. The steps of obtaining the completion status of the task to be executed and the current moment's task status data are carried out until the task to be executed is completed.

[0011] In one embodiment, the multimodal sensor further includes a depth camera sensor for acquiring colored images. Based on the viewport node of the depth camera sensor, acquire the surface texture image of the simulated collider associated with the task to be performed; Perform fisheye distortion calculation on the surface texture image of the simulated collider to obtain the distorted surface texture image; Convert the distorted surface texture image into a colored image.

[0012] Secondly, this application also provides a simulation device for robot motion, comprising: The building module is used to construct a simulated robot based on an editable joint information tree. The joint information tree includes a root node, at least one joint and its motion parameters, and each joint is connected to the root node through links. Each joint is connected to a parent link and a child link. The task status data acquisition module is used to call the multimodal sensors of the simulation robot to extract features from the simulation environment and obtain the task status data of the simulation robot's task to be executed. The motion module is used to determine the execution action information of the simulated robot based on task status data, so as to simulate the robot's motion.

[0013] Thirdly, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a simulation method for any of the robot movements described above.

[0014] Fourthly, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a simulation method for robot motion as described above.

[0015] Fifthly, this application also provides a game engine, including a computer program product, including a computer program, which, when executed by a processor, implements a simulation method for any of the robot movements described above.

[0016] The robot motion simulation method, apparatus, equipment, medium, and program products provided in this application construct a simulated robot based on an editable joint information tree. The joint information tree includes a root node, at least one joint, and motion parameters for each joint. Each joint is connected to the root node via links; each joint is connected to a parent link and a child link. The application utilizes the robot's multimodal sensors to extract features from the simulation environment and obtain task state data for the robot's task. Based on the task state data, it determines the robot's execution action information to simulate the robot's motion. This application, by importing the editable joint information tree into a game engine, enables customized modification and creation of the simulated robot within the simulation environment, improving the flexibility of robot motion simulation. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the robot motion simulation method provided in this application.

[0019] Figure 2 This is a flowchart illustrating the process of generating the simulation environment provided in this application.

[0020] Figure 3 This is a flowchart illustrating the process of determining the execution action information provided in this application.

[0021] Figure 4 This is a schematic diagram of the robot motion simulation device provided in this application.

[0022] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] The Godot engine is a powerful, cross-platform, completely free and open-source game engine. It uses a very developer-friendly, permissive license. Products developed using the Godot engine (both commercial and non-commercial) belong entirely to the developers, without the need to pay royalties or licensing fees, and developers are free to modify the Godot engine's source code.

[0025] The following is combined Figures 1-5 This application describes the simulation method, apparatus, and electronic equipment for robot motion.

[0026] Figure 1 This is a flowchart illustrating the robot motion simulation method provided in this application, as shown below. Figure 1 As shown, the simulation method for robot motion includes steps S100 to S300, and the specific steps are as follows.

[0027] S100: Building a simulated robot based on an editable joint information tree.

[0028] The joint information tree includes a root node, at least one joint and its motion parameters, and each joint is connected to the root node through links; each joint is connected to a parent link and a child link.

[0029] The execution entity of this application embodiment includes a game engine, and servers, computers, etc., that drive the game engine. Game engines include the Godot engine, Unreal Engine, Unity, etc. This application embodiment uses the Godot engine as an example for illustration.

[0030] Joints include the Generic 6 Degrees Of Freedom Joint (Generic6DOFJoint).

[0031] This plugin in the Godot engine imports Unified Robot Description Format (URDF) files for robots. It parses the imported URDF files and constructs a joint information tree for the robot. This tree displays the robot's links and joints, enabling visual manipulation of the joints and facilitating robot control.

[0032] Links consist of parent links and child links. The joint information tree includes multiple joints of the robot and a root node. The root node represents the robot's torso. Each joint connects to a parent link and a child link. The parent link is the link closest to the root node. The child link is the link furthest from the root node. All joints are connected to the root node via links. Alternatively, all joints can be indexed to the root node via links. For example, the elbow joint's child link is the forearm, and the elbow joint's parent link is the upper arm; the robot's elbow joint is connected to the root node via its parent link (upper arm). The robot's wrist joint's parent link is the forearm, and the wrist joint's child links are the fingers. The wrist joint is connected to the root node via its parent link (forearm) and the elbow joint's parent link (upper arm).

[0033] The joint information tree also includes motion parameters for each joint, such as the joint's motion mode and range of motion.

[0034] Based on the robot's joint information tree, a robot tree node class (RobotRigidbody) is constructed. Based on the robot tree node class (RobotRigidbody), a simulated robot and its functions are created.

[0035] Furthermore, users can write the functions of the simulation robot into its scripts. The functions of the simulation robot are as follows. Through these functions, the simulation robot can perform at least one action.

[0036] 1. Based on the joint index, retrieve the creation information of the joint's child nodes. Child nodes include the joint, the joint's parent link, and the joint's child links. For example, when the joint is the elbow joint, the child nodes include the elbow joint, the upper arm connected to the elbow joint (parent link), and the forearm connected to the elbow joint (child link).

[0037] 2. Obtain the visual information carried by the child nodes of the joint, and the number of visual information items. A single joint may correspond to multiple visual information items. For example, when the joint is the elbow joint, the visual information includes a visual image of an arm with the upper arm on top and the forearm below (an image of the arm hanging naturally); and a visual image of an arm with the forearm on top and the upper arm below (an image of a raised hand gesture).

[0038] 3. Obtain the number of joints of the simulated robot.

[0039] 4. Obtain the number of collision points where the simulated robot and the simulated collider (or simulated object) come into contact.

[0040] 5. Obtain the local coordinates of the collision point in the simulated robot.

[0041] 6. Obtain the normal of the collision point.

[0042] 7. Obtain the collision impulse.

[0043] 8. Obtain the joint number of the collision point in the simulated robot.

[0044] 9. Obtain the resource identifier (RID) of the simulated collider.

[0045] 10. Obtain the global coordinates of the collision point in the simulation scene.

[0046] 11. Obtain the object number of the simulated collider.

[0047] 12. Obtain the shape sequence number of the simulated collision body.

[0048] 13. Obtain the velocity of the collision point of the simulated robot.

[0049] 14. Obtain the link number corresponding to the collision point in the simulated robot.

[0050] 15. Obtain the simulated collision object.

[0051] 16. Obtain the names of the sub-links of the joint.

[0052] 17. Obtain the transformation of the joint's coordinate system relative to the root node's coordinate system.

[0053] 18. Set the transformation of the joint's coordinate system relative to the root node's coordinate system.

[0054] 19. Speed ​​of obtaining the root node.

[0055] 20. Obtain the state of the link, including its pose, linear velocity in world coordinates, angular velocity, etc.

[0056] 21. Obtain the state of the joint, including pose, velocity (angular velocity for rotary joints; linear velocity for translational joints), and joint torque.

[0057] 22. Obtain joint information, including joint name, joint type, damping coefficient, friction coefficient, upper and lower limits, maximum torque that the joint can withstand, maximum speed, direction of motion axis, parent link index, etc.

[0058] 23. Select the motion mode for controlling the joints of the robot simulation. Use different control modes to drive the joints, such as position control, speed control, torque control, etc.

[0059] 24. Reset the state of a joint, setting its position and velocity to desired values ​​without considering dynamics and constraints.

[0060] 25. Calculate the angles and positions of each joint of the simulated robot, i.e., perform inverse kinematics solution.

[0061] The calculation formula for solving inverse kinematics is as follows.

[0062] ; in, For the increment of the joint angle, This is the pseudo-inverse of the Jacobian matrix. This refers to the pose error of the end effector of the sub-link of the joint. It is the inverse of the Jacobian matrix. The damping coefficient is... The identity matrix is ​​determined by the number of joints in the simulated robot. It is a Jacobian matrix.

[0063] In the RobotRigidbody tree node class, when implementing joint constraints using Generic6DOFJoint, it is necessary to convert the dynamic labels in the robot's URDF file into constraint parameters for the physics engine (Bullet). Dynamic labels include damping coefficients, friction coefficients, etc.

[0064] Import the simulated robot and its functions into the simulation environment.

[0065] S200: Calls the multimodal sensors of the simulated robot to extract features from the simulation environment and obtains the task status data of the simulated robot's task to be executed.

[0066] Multimodal sensors are sensing devices or systems capable of simultaneously acquiring and processing two or more different types (modal) of information. Multimodality refers to multiple sensing methods. Multimodal sensors are used to simulate the senses of robotic robots.

[0067] The simulation environment is a replica of the robot's actual environment. For example, the simulation environment includes the room where the robot is located, the road where the robot is located, etc.

[0068] The multimodal sensors of the simulated robot are used to extract features from the simulation environment and obtain task status data for the task to be performed. For example, if the task to be performed is for the simulated robot to pick up a cup from a table in the simulation environment, then the multimodal sensors are used to extract features from the simulation environment to obtain data such as the shape, position, distance, and material of the cup, thus obtaining the task status data.

[0069] This application uses multimodal sensors to simulate the sensory abilities of a robot in a simulated environment, enabling accurate and efficient extraction of task status data, thereby improving the accuracy and efficiency of robot motion simulation.

[0070] S300: Based on task status data, determine the execution action information of the simulated robot in order to simulate the robot's motion.

[0071] By combining task status data, the execution action information of the simulated robot is determined. The execution action information includes the motion parameters of the executed sub-links (including motion state, motion speed, angular velocity, motion mode, and distance traveled), and the information of the joints corresponding to the executed sub-links.

[0072] For example, the task to be performed is to pick up a cup from the table. The sub-links to be performed are the fingers, forearm, and upper arm. The information on the performed motion includes the degree of finger flexion, the rotation angle of the upper arm, the rotation angle of the forearm, the rotation method of the upper arm, and the rotation method of the forearm.

[0073] The simulated robot moves according to the information of the executed actions, thereby realizing the simulation of robot motion.

[0074] The robot motion simulation method provided in this application constructs a simulated robot based on an editable joint information tree. The joint information tree includes a root node, at least one joint, and its motion parameters. Each joint is connected to the root node via links; each joint is connected to a parent link and a child link. The method utilizes the robot's multimodal sensors to extract features from the simulation environment and obtain task state data for the robot's task. Based on the task state data, the method determines the robot's execution action information to simulate its motion. This application, by importing the editable joint information tree into a game engine, enables customized modification and creation of the simulated robot within the simulation environment, improving the flexibility of robot motion simulation.

[0075] Based on the above embodiments, constructing a simulated robot based on the robot's editable joint information tree includes the following steps: Based on the robot's joint information tree, connection information, visual information, and motion information of multiple child nodes are created. Child nodes include joints, parent links of joints, and child links of joints. The connection information of a child node includes the connection information between the parent link and child links of the joint within the child node, as well as the connection information between the child node and other child nodes. The visual information of a child node includes its appearance structure and position. The motion information of a child node includes its position attributes and collision response. Position attributes are used to adjust the rotation or position of the joint to adjust its initial posture. The collision response is used to determine the response when a child node collides with a simulated collider. Based on the parameter information, visual information, and motion information of each sub-node, the state of each sub-node is simulated to obtain a simulated robot.

[0076] Based on the robot's joint information tree, we create connection information (RobotBodyNodeCreationInfo), visual information (RobotBodyNodeVisualInfo), and motion information (RobotBodyNode) for multiple child nodes of the simulated robot.

[0077] The connection information of child nodes includes the connection information between the parent links and child links of the joints within the child node, as well as the connection information between the child node and other child nodes. The connection information and visual information of child nodes can be imported into the robot's script. When the connection information and visual information of child nodes are needed, the robot's script is invoked to query the connection information and visual information of the child nodes.

[0078] The motion information of a child node includes its position attributes and collision response. By selecting a child node of the simulated robot, its motion information can be directly modified.

[0079] When a child node is selected, modifying its position attributes allows you to observe changes in the rotation or position of the joint corresponding to that child node. This function can be used to modify the initial posture of the joints in a simulated robot.

[0080] When a child node is selected, its collision response is modified. The collision response includes a collision group and a collision mask. The collision group corresponds to the collision layer in the Godot engine's bullet and represents the collision response between a child node of the simulated robot that generates contact / collision and the simulated collider. The collision mask is the encoding of the collision group.

[0081] By using the connection information, visual information, and motion information of each sub-node, the state of each sub-node is simulated to obtain a simulated robot.

[0082] This application enables the editing of static data and states of each sub-node of a simulated robot by constructing connection information, visual information, and motion information of sub-nodes. This, in turn, allows for the custom editing and modification of the initial posture of each joint of the simulated robot, thereby improving the flexibility of the simulated robot.

[0083] Based on the above embodiments, the simulation environment includes simulated collision bodies and simulated scenarios, and the simulation environment is determined in the following manner: In the 3D creation software of the game engine, each simulated collision body is obtained based on the 3D shape data, material data, surface texture data and structural information of each object; the structural information of the object includes the category label of the combined structure object or the category label of the single structure object, as well as the component connection relationship of the combined structure object. In 3D creation software, the scene in which the robot is located is modeled, and the positions and orientations of objects are arranged to obtain the simulation scene and its root scene file; Based on the names, positions, and orientations of the objects in the root scene file, each simulated collider is imported into the simulation scene to obtain the simulation environment.

[0084] Building a simulation environment includes workflows for constructing simulation colliders and workflows for generating the simulation environment. Simulation colliders are used to simulate objects in the robot's actual environment. For example, the robot's actual environment is a living room. Simulation colliders include sofas, tables, cups on the tables, refrigerators, etc., in the simulated living room (simulation scene).

[0085] like Figure 2 As shown, the workflow for generating the simulation environment includes the following steps.

[0086] Determine whether manual modeling is required. If manual modeling is required, create a mesh model of the object, and further generate the object's 3D shape data, material data, surface texture data, and structural information. If manual modeling is not required (non-manual modeling), export the object's 3D shape data, material data, surface texture data, and structural information from an open-source object simulation model database.

[0087] Verify the object's files (including 3D shape data, material data, surface texture data, and structural information), and filter out valid files. Construct a simulated collider based on the valid files and generate a standardized document for the simulated collider. Import the simulated collider into the simulation scene.

[0088] Taking manual modeling as an example, the workflow for constructing a simulated collision body includes the following steps.

[0089] 1.1 Create a mesh model of the item, obtaining its 3D shape data, material data, surface texture data, and structural information. The 3D shape data includes object (obj) files, which describe the item's 3D geometry. One item can correspond to one or more obj files.

[0090] Material data includes Material Library Files (mtl). Material Library Files are used to describe the materials of an item.

[0091] Surface texture data includes texture files that describe the surface texture and patterns of an object. For example, for a mineral water bottle made of plastic, the surface texture data might be a red logo, or a green logo, etc.

[0092] 1.2 Furthermore, the structural information of an item also includes its orientation, size (scaling ratio), category label, and URDF file. The category label includes composite structure items and single structure items. Composite structure items are items that can be disassembled into multiple connecting parts. For example, a refrigerator is a composite structure item. A refrigerator can be disassembled into doors, drawers, a rotating shaft connecting the doors, and a cabinet connected to the rotating shaft. Single structure items are items that cannot be disassembled, such as a soccer ball.

[0093] When an item is categorized as a composite structure item, its structural information also includes a URDF file. The URDF file stores the link / joint relationships between the various connecting components of the composite structure item. However, when an item is categorized as a single structure item, a URDF file is not required for its structural information. Retrieve the item's structural information.

[0094] 1.3 Importing the Object's Mesh Model. 3D creation software includes Blender. Import the object's .obj file into Blender to import the object's mesh model. Set the orientation of the object's mesh model's coordinate system, for example, Z-axis upwards and X-axis forwards.

[0095] 1.4 Constructing Collision Models (Simulated Collision Bodies). In Blender, the Voxelized Hierarchical Approximate Convex Decomposition (V-HACD) tool is used to generate collision models for the mesh models of each object, resulting in simulated collision bodies. Collision models can simulate the responses to collisions or contact between objects or between objects and the robot, such as deformation during a collision.

[0096] 1.5 Generate standardized information for the simulated collider. Standardized information includes the bounding box size of the simulated collider and the materials used for each part of the simulated collider.

[0097] 1.6 Import the URDF information of the item. Import the URDF file of the composite item into Blender.

[0098] 1.7 Use Blender to generate the front view, rear view, left view, right view, top view, and bottom view of the simulated collider.

[0099] The workflow for generating a simulation environment includes the following steps.

[0100] 2.1 In Blender, model the scene in which the robot is located and arrange the positions and orientations of various objects to obtain the simulation scene.

[0101] 2.2 Export the root scene file of the simulation scene. The root scene file contains the name, location, and orientation of each item.

[0102] 2.3. Based on the object's name, locate the 3D shape data (obj file), material data, and surface texture data of the simulated collider. Simultaneously, import the URDF file of the simulated collider using the URDF plugin to import it into the simulation scene, thus obtaining the simulation environment.

[0103] This application supports building simulation environments using URDF or JSON format description files. It also supports storing the scene layout in URDF or JSON format after the simulation environment is built.

[0104] This application obtains simulated colliders by analyzing the 3D shape data, material data, surface texture data, and structural information of objects. It allows for customized settings and modifications of the simulated colliders, improving their flexibility. Through the root scene file, the simulated colliders can be accurately and completely indexed, ensuring error-free import into the simulation scene.

[0105] Based on the above embodiments, the multimodal sensor includes a colored camera sensor and a lidar sensor. The colored camera sensor is created based on the vision node of the Godot engine, and the lidar sensor is created based on the ray projection node of the Godot engine. The process of calling the multimodal sensor of the simulated robot to extract features from the simulation environment and obtain the task status data of the simulated robot's task to be executed includes the following steps: Based on the viewport node of the colored camera sensor, feature extraction is performed on the simulation environment to obtain the colored image associated with the task to be performed; At least one RayCast3D node based on a lidar sensor is used to simulate at least one laser beam; the code based on the lidar sensor controls the rotation of the RayCast3D node to achieve scanning within a set angle range, and obtains scanning data in the simulation environment. The scanning data includes the distance of the RayCast3D node to each simulated collider in the simulation environment. Task status data is obtained based on colored images and scanned data.

[0106] Multimodal sensors include color camera sensors and LiDAR sensors. Color camera sensors include red-green-blue (RGB) cameras.

[0107] Create a viewport node in the Godot engine, set the width and height of the viewport, and obtain the RGB camera (color camera sensor).

[0108] An RGB camera can acquire surface texture images of simulated colliders associated with the task to be performed in a simulated environment, and convert them into GRB8 pixel format (FORMAT RGB8) images to obtain colored images. For example, when the task to be performed is to pick up a cup on a table, the RGB camera can acquire the surface texture data of the cup on the table to obtain a colored image associated with the task to be performed.

[0109] Laser beams are simulated using RayCast 3D nodes in the Godot engine. Multiple RayCast 3D nodes simulate multi-line lasers. The rotation of the RayCast 3D nodes is controlled by code from a lidar sensor to achieve scanning within a set angle range (e.g., 360°). During scanning, the distances between the RayCast 3D nodes and various simulated colliders in the simulation environment are acquired, resulting in scan data. The scan data is output as an array and expanded into a point cloud.

[0110] Based on the obtained colored images and scan data, the task status data of the simulation robot is obtained.

[0111] This application acquires colored images associated with the task to be performed using a colored camera sensor, enabling it to capture the surface texture patterns of the simulated collision object. By scanning the simulation environment using RayCast3D nodes, it vividly simulates the robot's eye scanning its surroundings.

[0112] Based on the above embodiments, the multimodal sensor further includes a depth camera sensor, which is used to acquire colored images: Based on the viewport node of the depth camera sensor, acquire the surface texture image of the simulated collider associated with the task to be performed; Perform fisheye distortion calculation on the surface texture image of the simulated collider to obtain the distorted surface texture image; Convert the distorted surface texture image into a colored image.

[0113] Create a viewport node for the depth camera sensor, setting its width and height. Use this viewport node to acquire the surface texture image of the simulated collider associated with the task to be performed. Perform fisheye distortion calculation on the surface texture image using the depth camera sensor to obtain the distorted surface texture image. The formula for fisheye distortion calculation is as follows.

[0114] ; in, The radius (in pixels) of the distorted surface texture image. Focal length (pixels). It is the incident angle or incident radian (the angle between the incident ray and the optical axis). , , , and This represents the fisheye distortion coefficient.

[0115] The distorted surface texture image is converted into a FORMAT RGB8 image to obtain a colored image.

[0116] This application improves the accuracy of acquiring colored images by performing fisheye distortion calculations on the surface texture image of the simulated collision object, which can more vividly simulate the image of the simulated collision object seen by the simulated robot.

[0117] Based on the above embodiments, the execution action information of the simulated robot is determined based on the task status data, including the following steps: The motion of the simulated robot is simulated based on the execution action information at the current moment, so as to obtain the completion degree of the task to be executed and the task status data at the current moment; Based on the completion status of the tasks to be performed, evaluate the post-execution status of the execution action information at the current moment, and obtain the reward score at the current moment; The driving strategy decision-making agent calculates the task state data and reward score at the current moment to determine the next action information of the simulated robot; The next moment's execution action information is used as the current moment's execution action information. The simulation robot's motion is performed based on the current moment's execution action information. The steps of obtaining the completion status of the task to be executed and the current moment's task status data are carried out until the task to be executed is completed.

[0118] like Figure 3 As shown, a robot environment is created in the server driving the game engine. The robot environment sends task status data (including colored images and scan data) to the policy decision-making agent via socket communication. The policy decision-making agent, acting as the game engine's backend, determines the action information to be executed.

[0119] The policy decision-making agent is trained through reinforcement learning. It reads task state data, feeds it into the algorithm model, and outputs action information. This action information is then sent to the robot environment within the game engine. The robot environment controls the movement of the simulated robot based on this action information.

[0120] Furthermore, a reward function is constructed based on the degree of completion of the task to be performed. This reward function generates reward scores based on the degree of completion of the task. For example, if the task is to pick up a cup from a table, when the robot's hand touches the cup, the completion level is 30%, and the reward score is 30 points. When the robot pinches the cup, the completion level is 60%, and the reward score is 60 points. When the robot picks up the cup, the completion level is 100%, and the reward score is 100 points. When the robot crushes the cup, the completion level is -100%, and the reward score is -100 points. When the robot knocks over the cup after touching it, the completion level is -30%, and the reward score is -30 points.

[0121] The robot environment simulates the robot's movement based on the current action information, obtaining the completion rate of the task and the current task status data. For example, based on the current action information, the robot simulates contacting a cup, and the completion rate is obtained as 30%. Using the robot's color camera sensor, depth camera sensor, and LiDAR sensor, the simulation environment and the robot's state after the current action information is executed are scanned and features are extracted to obtain the current task status data.

[0122] The robot's environment evaluates the state of the executed actions based on the completion status of the task to be performed, and obtains the reward score for the current moment.

[0123] The robot's environment sends the current task status data and the current reward score to the policy decision-making agent.

[0124] The strategic decision-making agent determines the next action information for the simulated robot based on the current task status data and the current reward score, and sends the next action information to the robot's environment.

[0125] The robot environment takes the execution action information of the next moment as the execution action information of the current moment, and repeatedly executes the steps of "simulating the movement of the simulation robot based on the execution action information of the current moment, obtaining the completion degree of the task to be executed and the task status data of the next moment" until the simulation robot completes the task to be executed (the completion degree of the task to be executed is 100%).

[0126] Furthermore, if the current time is the initial time, the execution action information at the initial time is determined based on the task state data at the initial time.

[0127] This application, by setting reward scores, enables the evaluation of the post-execution status of the action information at each moment, allowing for timely adjustment of deviations and errors in the execution process of the task to be executed, thereby improving the accuracy of the simulated robot's task execution.

[0128] The simulation device for robot motion provided in this application is described below. The simulation device for robot motion described below can be referred to in correspondence with the simulation method for robot motion described above.

[0129] like Figure 4 As shown, the robot motion simulation device includes: Module 401 is used to build a simulated robot based on an editable joint information tree. The joint information tree includes a root node, at least one joint and its motion parameters, and each joint is connected to the root node through a link. Each joint is connected to a parent link and a child link. The task status data acquisition module 402 is used to call the multimodal sensors of the simulation robot to extract features from the simulation environment and acquire the task status data of the simulation robot's task to be executed. The motion module 403 is used to determine the execution action information of the simulated robot based on the task state data, so as to simulate the robot's motion.

[0130] The robot motion simulation device provided in this application constructs a simulated robot based on an editable joint information tree. The joint information tree includes a root node, at least one joint, and its motion parameters. Each joint is connected to the root node via links; each joint is connected to a parent link and a child link. The device utilizes the robot's multimodal sensors to extract features from the simulation environment and obtain task state data for the robot's task. Based on the task state data, it determines the robot's execution action information to simulate the robot's motion. This application, by importing the editable joint information tree into a game engine, enables customized modification and creation of the simulated robot within the simulation environment, improving the flexibility of robot motion simulation.

[0131] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0132] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a robot motion simulation method. The method includes: constructing a simulated robot based on an editable joint information tree of the robot, the joint information tree including a root node, at least one joint and its motion parameters, each joint being connected to the root node through links; each joint being connected to a parent link and a child link; calling the multimodal sensors of the simulated robot to extract features from the simulation environment and obtain task state data of the simulated robot's task to be executed; and determining the execution action information of the simulated robot based on the task state data to simulate the robot's motion.

[0133] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0134] On the other hand, this application also provides a computer program product, wherein the game engine includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the robot motion simulation method provided by the above methods. The method includes: constructing a simulated robot based on an editable joint information tree of the robot, wherein the joint information tree includes a root node, at least one joint and its motion parameters, and each joint is connected to the root node through a link; each joint is connected to a parent link and a child link; calling the multimodal sensors of the simulated robot to extract features from the simulation environment and obtain task state data of the task to be executed by the simulated robot; and determining the execution action information of the simulated robot based on the task state data to simulate the motion of the robot.

[0135] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the simulation method for robot motion provided by the above methods. The method includes: constructing a simulated robot based on an editable joint information tree of the robot. The joint information tree includes a root node, at least one joint and its motion parameters, and each joint is connected to the root node through a link; each joint is connected to a parent link and a child link; calling the multimodal sensors of the simulated robot to extract features from the simulation environment and obtain task state data of the task to be performed by the simulated robot; and determining the execution action information of the simulated robot based on the task state data to simulate the motion of the robot.

[0136] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for simulating robot motion, characterized in that, include: A simulated robot is constructed based on an editable joint information tree. The joint information tree includes a root node, at least one joint and its motion parameters, and each joint is connected to the root node through a link. Each joint is connected to a parent link and a child link. The multimodal sensors of the simulated robot are invoked to extract features from the simulation environment and obtain the task status data of the simulated robot's task to be executed. Based on the task status data, the execution action information of the simulated robot is determined in order to simulate the robot's motion.

2. The simulation method for robot motion according to claim 1, characterized in that, The robot-based editable joint information tree construction simulation robot includes: Based on the robot's joint information tree, connection information, visual information, and motion information of multiple child nodes are created. Each child node includes the joint, its parent link, and its child links. The connection information of each child node includes the connection information between the parent link and the child links of the joint, as well as the connection information between the child node and other child nodes. The visual information of each child node includes its appearance structure and position. The motion information of each child node includes its position attributes and collision response. The position attributes are used to adjust the rotation or position of the joint to adjust its initial posture. The collision response is used to determine the response when the child node collides with a simulated collider. Based on the parameter information, visual information, and motion information of each sub-node, the state of each sub-node is simulated to obtain the simulated robot.

3. The simulation method for robot motion according to claim 1, characterized in that, The multimodal sensors include a colored camera sensor and a lidar sensor. The colored camera sensor is created based on the vision node of the Godot engine, and the lidar sensor is created based on the ray projection node of the Godot engine. The process of calling the multimodal sensors of the simulated robot to extract features from the simulation environment and obtain task status data of the simulated robot's task to be executed includes: Based on the viewport node of the colored camera sensor, feature extraction is performed on the simulation environment to obtain a colored image associated with the task to be performed; Based on at least one RayCast3D node of the lidar sensor, at least one laser beam is simulated; the code of the lidar sensor controls the rotation of the RayCast3D node to achieve scanning within a set angle range, thereby obtaining scanning data in the simulation environment. The scanning data includes the distance of the RayCast3D node to each simulated collider in the simulation environment. The task status data is obtained based on the colored image and the scan data.

4. The simulation method for robot motion according to claim 1, characterized in that, The simulation environment includes simulated collision objects and simulated scenarios, and the simulation environment is determined based on the following method: In the 3D creation software of the game engine, each simulated collision body is obtained based on the 3D shape data, material data, surface texture data and structural information of each item; the structural information of the item includes the category label of the combined structure item or the category label of the single structure item, as well as the component connection relationship of the combined structure item. In the 3D creation software, the scene where the robot is located is modeled, and the positions and orientations of the objects are arranged to obtain the simulation scene and its root scene file; Based on the names, positions, and orientations of the items in the root scene file, each of the simulated colliders is imported into the simulation scene to obtain the simulation environment.

5. The simulation method for robot motion according to claim 1, characterized in that, The step of determining the execution action information of the simulated robot based on the task status data includes: The motion of the simulated robot is simulated based on the execution action information at the current moment, and the completion degree of the task to be executed and the task status data at the current moment are obtained. Based on the completion status of the task to be executed, evaluate the post-execution status of the execution action information at the current moment, and obtain the reward score at the current moment; The driving strategy decision-making agent calculates the task status data and reward score at the current moment to determine the execution action information of the simulated robot at the next moment; The execution action information of the next moment is used as the execution action information of the current moment. The steps of simulating the motion of the simulation robot based on the execution action information of the current moment are performed to obtain the completion degree of the task to be executed and the task status data of the current moment, until the task to be executed is completed.

6. The simulation method for robot motion according to claim 3, characterized in that, The multimodal sensor also includes a depth camera sensor, which is used to acquire the colored image. Based on the viewport node of the depth camera sensor, acquire the surface texture image of the simulated collider associated with the task to be performed; Perform fisheye distortion calculation on the surface texture image of the simulated collider to obtain the distorted surface texture image; The distorted surface texture image is converted into the colored image.

7. A simulation device for robot motion, characterized in that, include: A building module is used to construct a simulated robot based on an editable joint information tree of the robot. The joint information tree includes a root node, at least one joint and its motion parameters, and each joint is connected to the root node through a link. Each joint is connected to a parent link and a child link. The task status data acquisition module is used to call the multimodal sensors of the simulation robot to extract features from the simulation environment and acquire the task status data of the simulation robot's task to be executed. The motion module is used to determine the execution action information of the simulated robot based on the task status data, so as to simulate the robot's motion.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the simulation method for robot motion as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the simulation method for robot motion as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the simulation method for robot motion as described in any one of claims 1 to 6.

Images