Leg-foot robot digital twinning system and operation method thereof
By designing the digital twin system of leg foot robot, using multi-source heterogeneous data fusion and digital twin model construction, the real-time and virtual and real synchronization of leg foot robot interaction with complex environments is solved, and efficient virtual and real interaction and system superiority is achieved. .
Patent Information
- Application Number
- CN202411321775.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2025-02-18
AI Technical Summary
The prior art is difficult to achieve real-time interaction and virtual and real synchronization between leg foot robots and complex environments, especially in terms of dynamic behavioral migration.
A digital twin system of leg foot robot is designed, including physical entity systems, data information systems, digital twin systems and service systems. Through multi-source heterogeneous data fusion, digital twin model construction and virtual and real synchronization methods, robots and complexity are realized. High-conservation real-time interaction of the environment.
It realizes efficient interaction between leg and foot robots and complex environments, provides a dynamic behavior migration solution based on physical models, reduces the difficulty of data transmission and communication, and improves virtual and real interaction, model accuracy, system feasibility and reliability.
Smart Images
Figure CN119475207A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of digital twin technology, and in particular, to a leg-foot robot digital twin system and an operation method thereof. Background Art
[0002] With the rapid development of science and technology, robotics has become one of the most popular research fields in the world. In order to meet the needs of human exploration of unknown environments and auxiliary operations, the application scope of contemporary robots has evolved from fixed operations in a single environment to autonomous operations in complex environments such as outer space exploration, military reconnaissance, fire prevention and disaster relief, and resource exploration. Compared with wheeled or tracked robots, legged robots have the advantages of flexible gait, strong adaptability, and high stability. They can adapt to various complex terrain environments and perform high-risk tasks. Improving the digitization level of legged robots is a key issue facing legged robots.
[0003] Digital twins are digital virtual bodies of physical entities created in a digital way, using data to simulate the behavior of physical entities in real environments, and adding or expanding new capabilities for physical entities through virtual-real interactive feedback, data fusion analysis, and iterative decision optimization. As a technology that fully utilizes models, data, and intelligent technologies and integrates multiple disciplines, digital twins can reflect the current state of the system in real time, and perform real-time optimization, decision-making, and predictive maintenance based on perceived information, providing new possibilities for the digitalization, informatization, automation, and interactivity of legged robots.
[0004] In the prior art, some research has been done on the virtual-reality synchronization method of the digital twin system. Based on the "three-dimensional structural model" of the data twin, or the "five-dimensional model" composed of physical entities, virtual entities, services, twin services and connections, some problems related to the status of industrial processes in the digital twin system have been solved. However, most of them are concentrated in the stages of auxiliary production, manufacturing and simple testing, and rarely involve the interaction process between legged robots and complex environments, and rarely involve mature and feasible solutions based on the dynamic behavior migration of physical models, which is not convenient for real-time display and provision of interactive services. Summary of the invention
[0005] In order to overcome at least one shortcoming in the prior art, the present application provides a leg-foot robot digital twin system and an operating method thereof.
[0006] First, provide a robot digital twin system, including: physical entity system, data information system, digital twin system and service system;
[0007] The physical entity system is used to obtain the perceived environment information and send it to the data information system. The data information system is used to perform multi-source heterogeneous data fusion on the perceived environment information to obtain the robot position data and map construction data. The robot position data and map construction data are sent to the physical entity system. The physical entity system is used to construct an environment map based on the robot position data and map construction data, perform path planning, and control the robot movement based on the environment map and the planned path.
[0008] The data information system sends the perceived environmental information to the digital twin system, and the digital twin system builds a digital twin model based on the perceived environmental information and the physical feature information of the robot;
[0009] The physical entity system sends the robot's joint angle information and planned path to the digital twin system through the data information system. The digital twin model in the digital twin system moves in the simulation environment according to the robot's joint angle information and planned path and generates simulation data, and sends the simulation data to the service system through the data information system.
[0010] The physical entity system sends the robot's joint angle information, environmental map, planned path, and robot position data to the service system through the data information system. The service system realizes three-dimensional visual monitoring of the robot's real-time motion status based on the robot's joint angle information, environmental map, planned path, robot position data, and simulation data.
[0011] In one embodiment, the service system sends control instructions to the physical entity system and the digital twin system through the data information system according to the complexity of the environment, so that the physical entity system and the digital twin system can complete corresponding complex actions at the same time.
[0012] In one embodiment, the physical entity system includes a perception module, a high-performance computing module and a motion control module; the perception module is used to obtain perceived environmental information, the high-performance computing module is used to build an environmental map and perform path planning based on the robot position data, and the motion control module is used to control the robot movement based on the environmental map and the planned path.
[0013] In one embodiment, the perception module includes a depth camera, an IMU, a lidar, and a UWB positioning module.
[0014] In one embodiment, the perceived environment information includes positioning and mapping data, motion and posture data.
[0015] In one embodiment, building a digital twin model includes building it from four dimensions: geometry, physics, behavior, and rules.
[0016] In one embodiment, multi-source heterogeneous data fusion includes data preprocessing, data feature extraction and selection, and data fusion.
[0017] In a second aspect, a method for operating a leg-foot robot digital twin system is provided, comprising:
[0018] Receive and perceive environmental information;
[0019] Perform multi-source heterogeneous data fusion on the perceived environment information to obtain robot position data and map construction data, and send the robot position data and map construction data to the physical entity system; the physical entity system is used to build an environment map based on the robot position data and map construction data, perform path planning, and control the robot movement based on the environment map and the planned path;
[0020] The perceived environment information is sent to the digital twin system; the digital twin system builds a digital twin model based on the perceived environment information and the physical feature information of the robot;
[0021] Send the robot's joint angle information and planned path to the digital twin system; the digital twin model in the digital twin system moves in the simulation environment according to the robot's joint angle information and planned path and generates simulation data, and sends the simulation data to the service system through the data information system;
[0022] The robot's joint angle information, environmental map, planned path, and robot position data are sent to the service system. The service system realizes three-dimensional visual monitoring of the robot's real-time motion status based on the robot's joint angle information, environmental map, planned path, robot position data, and simulation data.
[0023] In one embodiment, the method further includes: sending control instructions to the physical entity system and the digital twin system, so that the physical entity system and the digital twin system simultaneously complete corresponding complex actions.
[0024] Compared with the prior art, this application has the following beneficial effects:
[0025] 1. The present application constructs a digital twin system of a legged robot, including a physical entity system, a digital twin system, a data information system, and a service system. The systems have clear division of labor and are highly integrated. Through a high-fidelity real-time digital twin driven by physics and data information, combined with a virtual-reality synchronization method for the robot's dynamic behavior state migration, the interaction process between the legged robot and the complex environment is realized, providing a mature and feasible solution for dynamic behavior migration based on physical models, reducing the difficulty of data transmission and communication, facilitating real-time display and providing interactive services, and improving the virtual-reality interactivity, the accuracy of the model, and the feasibility, reliability and superiority of the established digital twin system.
[0026] 2. This application uses the robot's digital twin to verify and optimize the robot's mapping and path planning algorithms, builds a map in real time and performs path planning according to the simulation environment, and then verifies the effects of mapping and path planning, so as to continuously optimize the mapping and path planning effects; by conducting these tests in the Gazebo high-fidelity simulation environment, various situations and obstacles in the actual environment can be accurately simulated, reducing dependence on physical robots; this not only saves time and cost, but also avoids wear and tear on the robot hardware and potential damage risks; repeatedly testing and adjusting the algorithm in a virtual environment to make it more robust and efficient, and then applying it to the actual physical robot, can greatly improve the robot's overall performance and task completion rate;
[0027] 3. After the digital twin completes the verification and optimization of real-time mapping and path planning in a simulation environment, the coordinated operation of the robot's physical entity system, digital twin system, data information system and service system can achieve real-time positioning and mapping of the legged robot and accurate path planning, and can monitor the real-time movement of the robot based on the joint and motor position data received by the service system. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The present application may be better understood by referring to the following description given in conjunction with the accompanying drawings, which together with the following detailed description are included in this specification and form a part of this specification. In the drawings:
[0029] Figure 1 A structural block diagram of a leg-foot robot digital twin system according to an embodiment of the present application is shown;
[0030] Figure 2 A flowchart of building a digital twin model is shown;
[0031] Figure 3 A flowchart of the operating method of the leg-foot robot digital twin system is shown. DETAILED DESCRIPTION
[0032] The exemplary embodiments of the present application will be described below in conjunction with the accompanying drawings. For the sake of clarity and conciseness, not all features of the actual embodiments are described in the specification. However, it should be understood that many implementation-specific decisions can be made in the process of developing any such actual embodiments in order to achieve the specific goals of the developer, and these decisions may vary from embodiment to embodiment.
[0033] It is also necessary to explain here that, in order to avoid obscuring the present application due to unnecessary details, only the device structure closely related to the scheme according to the present application is shown in the drawings, while other details that are not closely related to the present application are omitted.
[0034] It should be understood that the present application is not limited to the described implementation forms due to the following description with reference to the accompanying drawings. In this article, where feasible, the embodiments can be combined with each other, features between different embodiments can be replaced or borrowed, and one or more features can be omitted in one embodiment.
[0035] This application provides a leg-foot robot digital twin system, Figure 1 The structure diagram of the digital twin system of the leg-foot robot according to the embodiment of the present application is shown in FIG. Figure 1 , the system includes: physical entity system, data information system, digital twin system and service system.
[0036] The physical entity system is used to obtain the perceived environment information and send it to the data information system. The data information system is used to perform multi-source heterogeneous data fusion on the perceived environment information to obtain the robot position data and map construction data, and send the robot position data and map construction data to the physical entity system. The physical entity system is used to construct an environment map according to the robot position data and map construction data, perform path planning, and control the robot movement according to the environment map and the planned path.
[0037] The data information system sends the perceived environmental information to the digital twin system, and the digital twin system builds a digital twin model based on the perceived environmental information and the physical feature information of the robot;
[0038] The physical entity system sends the robot's joint angle information and planned path to the digital twin system through the data information system. The digital twin model in the digital twin system moves in the simulation environment according to the robot's joint angle information and planned path and generates simulation data, and sends the simulation data to the service system through the data information system.
[0039] The physical entity system sends the robot's joint angle information, environmental map, planned path, and robot position data to the service system through the data information system. The service system realizes three-dimensional visual monitoring of the robot's real-time motion status based on the robot's joint angle information, environmental map, planned path, robot position data, and simulation data.
[0040] Here, the service system displays data through the robot operating system ROS and the visualization development environment Qt. In Qt, first instantiate the RenderPanel component to display the rendering view of RViz, and add the component to the Qt layout. Next, instantiate the VisualizationManager to manage all display objects and views, and create a ToolManager to manage selection tools, measurement tools, etc. According to the needs, configure these instantiated objects and use the functions of RViz to achieve visual management of the robot, including obtaining robot position data, environmental maps, planned paths, and temporarily changing the station point.
[0041] The service system uses the robot's joint angle information received by the service system to realize three-dimensional visual monitoring of the robot's real-time motion status and to draw the robot's joint angle data curve in real time on the host computer. This allows real-time monitoring of the robot's motion conditions at each joint and, when the robot exhibits abnormal behavior, the joint angle data curve can be used to help quickly diagnose the problem.
[0042] Specifically, the physical entity system adopts a legged robot, including a perception module, a high-performance computing module and a motion control module; the perception module is used to obtain perceived environmental information, the high-performance computing module is used to build an environmental map and perform path planning based on the robot's position data, and the motion control module is used to control the robot's movement based on the environmental map and the planned path.
[0043] Here, the perception module includes Intel D435i depth camera, IMU, LiDAR, and UWB positioning module; the depth camera, LiDAR, and UWB positioning module are used to obtain positioning and mapping data; the IMU is used to obtain motion and posture data, including speed, acceleration, position, roll angle, pitch angle, and yaw angle, etc. The high-performance computing module uses NVIDIA Jetson AGX Orin, and the motion control module uses UP Board to achieve stable and reliable motion execution.
[0044] The UWB positioning module communicates with four UWB base stations arranged in the robot's operating environment by transmitting and receiving ultra-wideband signals. These base stations can communicate with each other and record each other's time synchronization information. The arrival time difference algorithm is used to calculate the distance between the robot and each base station, and the specific position of the robot is determined by combining the triangulation positioning algorithm and IMU.
[0045] Specifically, the construction of the digital twin model includes four dimensions: geometry, physics, behavior, and rules. The digital twin model is constructed based on the perceived environment information and the physical feature information of the robot, where the perceived environment information is used to construct the behavior dimension and the rule dimension. Figure 2 The flowchart of building a digital twin model is shown in Figure 2 , specifically including:
[0046] Construction of geometric dimensions: Based on the physical feature information of the robot, that is, the external features of the physical entity system, the design is carried out with the help of SolidWorks software, and the proportional construction is carried out according to the solid model. After construction, the three-dimensional model is converted into a lightweight description file in URDF format. URDF is an XML-based description language specifically used to define the virtual model of the robot in the computer. It can describe the physical structure, joint connection and sensor layout of the robot in a lightweight way, which is easy to parse and process to support the operation call of the digital twin system. It is the basis for visualizing the three-dimensional features of the digital twin model.
[0047] Construction of the physical dimension: The dynamic description of the digital twin model introduces natural factors such as gravity on the basis of the dynamic model to enhance the simulation of the real environment, which is the basis for the expression of motion in the simulation system.
[0048] Construction of the behavioral dimension: Providing corresponding control loops for the robot is the basis for the stable operation and real-time control of the digital twin model.
[0049] Construction of rule dimension: Provide rule constraints for the digital twin model to perform tasks corresponding to the physical entity system during the simulation process, ensuring the accuracy of task execution.
[0050] In this embodiment, by constructing a bidirectional data transmission channel between the physical layer and the data layer, the robot posture information is transmitted to the model layer, so that the digital twin model can track the state of the physical entity in real time and form a digital twin with perception, analysis, and decision-making capabilities.
[0051] Specifically, multi-source heterogeneous data fusion includes data preprocessing, data feature extraction and selection, and data fusion.
[0052] Data preprocessing: The first step is to deal with missing values, outliers and noise to ensure data quality. The second step is data synchronization and alignment. Since data from different data sources may have timestamp asynchrony, time synchronization is required. The timestamp alignment method is used to ensure that data from different sensors are processed within the same time frame. The third step is data format conversion: data from different communication protocols (serial port, TCP, ROS) are unified into a unified ROS message format.
[0053] Data feature extraction and selection: The first step is to extract key features: extract position data from UWB data, extract posture data from IMU data, extract motion state data from joint data and motor data, extract visual data (such as RGB image data, depth image data) from camera image data, and extract environmental data (such as obstacle detection data, map construction data) from LiDAR data. The second step is feature selection and dimensionality reduction: select features that have an important impact on the task and reduce redundant information. Principal component analysis (PCA), linear discriminant analysis (LDA) and other methods can be used for feature dimensionality reduction.
[0054] Data fusion: Use Kalman filter to fuse UWB positioning data and IMU attitude data, predict the robot position and attitude at the current moment based on the state estimation and IMU data of the previous moment, and then use UWB positioning data to correct the prediction results to obtain more accurate position information. In addition, convert the lidar point cloud data from the lidar coordinate system to the camera coordinate system, project the point cloud in the camera coordinate system to the image plane, use the camera's internal and external parameters to calculate, use the SIFT algorithm to extract feature points in the image, use the PCL library to extract feature points in the point cloud, use the RANSAC algorithm for feature matching, fuse feature information to generate high-precision map construction data, and improve the robot's perception ability in complex environments.
[0055] In one embodiment, the service system can also manually control the robot's actions as needed through TCP network communication; that is, the service system sends control instructions to the physical entity system and the digital twin system through the data information system according to the complexity of the environment, so that the physical entity system and the digital twin system can complete the corresponding complex actions at the same time.
[0056] Furthermore, during the construction of the digital twin system of the legged robot, the accuracy and reliability of the kinematic model, mapping algorithm, positioning accuracy and path planning algorithm of the digital twin in the simulation environment are verified based on the completed system, and the model and algorithm are iteratively optimized and dynamically corrected.
[0057] The present application also provides a method for operating a leg-foot robot digital twin system. Figure 3 A flowchart of an operation method of a leg-foot robot digital twin system is shown, and the method is applied to a data information system, including:
[0058] Receive and perceive environmental information;
[0059] Perform multi-source heterogeneous data fusion on the perceived environment information to obtain robot position data and map construction data, and send the robot position data and map construction data to the physical entity system; the physical entity system is used to build an environment map based on the robot position data and map construction data, perform path planning, and control the robot movement based on the environment map and the planned path;
[0060] The perceived environment information is sent to the digital twin system; the digital twin system builds a digital twin model based on the perceived environment information and the physical feature information of the robot;
[0061] Send the robot's joint angle information and planned path to the digital twin system; the digital twin model in the digital twin system moves in the simulation environment according to the robot's joint angle information and planned path and generates simulation data, and sends the simulation data to the service system through the data information system;
[0062] The robot's joint angle information, environmental map, planned path, and robot position data are sent to the service system. The service system realizes three-dimensional visual monitoring of the robot's real-time motion status based on the robot's joint angle information, environmental map, planned path, robot position data, and simulation data.
[0063] Furthermore, the method also includes: sending control instructions to the physical entity system and the digital twin system, so that the physical entity system and the digital twin system simultaneously complete corresponding complex actions.
[0064] The operating method of the leg-foot robot digital twin system of this embodiment has the same inventive concept as the leg-foot robot digital twin system mentioned above, so the specific implementation method of this method can be seen in the embodiment part of the leg-foot robot digital twin system in the previous text, and its technical effect corresponds to the technical effect of the above method, which will not be repeated here.
[0065] In summary, this application has the following technical effects:
[0066] 1. This application constructs a digital twin system of a legged robot, including a physical entity system, a digital twin system, a data information system, and a service system. The systems have clear division of labor and are highly integrated. Through a high-fidelity real-time digital twin driven by physics and data information, combined with a virtual-real synchronization method for dynamic behavior state migration of the robot, the interaction process between the legged robot and the complex environment is realized, providing a mature and feasible solution for dynamic behavior migration based on a physical model, reducing the difficulty of data transmission and communication, facilitating real-time display and providing interactive services, and improving the interactivity of the virtual and real, the accuracy of the model, and the feasibility, reliability and superiority of the established digital twin system.
[0067] 2. This application uses the robot's digital twin to verify and optimize the robot's mapping and path planning algorithms, builds a map in real time and performs path planning according to the simulation environment, and then verifies the effects of mapping and path planning, so as to continuously optimize the mapping and path planning effects; by conducting these tests in the Gazebo high-fidelity simulation environment, various situations and obstacles in the actual environment can be accurately simulated, reducing dependence on physical robots; this not only saves time and cost, but also avoids wear and tear on the robot hardware and potential damage risks; repeatedly testing and adjusting the algorithm in a virtual environment to make it more robust and efficient, and then applying it to the actual physical robot, can greatly improve the robot's overall performance and task completion rate;
[0068] 3. After the digital twin completes the verification and optimization of real-time mapping and path planning in a simulation environment, the coordinated operation of the robot's physical entity system, digital twin system, data information system and service system can achieve real-time positioning and mapping of the legged robot and accurate path planning, and can monitor the real-time movement of the robot based on the joint and motor position data received by the service system.
[0069] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A digital twin system of a leg-foot robot, characterized in that: include: Physical entity systems, data information systems, digital twin systems, and service systems; The physical entity system is used to obtain the perceived environment information and send it to the data information system, and the data information system is used to perform multi-source heterogeneous data fusion on the perceived environment information to obtain robot position data and map construction data; the robot position data and the map construction data are sent to the physical entity system; the physical entity system is used to construct an environment map according to the robot position data and the map construction data, perform path planning, and control the robot movement according to the environment map and the planned path; The data information system sends the perceived environment information to the digital twin system, and the digital twin system builds a digital twin model according to the perceived environment information and physical feature information of the robot; The physical entity system sends the robot's joint angle information and the planned path to the digital twin system through the data information system. The digital twin model in the digital twin system moves in a simulation environment according to the robot's joint angle information and the planned path and generates simulation data, and sends the simulation data to the service system through the data information system. The physical entity system sends the robot's joint angle information, the environmental map, the planned path, and the robot's position data to the service system through the data information system. The service system implements three-dimensional visual monitoring of the robot's real-time motion status based on the robot's joint angle information, the environmental map, the planned path, the robot's position data, and the simulation data.
2. The system according to claim 1, characterized in that The service system sends control instructions to the physical entity system and the digital twin system through the data information system according to the complexity of the environment, so that the physical entity system and the digital twin system can complete corresponding complex actions at the same time.
3. The system according to claim 1, characterized in that The physical entity system includes a perception module, a high-performance computing module and a motion control module; the perception module is used to obtain perceived environmental information, the high-performance computing module is used to build an environmental map and perform path planning based on the robot position data, and the motion control module is used to control the robot movement based on the environmental map and the planned path.
4. The system according to claim 3, characterized in that The perception module includes a depth camera, an IMU, a lidar and a UWB positioning module.
5. The system according to claim 1, characterized in that The perceived environment information includes positioning and mapping data, motion and posture data.
6. The system according to claim 1, wherein: The construction of the digital twin model includes construction from four dimensions: geometry, physics, behavior and rules.
7. The system according to claim 1, characterized in that The multi-source heterogeneous data fusion includes data preprocessing, data feature extraction and selection, and data fusion.
8. A method for operating a digital twin system of a leg-foot robot, characterized in that: include: Receive and perceive environmental information; Perform multi-source heterogeneous data fusion on the perceived environment information to obtain robot position data and map construction data, and send the robot position data and the map construction data to a physical entity system; the physical entity system is used to construct an environment map according to the robot position data and the map construction data, perform path planning, and control the robot movement according to the environment map and the planned path; The sensed environment information is sent to a digital twin system; the digital twin system constructs a digital twin model according to the sensed environment information and physical feature information of the robot; Sending the joint angle information of the robot and the planned path to the digital twin system; the digital twin model in the digital twin system moves in a simulation environment according to the joint angle information of the robot and the planned path and generates simulation data, and sending the simulation data to the service system through the data information system; The joint angle information of the robot, the environmental map, the planned path, and the robot position data are sent to the service system. The service system realizes three-dimensional visual monitoring of the real-time motion state of the robot based on the joint angle information of the robot, the environmental map, the planned path, the robot position data, and the simulation data.
9. The method according to claim 8, characterized in that The method also includes: sending control instructions to the physical entity system and the digital twin system, so that the physical entity system and the digital twin system simultaneously complete corresponding complex actions.