A virtual-real combined digital twin system based on unreal engine and mobile robot

Abstract

The application discloses a virtual-real combined digital twin system based on an Unreal Engine and a mobile robot, and relates to the field of mobile robots.A physical test platform in the system comprises an omnidirectional wheel robot, a camera array and an AprilTag-based positioning system; the omnidirectional wheel robot comprises a robot chassis with multiple omnidirectional wheels and a controller with different processors; a virtual test platform is built based on the physical test platform, the Unreal Engine 5 and a PhysX physical engine; a virtual-real interaction framework comprises a communication framework based on a MySQL database and a heuristic parameter search algorithm; the communication framework based on the MySQL database serves as a data storage and transfer station for the virtual test platform and the physical test platform; and the heuristic parameter search algorithm searches for part of physical parameters of the virtual test platform.The application can improve control effect and reduce virtual-real errors when facing the same type of control tasks.

Description

Technical Field

[0001] This invention relates to the field of mobile robots, and in particular to a virtual-real combined digital twin system based on Unreal Engine and mobile robots. Background Technology

[0002] Mobile robot research has garnered increasing attention in recent years and has been applied in numerous industrial and military scenarios. The importance of mobile robot testbeds as a crucial link in verifying theoretical results and measuring product quality is self-evident. Currently, mobile robot testbeds are mainly divided into two categories: physical testbeds and virtual testbeds. Physical testbeds can provide the most accurate experimental results and strong data support; however, due to strict constraints in space, time, and cost, and the inability to guarantee robot safety, physical testbeds are difficult to meet the needs of researchers. Virtual testbeds are widely used in academic research due to their high customizability and lower cost; however, limited by current physical simulation technology, the simulation results of virtual testbeds are often inaccurate, and the trained controllers are difficult to transfer to real-world applications. To simplify the transfer of control strategies from virtual to real, sim-to-real techniques are typically used to adjust the virtual simulation environment to reduce the virtual-to-real error.

[0003] Yevgen Chebotar et al. from NVIDIA proposed a "sim-to-real" approach for two specific tasks: a robotic arm opening and closing a cabinet, and a swinging rope controlling a connected rigid body to enter a hole. First, a virtual simulation system is built, creating a 3D model consistent with reality. Next, a parameterized Markov process is constructed, where the simulator parameters are adjustable and each has its own initial distribution, which is also controlled by parameters. The parameters are randomly sampled using domain randomization to generate a large number of scenarios. Reinforcement learning is then performed on these generated scenarios to obtain the optimal controller for the task under the current parameter distribution. This controller is then used simultaneously in both the virtual simulation and actual experimental environments to collect virtual and actual operational data, thereby calculating an index describing the magnitude of the virtual-to-real error (this index can be flexibly set according to the task). Based on the calculated virtual-to-real error, the current distribution of each parameter, and the current optimal controller, the designed reinforcement learning-based method SimOpt is used to update the distribution of all parameters. This process is repeated until the virtual-to-real error is reduced to a certain range.

[0004] Zhaoming Xie et al. proposed an iterative design method that uses Deterministic Action Stochastic State (DASS) tuples to represent deterministic policy behaviors associated with states accessed by stochastic policies, thus achieving transfer learning. Without dynamic randomization, they successfully trained a highly robust control policy by introducing noise into the actuator's dynamics, performing accurate system identification, calling state estimators in both virtual and physical environments for training, and selecting appropriate state and action spaces. This enabled the transfer of the simulated policy to a physical bipedal robot.

[0005] However, existing solutions often start from a specific task, using domain randomization or other methods to obtain a control policy during the training phase, then modify the parameter distribution, and then sample again based on the new parameter distribution, repeatedly repeating the above iterative process to obtain a control policy with strong robustness. The parameter distribution trained by this method is strongly correlated with the environment, which has significant limitations. If the task changes, or even if only the environment changes (such as changes in the weight or friction coefficient of an object), a new set of parameter distributions must be retrained; otherwise, the same control policy will not be able to guarantee a small virtual-to-real error.

[0006] Current technologies only consider the impact of physical parameters in the virtual testbed's physics engine, without delving into the fundamental causes of the virtual-to-real error. Besides the simulation algorithm parameters, the model accuracy of the physical simulation algorithms used by current physics engines is limited, and many real-world force calculations are simplified or even ignored. Furthermore, most current physics simulation algorithms possess a degree of randomness. All these factors contribute to the significant virtual-to-real error between the virtual and real worlds.

[0007] Furthermore, in order to find a suitable control strategy, existing technologies require the use of simulation algorithms with high real-time performance. However, since the accuracy and speed of simulation algorithms are often mutually exclusive, the accuracy of simulation algorithms is usually sacrificed given current computing power. Therefore, the inherent characteristic of learning algorithms to achieve their goals through high-number of iterations of training makes it impossible to use high-precision simulation algorithms, which brings some inherent disadvantages. Summary of the Invention

[0008] The purpose of this invention is to provide a virtual-real combined digital twin system based on Unreal Engine and mobile robot, which can improve control effect and reduce virtual-real error when facing the same type of control task.

[0009] To achieve the above objectives, the present invention provides the following solution:

[0010] A virtual-real integrated digital twin system based on Unreal Engine and mobile robots includes: a physical test bench, a virtual test bench, and a virtual-real interaction framework;

[0011] The physical test bench includes: an omnidirectional wheel robot, a camera array, and an AprilTag-based positioning system; the omnidirectional wheel robot includes a robot chassis with multiple omnidirectional wheels and controllers with different processors; the robot chassis is connected to the controllers via a unified connector;

[0012] The virtual test bench is built based on the physical test bench, using Unreal Engine 5 and the PhysX physics engine.

[0013] The virtual-physical interaction framework includes: a communication framework based on a MySQL database and a heuristic parameter search algorithm; the communication framework based on the MySQL database is used to combine the virtual test bench and the physical test bench, and serves as a storage and relay station for data between the virtual test bench and the physical test bench; the heuristic parameter search algorithm is used to search for some physical parameters of the virtual test bench.

[0014] Optionally, a camera array mounted on a physical test bench can capture the poses of multiple robots on the platform using AprilTag positioning.

[0015] Optionally, the basic unit of the camera array of the physical test platform is a distributed node consisting of a two-degree-of-freedom camera pan-tilt unit and a camera.

[0016] The camera array is based on a distributed camera pose optimization algorithm with multiple indicators. It constructs new evaluation indicators from the perspectives of viewpoint and distortion, and uses particle swarm optimization to detect the pose of the omnidirectional wheel robot in the physical test bench.

[0017] Optionally, the chassis of the omnidirectional wheel robot on the physical test bench is equipped with four brushless motors with a maximum power of 176W.

[0018] Optionally, the processor of the omnidirectional wheel robot on the physical test bench includes three types of processors: 8-bit ATmega2560, 32-bit ATSAM 3X, and Nvidia Jetson TX2.

[0019] Optionally, the process of building a virtual testbed based on Unreal Engine 5 and the PhysX physics engine is as follows:

[0020] Build a 3D model of the omnidirectional wheel robot, import the STL file into Blender for assembly and skeleton skinning, and determine the skeleton mesh model;

[0021] Import the skeletal mesh model into Unreal Engine 5 to generate complex collision bodies for physical simulation;

[0022] Develop a communication plugin between Unreal Engine 5 and MySQL database, as well as a virtual robot control module, to enable the control of virtual robots.

[0023] Optionally, some physical parameters of the virtual testing platform include: the robot's main body mass, the mass of the robot's four omnidirectional wheels, the linear damping of the four omnidirectional wheel joints, and the rotational damping of the four omnidirectional wheel joints.

[0024] Optionally, it also includes: a communication framework based on a MySQL database and a virtual-real interaction framework based on a heuristic parameter search algorithm, wherein the algorithm of the virtual-real interaction framework is:

[0025] Using formula As an indicator for measuring virtual and real trajectories, it comprehensively considers spatial and temporal errors;

[0026] Where, τ phy and τ vir Let represent the state trajectories generated by the robot under specific input control in the physical test bench and the virtual test bench, respectively. DTW is the similarity function of the time series, α is the ratio of the total time span of the virtual test bench and the physical test bench, k is a penalty coefficient used to amplify the error, and β is the weighting coefficient.

[0027] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0028] This invention provides a virtual-physical hybrid digital twin system based on Unreal Engine and mobile robots. By highly utilizing information from both virtual and physical testbeds, coupled with a highly compatible and scalable communication framework and an efficient virtual testbed parameter optimization algorithm, it organically integrates virtual and physical testbeds to achieve a "sim-to-real" connection. This framework is generalizable and can be applied to various robotic platforms, reducing virtual-physical errors in research. A robot chassis with multiple omnidirectional wheels and controllers with different processors enables various driving modes to be applied to different experimental scenarios. Optimization is performed individually for different control tasks to achieve better control performance. A communication framework based on a MySQL database isolates direct data exchange between the virtual and physical test benches, using the database as a data transfer station and repository. A dedicated communication protocol within the framework standardizes the entire virtual-physical interaction data communication process, bringing greater flexibility and scalability to the system. A heuristic parameter search algorithm can add or remove constraints in real time based on simulation conditions, thereby quickly searching for a set of optimal parameters within a limited time. This maximizes the accuracy of the simulation algorithm used by the learning algorithm, making the dynamics of the virtual test bench more similar to the physical test bench, effectively reducing virtual-physical errors. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 A schematic diagram of a virtual-real integrated digital twin system based on Unreal Engine and mobile robot provided by the present invention;

[0031] Figure 2 This is a diagram of the physical test bench architecture.

[0032] Figure 3 Model the control pipeline flowchart for the virtual test bench;

[0033] Figure 4 This is a diagram of the virtual-real interactive communication framework architecture.

[0034] Figure 5 This is a diagram of the parameter search algorithm for the virtual-real interaction framework. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] The purpose of this invention is to provide a virtual-real combined digital twin system based on Unreal Engine and mobile robot, which can improve control effect and reduce virtual-real error when facing the same type of control task.

[0037] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] like Figure 1 As shown, the present invention provides a virtual-real integrated digital twin system based on Unreal Engine and mobile robot, including: a physical test bench, a virtual test bench, and a virtual-real interaction framework.

[0039] This invention provides a virtual-real hybrid digital twin system based on Unreal Engine and a mobile robot. To fully stimulate the dynamic characteristics of the system to be identified, the system is fully excited in both the time and frequency domains, ensuring that the input and output data encompass the essential features of the system. The excitation input signal of this invention is a mixture of control signal and Gaussian noise, covering all modes of the system. This prevents the input from being too singular due to the current control task, which could lead to the input and output data failing to include all the essential features of the system, thereby improving the accuracy and reliability of identification. This method can effectively help the virtual-real interaction framework capture more features of the robot in both the physical and virtual test benches, thereby improving the understanding and control of system dynamics and providing more reliable and accurate guidance for system control.

[0040] like Figure 2 As shown, the physical test bench includes: an omnidirectional wheel robot, a camera array, and an AprilTag-based positioning system; the omnidirectional wheel robot includes a robot chassis with multiple omnidirectional wheels and controllers with different processors; the robot chassis is connected to the controllers via a unified connector.

[0041] As a specific embodiment, the omnidirectional wheeled robot includes a robot chassis with six omnidirectional wheels. Four brushless motors with a maximum power of 176W are mounted on the robot chassis, with a total load of approximately seven kilograms. Each motor is equipped with an independent suspension system to ensure good ground contact and stable robot operation. For the controller, three different processors are used to meet the computational needs of different control scenarios.

[0042] The processors include three types: the 8-bit ATmega2560, the 32-bit ATSAM 3X, and the Nvidia Jetson TX2.

[0043] The control architecture is consistent across all types of robots. The system input commands include target position and target velocity. Based on feedback from IMU, magnetic sensors, and encoders, the robot position and velocity estimator can calculate the estimated values ​​of the robot position and velocity.

[0044] The 32-bit processor, paired with positioning devices such as ultra-wideband positioning, LiDAR, and Global Navigation Satellite System (GNSS), enables self-localization and distributed control tasks. The Nvidia Jetson TX2 allows robots with powerful computing capabilities to perform intelligent autonomous control tasks in conjunction with peripheral devices such as depth cameras, LiDAR, and real-time kinematics-based GNSS. A unified connector connecting the robot chassis and controls across multiple drive modes features 38 pins: 4 for battery power supply and charging, and 31 for connecting four motors and a steering gear to meet the needs of all drive modes.

[0045] The basic unit of the camera array is a distributed node consisting of a two-degree-of-freedom camera pan-tilt unit and cameras mounted on it.

[0046] The camera array is based on a distributed camera pose optimization algorithm with multiple indicators. It constructs new evaluation indicators from the perspectives of viewpoint and distortion, and uses particle swarm optimization to detect the pose of the omnidirectional wheel robot in the physical test bench.

[0047] The camera achieves localization by capturing AprilTags, and the localization system further estimates the robot's speed by analyzing the position of the AprilTag in adjacent samples and the sampling interval. AprilTag is an image-based QR code used for visual measurement and target recognition. Currently used visual tracking algorithms include feature point tracking, Kalman filtering, and extended Kalman filtering, but these methods are often affected by factors such as ambient lighting, target occlusion, and motion blur, making it difficult to guarantee high tracking accuracy and robustness. In contrast, QR code-based visual tracking technology can guarantee high tracking accuracy and robustness. Furthermore, compared to other QR code-based visual tracking technologies, AprilTag-based visual tracking technology is faster, more accurate, and more robust. AprilTag markers have fixed characteristics, enabling stable and accurate recognition and localization under different postures and distances. They can be used in various environments (e.g., indoor, outdoor, low-light conditions) and have good tolerance to occlusion and distortion.

[0048] AprilTag enables localization in six degrees of freedom from a single image. It employs a novel lexicode-based encoding system, representing a 36-bit code with a minimum Hamming distance of 10 as a 36h10 code. Through a detection algorithm process involving line detection, frame detection, homography transformation, extrinsic parameter estimation, and decoding, AprilTag localization is ultimately achieved.

[0049] To ensure good image quality and accurate positioning, it is necessary to control the camera array to track the robot and keep it as close to the center of the image as possible. This invention uses a novel distributed camera coverage algorithm, which redefines the image quality index from three perspectives: viewpoint quality, resolution quality, and focus quality. The Particle Swarm Optimization (PSO) algorithm is used to optimize the orientation of the cameras.

[0050] The two-DOF gimbal is printed using PLA (polylactic acid) material. The gimbal uses two servos to achieve pan and tilt degrees of freedom. Combined with a zoom lens module, it forms a PTZ gimbal camera capable of omnidirectional movement (including left, right, and up / down) and lens zoom control.

[0051] Each PTZ camera is treated as a distributed node, and a type of small, high-precision bus servo motor is used in the pan-tilt unit to meet the system's requirements for high precision and real-time performance. Each camera is calibrated and distortion-corrected using MATLAB's Camera Calibration Toolbox. Furthermore, to achieve efficient and flexible visual monitoring, this invention defines a novel index that comprehensively considers viewpoint quality, resolution quality, and focus quality, as follows:

[0052]

[0053] Where d x k represents the distance from the imaging point to the midpoint of the pixel plane. x h is the ratio of the current imaging distance to the optimal imaging distance. x λ represents the distance from the object to the imaging plane, h0 represents the distance from the focus point to the imaging plane (defaulting to the center of the image), and λ is an adjustable parameter. To solve the camera coverage problem, a particle swarm optimization algorithm is used to obtain a feasible solution close to the optimal solution within a controllable time.

[0054] The process of setting up the virtual testing platform is as follows:

[0055] 1) Construct a 3D model of the omnidirectional wheel robot, import the STL file into Blender for assembly and skeleton skinning, and determine the skeleton mesh model;

[0056] 2) Import the skeletal mesh model into Unreal Engine 5 to generate complex collision bodies and perform physical simulation;

[0057] 3) Develop a communication plugin between Unreal Engine 5 and MySQL database, as well as a virtual robot control module, to enable the control of the virtual robot.

[0058] The virtual testbed, built on Unreal Engine 5, is a digital twin of the physical testbed, with models derived from real-world objects on a one-to-one scale. Unreal Engine's PhysX physics engine is an open-source real-time physics engine widely used for simulating game scenarios with relatively low precision requirements, and it can stably simulate multi-joint robots. Hummel et al. found that PhysX has the most stable constraints and less penetration, which is essential for multi-joint robot simulation. Furthermore, compared to the Bullet physics engine commonly used for robot simulation, PhysX can accelerate simulations on GPUs. Any CUDA-enabled GeForce graphics card can leverage PhysX's high parallel performance without additional processing. For more localized, higher-precision simulations (such as contact and collisions between the robotic arm's end effector and objects), Unreal Engine also allows users to reproduce cutting-edge physics simulation algorithms.

[0059] like Figure 3 As shown, to achieve a closed loop between the virtual and physical systems, this invention employs a one-to-one modeling approach starting from the component level. The four-omnidirectional wheel mobile robot of this invention comprises a total of 268 parts, mainly categorized as a controller, chassis, four suspension systems, four motors, and four omnidirectional wheels and their rotors. The modeling of each component was completed in the industrial 3D modeling software SolidWorks. Each omnidirectional wheel is equipped with 16 auxiliary rotors. The overall dimensions are 350mm × 350mm × 300mm. From shape and material to size and weight, all indicators are consistent with the actual robot. Furthermore, to facilitate control within the Unreal Engine, the x-axis of the local coordinate system of the omnidirectional wheel hubs and the driven rotors is aligned with the rotation axis, ensuring standardization.

[0060] During the modeling process, the model's outer contour, materials, and parameters of each part were ensured to be consistent with their corresponding real-world counterparts, enabling the generation of higher-precision colliders in Unreal Engine. Assembling the modeled parts yields a complete robot model, but because the various parts are not physically linked, the resulting jitter during control is severe. This invention imports the model into 3D design software (such as Blender) to apply bone skinning to each joint, binding the entire model to a skeleton to generate a skeletal mesh model. This skeleton contains a wealth of model-related information (such as the connection relationships between parts, parent-child relationships, and six degrees of freedom joints), aiding in control. Furthermore, to prevent the robot from exploding due to self-collision and self-friction during simulation, this invention imports the skeletal mesh model into Unreal Engine 5 to further add physical constraints, limiting unnecessary joint movements. For example, the rotational degree of freedom of the wheels is reduced from 3 to 1, retaining only its rotational degree of freedom around its axis. After these operations, torque can be directly applied to the joints to achieve control of individual joints.

[0061] Thanks to the standardization of the virtual modeling pipeline, the control of virtual robots has become very simple and accurate. The control signals of the virtual test bench consist of only four parts, corresponding to the four torques applied to the wheel joints, while the other joints move freely under the forces of the robot and the environment.

[0062] Specifically, the basic steps of skinning a skeleton are as follows:

[0063] Step 1: Calculate the robot's center of gravity and set it as the highest-level parent node of the entire robot skeleton;

[0064] Step 2: Identify the robot's main movable joints and their parent-child relationships, and calculate the center of gravity for the child node models of each movable joint;

[0065] Step 3: Extend the skeleton starting from the highest-level parent node. The start and end points of a complete skeleton must be the rotation centers of the parent and child node models, respectively. Segmented skeletons ensure that the end bones of the child nodes coincide with the rotation axis. Specifically, starting from the center of gravity of the main body, connect the rotation centers of the four omnidirectional wheel hubs. Then, starting from the rotation centers of the omnidirectional wheel hubs, extend the skeleton to connect the rotation centers of the 64 auxiliary rotors (each omnidirectional wheel connects to 16 rotors). The complete skeleton consists of 69 nodes and 68 bones.

[0066] Step 4: Begin skinning by binding each of the 69 nodes to its corresponding model, allowing the model and skeleton joints to rotate in tandem.

[0067] Step 5: Export the model as an FBX format. The model obtained after this step is called a skeletal mesh model.

[0068] After skinning the skeleton, the skeletal mesh model is imported into Unreal Engine. Each node in the skeleton is compiled into a joint. Each joint has three initial rotational degrees of freedom by default. For the specific scenario of this invention, physical constraints are added to each joint, retaining only the single degree of freedom of rotation relative to the axis of rotation. Finally, a complex collider is generated using Unreal Engine for physics simulation. During the simulation, any abnormal behavior of the machine in different states can be checked, and the model can be modified as necessary.

[0069] To reduce virtual-real errors, the virtual-real interaction framework organically combines the virtual and physical test benches through information interaction, driving the search for virtual physical parameters and continuously optimizing the virtual test bench so that its performance gradually approaches that of the physical test bench. Specifically, the key to implementing this virtual-real interaction framework lies in the following two points: i) a database-based communication framework; ii) a parameter search algorithm.

[0070] Since real-world testing environments are often complex and dynamic, a communication system with good scalability is essential to meet diverse experimental needs. This invention blocks direct information exchange between virtual and physical test benches, using a MySQL database as a data storage and relay station to enhance system flexibility. Applications on both physical and virtual test benches can upload their own data or obtain data from other test benches as needed. Thanks to MySQL's robust ecosystem, all CRUD operations can be implemented using simple, cross-platform SQL code. This invention packages all necessary SQL language functions, develops dedicated database communication plugins for each platform, and designs a standard communication protocol to enhance system scalability. Applications on various platforms can integrate into the system using the same interface, minimizing the workload required for switching scenarios when facing different experimental environments. The read / write speed of this communication framework fully meets the communication requirements of both physical and virtual test benches. In testing, transmitting 64 bytes of data between the test bench and the communication framework's database takes 3 to 6 milliseconds, meeting the control frequency requirements of typical physical test bench robots. In the virtual test bench, database read and write operations are required in every frame, and the maximum frame rate of the virtual test bench is 120 frames per second. The communication framework of this invention can fully meet its communication requirements.

[0071] In experiments on any physical system, the system's dynamics are completely determined by environmental conditions and robot hardware conditions. However, in a virtual environment, the system's dynamics are jointly determined by the simulator's parameters, simulation algorithm, and model. This invention uses the PhysX physics engine as the physical computation backend for a virtual testbed to perform rigid body simulation. The PhysX physics engine provides a large number of adjustable physical parameters, such as mass, linear damping, and angular damping. Utilizing this feature, this invention designs a heuristic parameter search algorithm to search for the physical parameters of various objects in the virtual environment. By finding a suitable set of physical parameters, the dynamics of the virtual testbed are corrected, making its input-output relationship as close as possible to that of a physical testbed.

[0072] Specifically, before the iterative search begins, an experiment is conducted on a physical test bench using a fixed control sequence, and the trajectory is recorded for comparison with the trajectory on the virtual test bench. Then, all parameters to be searched are sampled according to a manually defined search range and probability distribution to obtain a series of initial parameter values. These initial values ​​are applied to the virtual test bench to start the simulation, i.e., the first iteration. Each time a trajectory is completed on the virtual test bench, the simulation stops, and the acquired trajectory data is read to evaluate the virtual-real error in the current iteration.

[0073] Since the time span, sampling frequency, and number of sampling points of the trajectories measured in the virtual test bench and the physical test bench are inconsistent in most cases, this invention designs a new trajectory similarity evaluation index based on Dynamic Time Wrapping (DTW) to evaluate the dynamic error between the virtual and physical test benches as accurately as possible. After calculating the error, this algorithm searches for parameters following the approach of simulated annealing. At the end of each iteration, the parameters are sampled in the neighborhood of the current parameter. At this point, some custom constraints can be freely added to use some prior knowledge to accelerate the search process. For example, when the time span of the trajectory on the virtual test bench is much longer than that on the physical test bench, constraints can be added to stipulate that the mass of each robot part obtained from neighboring samples will not exceed the original value.

[0074] like Figure 4 As shown, this invention uses a MySQL database as the data transfer station and repository in the entire framework. Various applications or modules in the virtual and physical test benches can upload or pull data as needed. To ensure strong scalability of the communication framework, this invention designs a communication protocol. All applications must normalize data before storing it in the database, which facilitates software development. Due to the cross-platform nature of SQL, this invention develops corresponding data upload and normalization plugins for each operating system of each test bench, ensuring the uniformity of uploaded data. When each application requests data from the database, it also needs to filter the normalized data to obtain the required portion. Therefore, this invention also develops independent data acquisition plugins for each platform. For different network environments, since the two or more communicating terminals are not on the same network segment, the communication system cannot obtain the target device's MAC address, and direct communication between the application and the database is impossible. This invention uses an HTTP web server to implement data forwarding between the application and the database, and the data communication with HTTP is accomplished using the classic C++ library cpp-httplib.

[0075] like Figure 5As shown, since the Unreal Engine's built-in PhysX physics engine provides users with interfaces for a large number of physical parameters, this invention designs a heuristic parameter search algorithm to search for some physical parameters of the virtual test bench. In this invention, the parameters currently being searched include: the robot's main body mass, the mass of the robot's four omnidirectional wheels, the linear damping of the four omnidirectional wheel joints, and the rotational damping of the four omnidirectional wheel joints. The parameter search is mainly driven by the virtual-real error, which is evaluated by the similarity between the trajectories of the robots on the virtual and physical test benches under the same control input. The robot's trajectory error includes not only shape errors in the spatial dimension but also errors in the temporal dimension. Furthermore, since the sampling interval, number of sampling points, and total time span of the trajectories generated on the two test benches are different, this invention designs a trajectory distance measurement index that comprehensively considers both spatial and temporal errors, as follows:

[0076]

[0077] Wherein, τ is used phy and τ vir The parameters ξ represent the state trajectories generated by the robot under specific input u in the physical and virtual test benches, respectively. The search for parameter combinations ξ is performed based on the trajectories generated in the physical and virtual test benches. DTW is a classic time series similarity (distance) function, previously widely used in isolated word recognition. This algorithm, based on dynamic programming (DP), solves the problem of matching data templates of varying lengths. It is an early and classic algorithm in speech recognition, and its concept is well-suited to the trajectory evaluation scenario of this invention. α is the ratio of the total time span of the virtual and physical test benches, k is a penalty coefficient used to amplify the error, and β is a weighting coefficient. The entire error index is divided into two parts: the term containing the DTW function evaluates whether the virtual and physical trajectories are similar, considering spatial error; the latter term evaluates whether the virtual and physical trajectories are dynamically similar, i.e., whether the robot can complete the trajectory at a similar speed, considering temporal error. After calculating the error, this algorithm searches for parameters based on the idea of ​​simulated annealing. In each iteration, if the current error is less than the error of the previous iteration, then the parameters for the current iteration are accepted. If the current error is greater than or equal to the error of the previous iteration, then proceed according to a probability p = exp((E) last -E) / T) determines whether to accept the current parameter, where E lastLet E and E be the error values ​​calculated in the previous and current iterations, respectively, and let T be a coefficient that decreases in each iteration. Regardless of whether the current parameter is accepted, the algorithm will perform a random sampling within the neighborhood of the current parameter and use it in the next iteration. At this point, you can freely add some custom constraints to use prior knowledge to accelerate the search process. For example, when the time span of the virtual testbed trajectory is much longer than that of the physical testbed trajectory (i.e., α >> 1), then you can add constraints to stipulate that the mass of each robot part obtained from the neighboring samples will not exceed the original value.

[0078] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0079] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A virtual-real integrated digital twin system based on Unreal Engine and mobile robots, characterized in that, include: Physical test bench, virtual test bench, and virtual-physical interaction framework; The physical test bench includes: an omnidirectional wheel robot, a camera array, and an AprilTag-based positioning system; the omnidirectional wheel robot includes a robot chassis with multiple omnidirectional wheels and controllers with different processors; the robot chassis is connected to the controllers via a unified connector; The virtual test bench is built based on the physical test bench, using Unreal Engine 5 and the PhysX physics engine. The virtual-physical interaction framework includes: a communication framework based on a MySQL database and a heuristic parameter search algorithm; the communication framework based on the MySQL database is used to combine the virtual test bench and the physical test bench, and serves as a storage and relay station for data between the virtual test bench and the physical test bench; the heuristic parameter search algorithm is used to search for some physical parameters of the virtual test bench. Some physical parameters of the virtual test bench include: the robot's main body mass, the mass of the robot's four omnidirectional wheels, the linear damping of the four omnidirectional wheel joints, and the rotational damping of the four omnidirectional wheel joints. The aforementioned virtual-real hybrid digital twin system based on Unreal Engine and mobile robots further includes: a communication framework based on a MySQL database and a virtual-real interaction framework based on a heuristic parameter search algorithm, wherein the algorithm of the virtual-real interaction framework is as follows: Using formula As an indicator for measuring virtual and real trajectories, it comprehensively considers spatial and temporal errors; Where, τ phy and τ vir Let represent the state trajectories generated by the robot under specific input control in the physical test bench and the virtual test bench, respectively. DTW is the similarity function of the time series, α is the ratio of the total time span of the virtual test bench and the physical test bench, k is a penalty coefficient used to amplify the error, and β is the weighting coefficient.

2. The virtual-real integrated digital twin system based on Unreal Engine and mobile robot according to claim 1, characterized in that, A camera array mounted on a physical test bench captures the poses of multiple robots on the platform using AprilTag localization.

3. A virtual-real integrated digital twin system based on Unreal Engine and mobile robot according to claim 2, characterized in that, The basic unit of the camera array of the physical test platform is a distributed node consisting of a two-degree-of-freedom camera pan-tilt unit and a camera mounted on it. The camera array is based on a distributed camera pose optimization algorithm with multiple indicators. It constructs new evaluation indicators from the perspectives of viewpoint and distortion, and uses particle swarm optimization to detect the pose of the omnidirectional wheel robot in the physical test bench.

4. A virtual-real integrated digital twin system based on Unreal Engine and mobile robot according to claim 2, characterized in that, The chassis of the omnidirectional wheel robot on the physical test bench is equipped with four brushless motors with a maximum power of 176W.

5. A virtual-real integrated digital twin system based on Unreal Engine and mobile robot according to claim 2, characterized in that, The processors for the omnidirectional wheel robot on the physical test bench include three types: 8-bit ATmega 2560, 32-bit ATSAM 3X, and Nvidia Jetson TX2.

6. A virtual-real integrated digital twin system based on Unreal Engine and mobile robot according to claim 1, characterized in that, The process of building a virtual testbed based on Unreal Engine 5 and the PhysX physics engine is as follows: Build a 3D model of the omnidirectional wheel robot, import the STL file into Blender for assembly and skeleton skinning, and determine the skeleton mesh model; Import the skeletal mesh model into Unreal Engine 5 to generate complex collision bodies for physical simulation; Develop a communication plugin between Unreal Engine 5 and MySQL database, as well as a virtual robot control module, to enable the control of virtual robots.

Images