Unmanned cluster virtual-real interactive simulation method based on three-dimensional motion capture

Abstract

The invention relates to an unmanned cluster virtual-real interactive simulation method based on three-dimensional motion capture, which comprises the following steps of: operating autonomous navigation and autonomous exploration functions of a single real intelligent agent in a real environment, acquiring posture information of the real intelligent agent through a motion capture system, and acquiring image, point cloud and IMU (Inertial Measurement Unit) posture information through a sensor; the autonomous navigation and autonomous exploration functions of a single virtual intelligent body are operated in a simulation environment, and posture information, images, point clouds and IMU posture information of the virtual intelligent body are obtained; sending data stored in a real environment to a simulation platform in real time, comparing and calculating a dynamic error, a kinematics error, a sensor error and an environment error, and correcting a virtual agent model and a simulation environment model; in a real environment and a simulation environment, multi-machine topology flexible formation and multi-machine cluster autonomous navigation are carried out at the same time, whether various errors of each virtual agent and a real agent exceed a set threshold value or not is calculated through comparison, and virtual-real interaction iteration training is carried out until the errors converge.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned systems, and relates to a method for unmanned cluster virtual-real interaction simulation based on three-dimensional motion capture. Background Art

[0002] At present, unmanned systems are booming and undoubtedly become an inevitable trend in future technological development. Foreign countries started early and invested heavily in this field, and the technology is relatively mature. Unmanned systems in China are also developing rapidly at present and play a role in various fields such as agriculture and industry. In the existing technology, there are significant differences between the simulation training environment and the real application environment of intelligent algorithms for unmanned clusters, resulting in a serious decline in the performance of algorithms after real machine deployment. Many algorithms cannot be used after deployment, and there are problems such as low authenticity of the simulation environment and poor credibility of the simulation results. Therefore, it is necessary for technological development to construct a method for unmanned cluster virtual-real interaction simulation that can be used for technology verification and method deployment. Summary of the Invention

[0003] The technical problem solved by the present invention is: aiming at the problem that there are significant differences between the simulation training environment and the real application environment of current intelligent algorithms, resulting in a serious decline in the performance of algorithms after real machine deployment and many algorithms cannot be used after deployment, a method for unmanned cluster virtual-real interaction simulation based on three-dimensional motion capture is proposed to verify the related technologies of unmanned system cluster collaboration, improve the authenticity of the simulation environment, improve the credibility of the simulation results, and reduce the difficulty of migrating from the simulation environment to the real environment.

[0004] The solution of the present invention to solve the technical problem is: a method for unmanned cluster virtual-real interaction simulation based on three-dimensional motion capture, comprising the following steps:

[0005] Step 1, establish a real unmanned system platform, including a three-dimensional optical motion capture acquisition system and multiple real agents;

[0006] Step 2, run the autonomous navigation and autonomous exploration functions of a single real agent in the real environment, obtain the position and attitude information of the real agent through the three-dimensional optical motion capture acquisition system, obtain image, point cloud, and IMU pose information through the sensors of the real agent itself, and store them;

[0007] Step 3, establish a simulation platform with multiple virtual agents and the surrounding environment, run the autonomous navigation and autonomous exploration functions of a single virtual agent in the simulation environment, and obtain the position and attitude information, image, point cloud, and IMU pose information of the virtual agent through the simulation platform;

[0008] Step 4, send the data stored in the real environment to the simulation platform in real time through the UDP protocol, and calculate the dynamic error, kinematic error, sensor error, and environment error by comparison;

[0009] Step 5: In the simulation platform, correct the mathematical model of the virtual intelligent agent using dynamic error, kinematic error, and sensor error, and correct the simulation environment model using environmental error;

[0010] Step 6: In the real environment and the simulation environment, based on the same world coordinate system, the same number of intelligent agents, and the same formation trajectory, first perform multi-robot topological flexible formation simultaneously, and then perform multi-robot cluster autonomous navigation simultaneously to obtain the sensor data sets, motion trajectory sets, speed sets, acceleration sets, position sets, and attitude sets of the real intelligent agent cluster and the virtual intelligent agent cluster, as well as the environmental data sets of the real intelligent agent cluster and the virtual intelligent agent cluster; Compare and calculate whether each error between the virtual intelligent agents and the real intelligent agents exceeds the set threshold. If at least one error exceeds the threshold, re-execute Steps 3-5, and perform virtual-real interactive iterative training until the error converges.

[0011] Further, the position and attitude information includes XYZ position coordinates, motion trajectory, six-degree-of-freedom attitude, robot joint angles, speed, acceleration, quaternion, and Euler angles.

[0012] Further, the environmental data set includes wind speed, air pressure, obstacle distribution, and terrain.

[0013] Further, the data stored in the real environment is sent to the simulation platform in real time through the UDP protocol, specifically:

[0014] The position and attitude information of the real intelligent agent is stored in the three-dimensional optical motion capture acquisition system, and the position and attitude information of the intelligent agent stored in the real environment is sent to the simulation platform through the UDP protocol;

[0015] The real intelligent agent runs the Robot Operating System (ROS), and the sensor data of the intelligent agent itself is stored in ROS. The sensor data of the real intelligent agent itself is sent to the simulation platform in real time through the form of ROS topic publishing and subscribing.

[0016] Further, when the real unmanned system platform and the simulation platform perform virtual-real interactive simulation, the following two aspects of problems need to be solved:

[0017] In terms of spatial alignment: Define the world coordinate system and the body coordinate system of the real intelligent agent, and define the world coordinate system and the body coordinate system of the virtual intelligent agent. Both the world coordinate system and the body coordinate system include the axis direction and the origin position; Use a calibration board to calibrate the world coordinate system and the body coordinate system of the real environment and the simulation environment, and calculate the rotation matrix and translation vector from the real environment coordinate system to the simulation environment coordinate system according to the coordinate system difference between the real environment and the simulation system;

[0018] In terms of data time synchronization: Synchronize the clocks of the virtual environment and the real environment to the same clock source, add timestamps to the data packets collected by motion capture, and perform data interpolation or delay compensation according to the timestamps in the simulation environment to ensure the real-time performance and consistency of the data. In addition, the data acquisition and simulation of real agents and virtual agents can be made to keep in step by using a synchronization signal generator.

[0019] Furthermore, the comparison and calculation of dynamic errors, kinematic errors, sensor errors, and environmental errors are as follows:

[0020] Dynamic error: Adopt the difference based on equation solving, establish a dynamic equation according to Newton's second law and the law of rotation, set the same control input quantity and initial conditions, numerically solve their respective dynamic states, and calculate the difference of the corresponding state quantities as the error, including acceleration error and angular velocity error.

[0021] Kinematic error: For the attitude angle error, compare the pitch angle, roll angle, and yaw angle change curves of the virtual agent and the real agent within the same time period, and calculate the mean absolute error.

[0022] Sensor error: Based on data statistics, for the same physical quantity, including image, point cloud, and IMU pose information, measure the output values of the real sensor and the virtual sensor multiple times under the same environment and the same agent state, and calculate the mean value of the output error as the sensor error.

[0023] Environmental error: For wind speed error, set a wind speed function V1(t) that changes with time in the virtual environment, measure the real wind speed V2(t) at the corresponding moment with an anemometer in the real environment, and calculate the average error between V1(t) and V2(t) over a period of time. For air pressure error, set a virtual air pressure function P1 that changes with altitude and position in the virtual environment, measure the corresponding real air pressure P2 with a barometer in the real environment, analyze the change of air pressure with altitude and position, and calculate the average error between P1 and P2 in different altitude layers or regions.

[0024] Furthermore, the use of dynamic errors, kinematic errors, and sensor errors to correct the mathematical model of the virtual agent includes:

[0025] Dynamic model correction: Adjust the acceleration, angular velocity, and aerodynamic coefficient of the virtual agent according to the dynamic error.

[0026] Kinematic model correction: For trajectory correction, adjust the attitude controller parameters of the virtual agent according to the attitude error to make the pitch, roll, and yaw angles of the virtual agent closer to the real values.

[0027] Sensor model correction: For sensor errors, calibrate and compensate the noise parameters in the virtual sensor model to make the images, point clouds, and IMU pose information output by the virtual sensor more realistic.

[0028] Further, the method for correcting the simulation environment model using environmental errors includes:

[0029] Wind speed model correction: According to the wind speed error, adjust the wind speed model parameters in the virtual environment, including adjusting the mean value, variance, or turbulence intensity of the wind speed to make it more consistent with the wind speed in the real environment;

[0030] Air pressure model correction: According to the air pressure error, adjust the air pressure model in the virtual environment, considering the influence of altitude and geographical location factors on the air pressure to make the virtual air pressure closer to the real value.

[0031] Further, when performing multi-vehicle topological flexible formation, design the following objective function:

[0032] Φ=min c,T [J t ,J o ,J f ,J r ,J d ,J u ·λ

[0033] where λ is the weight vector for weighing each cost function, and each cost function is as follows: total time J t , representing the motion time consumed by a section of the trajectory; agent obstacle avoidance J o , representing the distance between the planned trajectory points and the nearest surrounding obstacles; formation difference degree J f , representing the difference degree between the current formation shape and the desired formation shape; agent mutual collision avoidance J r , representing the cost function for agent collision avoidance, calculated by the sum of the distances between the current trajectory points and the trajectory points of all other agents; dynamic feasibility J d , representing that the planned trajectory is reasonable and does not exceed the limits of the agent's own speed and acceleration; uniform distribution of constraint points J u , representing the degree of uniform distribution of all trajectory points.

[0034] Further, during the formation control process in the real environment, there are sudden or abrupt environmental factors, including dynamic obstacles, gusts of wind, and communication delays. The solutions adopted include:

[0035] One is to build a virtual-real combined semi-physical simulation system. By combining physical flight controls, self-organizing network communication links, and three-dimensional optical motion capture acquisition systems with the simulation platform, data interaction between agent clusters is realized;

[0036] Second, for sudden dynamic obstacles, the artificial potential field method or Bezier curve is used to achieve dynamic obstacle avoidance;

[0037] Third, for the simulation and compensation of the influence of gusts, a wind field model is introduced into the simulation environment to simulate the influence of gusts on the intelligent agent, and through the dynamic semi-physical simulation software, the control strategy of the intelligent agent is adjusted in real time to compensate for the interference caused by gusts;

[0038] Fourth, for sudden communication delays, a clustering strategy of intelligent agent clusters combining master-slave and distributed is adopted, and the trajectory point data packets planned for transmission are transmitted in batches at different times.

[0039] The beneficial effects of the present invention compared with the prior art are as follows:

[0040] Aiming at the large difference between the simulation training environment and the real application environment of the current intelligent algorithms, which leads to a serious decline in the performance of the algorithms after they are deployed on the real machine, and many algorithms cannot be used after deployment. To solve this problem, the present invention uses an optical three-dimensional motion capture system to capture the pose data and environmental information of real intelligent agents, establishes a virtual-real fusion collaborative perception verification platform for unmanned system clusters, and verifies the relevant technologies of unmanned system cluster collaboration. By importing the real pose data into the simulation platform, the accuracy of the simulation system is improved. The real experimental results are fed back into the simulation platform to continuously improve the models and parameters of the simulation environment, gradually improve the authenticity of the simulation environment, improve the credibility of the simulation results, and reduce the difficulty of migrating from the simulation environment to the real environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of a virtual-real interaction simulation method for unmanned clusters based on three-dimensional motion capture;

[0042] Figure 2 It is a schematic diagram of error correction for the simulation training environment based on virtual-real interaction iterative training. DETAILED DESCRIPTION OF THE INVENTION

[0043] The present invention will be further described below in conjunction with the drawings and embodiments.

[0044] As Figure 1 shown, a virtual-real interaction simulation method for unmanned clusters based on three-dimensional motion capture proposed by the present invention includes the following steps:

[0045] Step 1, establish a real unmanned system platform, including a three-dimensional optical motion capture acquisition system and multiple real intelligent agents;

[0046] Step 2: Run the autonomous navigation and autonomous exploration functions of a single real intelligent agent in the real environment. Obtain the position and pose information of the real intelligent agent (including but not limited to XYZ position coordinates, motion trajectory, six-degree-of-freedom pose, robot joint angles, speed, acceleration, quaternion, and Euler angles, etc.) through a three-dimensional optical motion capture acquisition system. Obtain images (optical cameras), point clouds (lidar), pose information (IMU), etc. through the sensors of the real intelligent agent itself, and store the data obtained in Step 2.

[0047] Step 3: Establish a simulation platform with multiple virtual intelligent agents and the surrounding environment. Run the autonomous navigation and autonomous exploration functions of a single virtual intelligent agent in the simulation environment. Obtain the position and pose information of the virtual intelligent agent (including but not limited to XYZ position coordinates, motion trajectory, six-degree-of-freedom pose, robot joint angles, speed, acceleration, quaternion, and Euler angles, etc.), images, point clouds, and IMU pose information through the simulation platform.

[0048] Step 4: Real-time send the data stored in the real environment to the simulation platform through the UDP protocol, and calculate the dynamic error, kinematic error, sensor error (cameras, lidar, and IMU, etc.), environmental error, etc. (wind speed, air pressure, obstacle distribution, terrain, etc.).

[0049] These errors can be calculated in many ways. For example, residual analysis, outputting a residual signal through tools such as a sliding mode observer to quantify the deviation between the digital model and the actual situation. For example, comparing the motion capture data and the simulation data to calculate the root mean square error (RMSE).

[0050] Step 5: In the simulation platform, use the dynamic error, kinematic error, and sensor error to correct the mathematical models of the virtual intelligent agent, including the dynamic model, kinematic model, sensor model, and controller model, and use the environmental error to correct the simulation environment model.

[0051] Step 6: In the real environment and the simulation environment, based on the same world coordinate system, the same number of intelligent agents, and the same formation trajectory, first perform multi-robot topological flexible formation simultaneously, and then perform multi-robot cluster autonomous navigation simultaneously. Obtain the sensor data sets, motion trajectory sets, speed sets, acceleration sets, position sets, and pose sets of the real intelligent agent cluster and the virtual intelligent agent cluster, as well as the environmental data sets (wind speed, air pressure, obstacle distribution, terrain, etc.) of the real intelligent agent cluster and the virtual intelligent agent cluster; compare and judge whether each error between the virtual intelligent agents and the real intelligent agents exceeds the set threshold. If there is at least one error exceeding the threshold, re-execute Steps 3 - 5, and perform iterative training through virtual-real interaction until the error converges.

[0052] Example 1

[0053] A method for unmanned cluster virtual-real interaction simulation based on three-dimensional motion capture proposed in this embodiment has the following characteristics:

[0054] 1) Support virtual-real interaction simulation (real machines in the loop, such as virtual-real interaction simulation between 3 real drones and 6 virtual drones), and realize data interaction and virtual-real fusion between a pure digital training environment and a real machine training environment. Use an optical three-dimensional motion capture system to capture the pose data and environmental information of real agents, establish a virtual-real fusion unmanned system cluster collaborative perception verification platform, and verify the relevant technologies of unmanned system cluster collaboration. By importing real data into the simulation system, the accuracy of the simulation system is improved. Feed the real experimental results back into the simulation environment, continuously improve the model and parameters of the simulation environment, gradually improve the authenticity of the simulation environment, increase the credibility of the simulation results, and reduce the difficulty of migrating from the simulation environment to the real environment. As Figure 2 shown.

[0055] 2) Divide the unmanned system nodes into two categories: physical nodes (real agents) and virtual nodes (virtual agents). Among them, physical nodes have a higher model accuracy, and directly connect hardware entities such as flight control systems, airborne vision computers, and networking communication links into the simulation closed-loop to form a semi-physical simulation system, achieving a simulation experiment effect with higher accuracy and credibility; while virtual nodes adopt simplified motion and control models under the condition of ensuring the basic simulation accuracy requirements, and can realize the rapid simulation of a large number of drone cluster nodes. Therefore, physical nodes have high accuracy, but consume a lot of hardware and computing resources, and are deployed in small quantities in the overall cluster simulation; while virtual nodes have slightly lower accuracy, but consume less resources and have lower costs, and can be deployed in large quantities in the cluster simulation. By reasonably configuring the number of physical nodes and virtual nodes, the balance between cost and accuracy can be achieved, and a better algorithm verification effect can be realized.

[0056] In summary, the steps of the method for unmanned cluster virtual-real interaction simulation based on three-dimensional motion capture in this embodiment are as follows:

[0057] Step 1: Establish a real unmanned system platform, including a three-dimensional optical motion capture acquisition system and multiple real agents.

[0058] In this embodiment, the motion capture system preferably adopts a three-dimensional optical motion capture acquisition system, which covers the capture area of the indoor space through motion capture cameras arranged in space, and accurately captures the three-dimensional spatial positions of the reflective marker points (Markers) placed on the capture target. After processing and calculation through advanced algorithms, the three-dimensional optical motion capture acquisition system can obtain the three-dimensional spatial coordinates (X, Y, Z) of each reflective marker point in different time measurement units; it can also set the target object as a rigid body, and through further processing and calculation of the data by professional analysis software, three-dimensional data such as the accurate position and attitude of the target object can be obtained, including: the real-time high-precision three-dimensional spatial coordinates of each reflective marker point, the six-degree-of-freedom attitude information of the rigid body, basic motion information such as speed and acceleration, and the motion trajectory of the target object.

[0059] The three-dimensional optical motion capture acquisition system mainly includes motion capture cameras, switches, customized fixed trusses, optical calibration systems, reflective identification points, lens fixing device kits, and supporting software (including image positioning and resolution software, data optimization and post-processing modules). This system has the following capabilities: the ability to accurately record motion information, the ability to solve the six-degree-of-freedom pose in real time, and the ability of optical tracking and spatial positioning. As a whole, this system can achieve high-precision unmanned system tracking and positioning capabilities, and can accurately measure the spatial positions and attitude information of intelligent agents such as indoor unmanned vehicles and drones in real time.

[0060] Step 2: Run the autonomous navigation and autonomous exploration functions of a single real intelligent agent in the real environment, obtain the position and attitude information of the real intelligent agent through the three-dimensional optical motion capture acquisition system (including but not limited to XYZ position coordinates, motion trajectory, six-degree-of-freedom attitude, robot joint angles, speed, acceleration, quaternion, and Euler angles, etc.), obtain images (optical cameras), point clouds (lidar), and pose information (IMU) through the sensors of the real intelligent agent itself, and store the data obtained in Step 2.

[0061] In this embodiment, the autonomous navigation of a single real intelligent agent means that a single unmanned aerial vehicle can generate and execute a smooth, collision-avoiding, and dynamically feasible trajectory through positioning, autonomous perception, online planning and control, etc., and safely reach the target point.

[0062] The autonomous navigation function is divided into two stages: front-end global trajectory construction and back-end local trajectory optimization. At the front-end, the positioning information is input and the coordinates of the destination to be reached are given. First, the planner plans a global trajectory without considering the environmental information. This trajectory is essentially a polynomial parametric curve composed of multiple segments of polynomials. By inputting the surrounding local environmental information, the planner checks one by one whether each segment of the polynomial is in an obstacle and collides. If a collision occurs, the A* algorithm is used to generate a local guidance path to generate the gradient information for optimizing the collision avoidance term at the back-end. Through a gradient optimization algorithm involving collision avoidance terms, smoothing terms, and dynamically feasible terms, and combined with a time reallocation strategy, a high-quality executable trajectory that can safely reach the destination is finally generated.

[0063] The autonomous exploration of a single real intelligent agent means that based on autonomous navigation, a single drone can independently decide the next target point through real-time environmental information, and finally completely cover the unknown space and obtain all the environmental information.

[0064] The core idea of the frontier-based autonomous exploration method is to divide the map into a known map and an unknown map, and define the boundary between the two as the frontier. Each time, the robot will choose the nearest frontier as the observation point, control the robot to reach the observation point, and obtain environmental information until there is no frontier in the map to be explored, which means the exploration is completed.

[0065] When the sensor information is transmitted back, the map update module will update the global map according to the sensor information of this frame; first, convert the sensor range to the world coordinate system, define the points without obstacles in the sensor range as known free areas, set the points with obstacles in the sensor range as known occupied areas, and set the areas that are not in the sensor range and are known to be unexplored as unknown areas. Then, the frontier search module will traverse the global map, find the intersection points of the unknown areas and the known free areas, and set them as candidate boundary points. If the number of candidate boundary points in the current area reaches a certain threshold, it means that the candidate boundary points in this area are not noise points. Therefore, these candidate boundary points are set as the frontier until the entire global map is traversed, and all the frontiers that meet the conditions are selected. Calculate the distance between each frontier and the current position of the intelligent agent, and select the nearest frontier to the intelligent agent as the next observation point. The autonomous navigation module will navigate the intelligent agent to the observation point, then update the global map again according to the sensor information, re-traverse the global map, find new frontiers, select the nearest frontier to the intelligent agent at this moment and navigate there, and repeat this process until no frontiers can be screened out from the global map after updating the sensor, which means the exploration is over.

[0066] Step 3: Establish a simulation platform with multiple virtual agents and the surrounding environment. Run the autonomous navigation and autonomous exploration functions of a single virtual agent in the simulation environment. Obtain the position and pose information of the virtual agent through the simulation platform (including but not limited to XYZ position coordinates, motion trajectory, six-degree-of-freedom pose, robot joint angles, speed, acceleration, quaternion, and Euler angles, etc.). Obtain images (optical cameras), point clouds (lidar), and pose information (IMU) through the sensors of the virtual agent itself.

[0067] Step 4: Real-time send the data stored in the real environment to the simulation platform through the UDP protocol, and calculate the dynamic error, kinematic error, sensor error, and environmental error by comparison.

[0068] Among them, (1) Real-time send the data stored in the real environment to the simulation platform through the UDP protocol. The specific method of data interaction is as follows:

[0069] The position and pose information of the real agent are stored in a three-dimensional optical motion capture acquisition system. Send the position and pose information of the agent stored in the real environment to the simulation platform through the UDP protocol;

[0070] The real agent runs ROS (Robot Operating System), and the sensor data of the agent itself is stored in ROS. Real-time send the sensor data of the real agent itself to the simulation platform in the form of ROS topic publishing and subscribing.

[0071] (2) For the problems of coordinate transformation and time synchronization in the virtual-real interaction simulation of the agent cluster, it is necessary to proceed from two aspects: spatial alignment and time calibration.

[0072] In terms of spatial alignment (coordinate transformation). Define the world coordinate system and the body coordinate system of the real agent, and define the world coordinate system and the body coordinate system of the virtual agent. Both the world coordinate system and the body coordinate system include the axis direction and the origin position. Use a calibration board (such as a checkerboard) to calibrate the world coordinate system and the body coordinate system of the real environment and the simulation environment. Calculate the rotation matrix and translation vector from the real environment coordinate system to the simulation environment coordinate system according to the coordinate system differences between the real environment and the simulation system.

[0073] In terms of data time synchronization: Synchronize the clocks of the virtual environment and the real environment to the same clock source, such as the clock source of the motion capture acquisition system. Add timestamps to the data packets captured by the motion capture, and perform data interpolation or delay compensation according to the timestamps in the simulation environment to ensure the real-time performance and consistency of the data. In addition, hardware such as a synchronous signal generator can also be used to ensure that the data acquisition and simulation of the real agent and the virtual agent are in step.

[0074] (3) Comparing and calculating the dynamic error, kinematic error, sensor error, and environmental error. In this embodiment, the following methods are specifically adopted:

[0075] Dynamic error: Based on the difference in equation solving, establish dynamic equations according to Newton's second law and rotational law, etc. Set the same control input and initial conditions, numerically solve their respective dynamic states, and calculate the difference in corresponding state variables as the error, including acceleration error and angular velocity error.

[0076] Kinematic error: For the attitude angle error, compare the pitch angle, roll angle, and yaw angle change curves of the virtual intelligent agent and the real intelligent agent within the same time period, and calculate the mean absolute error.

[0077] Sensor error: Based on data statistics, for the same physical quantity, including images, point clouds, IMU pose information, etc., measure the output values of the real sensor and the virtual sensor multiple times under the same environment and the same intelligent agent state, and calculate the mean value of the output error as the sensor error.

[0078] Environmental error: For the wind speed error, set a time-varying wind speed function V1(t) in the virtual environment, measure the real wind speed V2(t) at the corresponding moment with an anemometer in the real environment, and calculate the average error between V1(t) and V2(t) over a period of time. For the air pressure error, set a virtual air pressure function P1 that varies with altitude and position in the virtual environment, measure the corresponding real air pressure P2 with a barometer in the real environment, analyze the change of air pressure with altitude and position, and calculate the average error between P1 and P2 at different altitude levels or regions.

[0079] Step 5: In the simulation platform, use the dynamic error, kinematic error, and sensor error to correct the virtual intelligent agent model (including the dynamic model, kinematic model, sensor model, and controller model), and use the environmental error to correct the simulation environment model.

[0080] In the process of migrating the intelligent algorithms (including perception, planning, decision-making, control, etc.) designed and trained in the digital simulation environment to the real world, due to certain deviations between the simulation environment, unmanned system models (aerodynamic models, six-degree-of-freedom models, etc.), sensor models, and state estimation models established in the simulation environment and the real environment, collect real environment data and simulation environment data through a motion capture system for comparison and establish an error model. Introduce the error model into the intelligent algorithm to improve the consistency of the algorithm in the virtual and real environments, so as to meet the actual control requirements. This error model is used to reduce the virtual-real environment error, realize the transformation of the intelligent control algorithm from development to deployment and application, and improve the actual control effect of the intelligent control algorithm. In order to reduce the error of virtual-real migration, adopt the method of "simulation-real" virtual-real interactive iterative training to reduce the error of virtual-real migration of the intelligent algorithm.

[0081] In this embodiment, the method for correcting the virtual intelligent agent model by using dynamic error, kinematic error, and sensor error is specifically implemented as follows:

[0082] Dynamic model correction: According to the dynamic error, parameters such as the acceleration, angular velocity, and aerodynamic coefficient of the virtual intelligent agent are adjusted. For example, if the lift is insufficient resulting in an error, the lift coefficient can be appropriately increased.

[0083] Kinematic model correction: For trajectory correction, according to the attitude error, the parameters of the attitude controller of the virtual intelligent agent are adjusted, such as the parameters of the PID controller, so that the pitch, roll, and yaw angles of the virtual intelligent agent are closer to the real values.

[0084] Sensor model correction: In response to the sensor error, the noise parameters in the virtual sensor model are calibrated and compensated, so that the images, point clouds, and IMU pose information output by the virtual sensor are more realistic.

[0085] The method for correcting the simulation environment model by using the environmental error is specifically implemented as follows:

[0086] Wind speed model correction: According to the wind speed error, the parameters of the wind speed model in the virtual environment are adjusted, such as adjusting the mean value, variance, or turbulence intensity of the wind speed, so that it is more in line with the wind speed in the real environment.

[0087] Air pressure model correction: According to the air pressure error, the air pressure model in the virtual environment is adjusted, considering the influence of factors such as altitude and geographical location on the air pressure, so that the virtual air pressure is closer to the real value.

[0088] Step 6: In the real environment and the simulation environment, based on the same world coordinate system, the same number of intelligent agents, and the same formation trajectory, first perform multi-vehicle topological flexible formation simultaneously, and then perform multi-vehicle cluster autonomous navigation simultaneously to obtain the sensor data sets, motion trajectory sets, speed sets, acceleration sets, position sets, and attitude sets of the real intelligent agent cluster and the virtual intelligent agent cluster, as well as the environmental data sets of the real intelligent agent cluster and the virtual intelligent agent cluster; compare and calculate whether each error between the virtual intelligent agents and the real intelligent agents exceeds the set threshold. If there is at least one error exceeding the threshold, re-execute steps 3 to 5, and perform iterative training through virtual-real interaction until the error converges.

[0089] In this embodiment, the formation control adopts the multi-vehicle topological flexible formation technology supporting virtual-real interaction. The multi-vehicle topological flexible formation supporting virtual-real interaction means that on the basis of multi-vehicle cluster autonomous navigation, the intelligent agent platforms in the virtual environment and the real environment form a collaborative system, share their respective information, and plan a safe formation trajectory to reach the designated target point according to the topological relationship between the vehicles.

[0090] When generating the trajectory of agents for formation movement, in order to take into account the requirements that the entire trajectory can both meet the maintenance of the encirclement formation and avoid obstacles, the next trajectory point of each UAV needs to consider many factors, including: ① the obstacle avoidance factor of a single agent; ② the collision avoidance factor between agents; ③ the factor of the dynamic feasibility of the trajectory after generating the next trajectory point; ④ the factor of whether the next trajectory point generated by each agent is conducive to the maintenance of the formation; ⑤ the factor of the total time used by the agent to fly along the trajectory after generating the next trajectory point; ⑥ the factor of whether the generated trajectory points are evenly distributed on the entire trajectory with respect to the previous trajectory points. Therefore, in this embodiment, these six factors are designed as six cost functions, and the swarm formation problem is formulated as an unconstrained optimization problem according to these six factors. Its objective function consists of the following six cost functions, and the minimized objective function can be written as:

[0091] Φ=min c,T [J t ,J o ,J f ,J r ,J d ,J u ·λ

[0092] where λ is the weight vector that weighs each cost function, and each cost function is as follows: the total time J t , which represents the movement time consumed by a section of the trajectory; the agent obstacle avoidance J o , which represents the distance between the planned trajectory point and the nearest surrounding obstacle; the formation difference J f , which represents the difference between the current formation and the desired formation; the mutual collision avoidance between agents J r , which represents the cost function of collision avoidance between agents and is calculated by the sum of the distances between the current trajectory point and the trajectory points of all other agents; the dynamic feasibility J d , which represents that the planned trajectory is reasonable and does not exceed the limits of the agent's own speed and acceleration; the uniform distribution of constraint points J u , which represents the degree of uniform distribution of all trajectory points. The magnitude of the value of this objective function can be used to measure the optimal one among many trajectory points, and the optimal trajectory point is used as the next trajectory point in the trajectory, and the process is continuously looped until the encirclement task is completed. At the same time, it can also be known that for the formation task, the trajectory generated according to this method is optimal in the current terrain environment.

[0093] During the formation control process in the real environment, there are sudden or abrupt changes in environmental factors (such as dynamic obstacles, gusts of wind, communication delays, etc.), resulting in a large deviation between the real environment and the simulation environment, affecting the consistency of the virtual and real environments, and thus not meeting the actual control requirements. To solve this problem, the present invention discusses from the following several situations and gives solutions to eliminate the excessive deviation between the virtual and real environments:

[0094] First, a virtual-real combined semi-physical simulation system is constructed. By combining physical flight controls, self-organizing network communication links, three-dimensional optical motion capture acquisition systems with the simulation platform, data interaction between agent clusters is realized.

[0095] Second, for sudden dynamic obstacles, the artificial potential field method or Bezier curve is used to achieve dynamic obstacle avoidance.

[0096] Third, for the simulation and compensation of the influence of gusts of wind, a wind field model is introduced into the simulation environment to simulate the influence of gusts of wind on agents, and through dynamic semi-physical simulation software, the control strategy of agents is adjusted in real time to compensate for the interference caused by gusts of wind.

[0097] Fourth, for sudden communication delays, a clustering strategy for agent clusters that combines the master-slave and distributed methods is adopted, and the trajectory point data packets planned are transmitted in batches at different times.

[0098] Although the present invention has been disclosed above with preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention all belong to the protection scope of the technical solution of the present invention.

[0099] The content not detailedly described in the specification of the present invention belongs to the well-known technology of those skilled in the art.

Claims

1. A virtual-real interaction simulation method for unmanned clusters based on three-dimensional motion capture, characterized in that, It includes the following steps: Step 1: Establish a real unmanned system platform, including a three-dimensional optical motion capture acquisition system and multiple real agents; Step 2: Run the autonomous navigation and autonomous exploration functions of a single real agent in the real environment, obtain the position and attitude information of the real agent through the three-dimensional optical motion capture acquisition system, obtain image, point cloud, and IMU pose information through the sensors of the real agent itself, and store them; Step 3: Establish a simulation platform with multiple virtual agents and the surrounding environment, run the autonomous navigation and autonomous exploration functions of a single virtual agent in the simulation environment, and obtain the position and attitude information, image, point cloud, and IMU pose information of the virtual agent through the simulation platform; Step 4: Real-time send the data stored in the real environment to the simulation platform through the UDP protocol, and compare and calculate the dynamic error, kinematic error, sensor error, and environmental error; Step 5: In the simulation platform, use the dynamic error, kinematic error, and sensor error to correct the mathematical model of the virtual agent, and use the environmental error to correct the simulation environment model; Step 6: In the real environment and the simulation environment, based on the same world coordinate system, the same number of agents, and the same formation trajectory, first perform multi-robot topological flexible formation simultaneously, and then perform multi-robot cluster autonomous navigation simultaneously, obtain the sensor data sets, motion trajectory sets, speed sets, acceleration sets, position sets, and attitude sets of the real agent cluster and the virtual agent cluster, as well as the environmental data sets of the real agent cluster and the virtual agent cluster; compare and calculate whether the various errors between each virtual agent and the real agent exceed the set threshold. If there is at least one error exceeding the threshold, re-execute Steps 3-5, and perform virtual-real interactive iterative training until the error converges.

2. The virtual-real interaction simulation method for unmanned clusters based on three-dimensional motion capture according to claim 1, characterized in that, The position and attitude information includes XYZ position coordinates, motion trajectory, six-degree-of-freedom attitude, robot joint angles, speed, acceleration, quaternion, and Euler angles.

3. A virtual-real interaction simulation method for unmanned clusters based on three-dimensional motion capture according to claim 1, characterized in that, The environmental data set includes wind speed, air pressure, obstacle distribution, and terrain.

4. A virtual-real interaction simulation method for unmanned clusters based on three-dimensional motion capture according to claim 1, characterized in that, The real-time sending of the data stored in the real environment to the simulation platform through the UDP protocol is specifically as follows: The position and attitude information of the real agent is stored in the three-dimensional optical motion capture acquisition system, and the position and attitude information of the agent stored in the real environment is sent to the simulation platform through the UDP protocol; The real agent runs the Robot Operating System (ROS), and the sensor data of the agent itself is stored in ROS. The sensor data of the real agent itself is sent to the simulation platform in real time through the form of ROS topic publishing and subscribing.

5. A virtual-real interaction simulation method for unmanned clusters based on three-dimensional motion capture according to claim 4, characterized in that, When the real unmanned system platform and the simulation platform perform virtual-real interactive simulation, the following two aspects of problems also need to be solved: In terms of spatial alignment: Define the world coordinate system and body coordinate system of the real agent, and define the world coordinate system and body coordinate system of the virtual agent. Both the world coordinate system and the body coordinate system include the axis direction and the origin position; Use a calibration board to calibrate the world coordinate system and body coordinate system of the real environment and the simulation environment, and calculate the rotation matrix and translation vector from the real environment coordinate system to the simulation environment coordinate system according to the coordinate system differences between the real environment and the simulation system; In terms of data time synchronization: Synchronize the clocks of the virtual environment and the real environment to the same clock source, add timestamps to the data packets collected by motion capture, and perform data interpolation or delay compensation in the simulation environment according to the timestamps to ensure the real-time and consistency of the data; In addition, it is also possible to ensure that the data collection and simulation of the real agent and the virtual agent are in step by using a synchronization signal generator.

6. A virtual-real interaction simulation method for unmanned clusters based on three-dimensional motion capture according to claim 1, characterized in that, The comparison and calculation of dynamic error, kinematic error, sensor error, and environmental error are as follows: Dynamic error: Adopt the difference based on equation solving, establish a dynamic equation according to Newton's second law and the law of rotation, set the same control input and initial conditions, numerically solve their respective dynamic states, and calculate the difference of the corresponding state variables as the error, including acceleration error and angular velocity error; Kinematic error: For the attitude angle error, compare the pitch angle, roll angle, and yaw angle change curves of the virtual agent and the real agent within the same time period, and calculate the mean absolute error; Sensor error: Based on data statistics, for the same physical quantity, including image, point cloud, and IMU pose information, measure the output values of the real sensor and the virtual sensor multiple times under the same environment and the same agent state, and calculate the mean value of the output error as the sensor error; Environmental error: For wind speed error, set a wind speed function V1(t) that changes with time in the virtual environment, measure the real wind speed V2(t) at the corresponding moment with an anemometer in the real environment, and calculate the average error between V1(t) and V2(t) over a period of time; For air pressure error, set a virtual air pressure function P1 that changes with altitude and position in the virtual environment, measure the corresponding real air pressure P2 with a barometer in the real environment, analyze the change of air pressure with altitude and position, and calculate the average error between P1 and P2 in different altitude layers or regions.

7. A virtual-real interaction simulation method for unmanned clusters based on three-dimensional motion capture according to claim 6, characterized in that, The use of dynamic error, kinematic error, and sensor error to correct the mathematical model of the virtual agent includes: Dynamic model correction: Adjust the acceleration, angular velocity, and aerodynamic coefficient of the virtual agent according to the dynamic error; Kinematic model correction: For trajectory correction, adjust the attitude controller parameters of the virtual agent according to the attitude error, so that the pitch, roll, and yaw angles of the virtual agent are closer to the real values; Sensor model correction: For sensor error, calibrate and compensate the noise parameters in the virtual sensor model to make the image, point cloud, and IMU pose information output by the virtual sensor more realistic.

8. A virtual-real interaction simulation method for unmanned clusters based on three-dimensional motion capture according to claim 6, characterized in that, The use of environmental error to correct the simulation environment model includes: Wind speed model correction: Adjust the wind speed model parameters in the virtual environment according to the wind speed error, including adjusting the mean value, variance or turbulence intensity of the wind speed to make it more consistent with the wind speed in the real environment; Air pressure model correction: Adjust the air pressure model in the virtual environment according to the air pressure error, considering the influence of altitude and geographical location factors on the air pressure to make the virtual air pressure closer to the real value.

9. A virtual-real interaction simulation method for unmanned clusters based on three-dimensional motion capture according to claim 1, characterized in that, When performing multi-aircraft topological flexible formation, design the following objective function: Φ = min c,T [J t , J o , J f , J r , J d , J u ·λ where λ is the weight vector that weighs each cost function, and the respective cost functions are as follows: total time J t , which represents the motion time consumed by a trajectory; agent obstacle avoidance J o , which represents the distance between the planned trajectory points and the nearest surrounding obstacles; formation difference degree J f , which represents the difference degree between the current formation shape and the desired formation shape; mutual collision avoidance among agents J r , which represents the cost function for collision avoidance among agents, calculated by the sum of the distances between the current trajectory points and the trajectory points of all other agents; dynamic feasibility J d , which represents that the planned trajectory is reasonable and does not exceed the limits of the agent's own speed and acceleration; uniform distribution of constraint points J u , which represents the degree of uniform distribution of all trajectory points.

10. A virtual-real interaction simulation method for unmanned clusters based on three-dimensional motion capture according to claim 1, characterized in that During the formation control process in the real environment, there are sudden or abrupt changes in environmental elements, including dynamic obstacles, gusts, and communication delays. The solutions adopted include: First, build a virtual-real hybrid semi-physical simulation system. By combining the physical flight control, self-organizing network communication link, and three-dimensional optical motion capture acquisition system with the simulation platform, data interaction between agent clusters can be achieved; Second, for sudden dynamic obstacles, use the artificial potential field method or Bezier curve to achieve dynamic obstacle avoidance; Third, for the simulation and compensation of the influence of gusts, introduce a wind field model in the simulation environment to simulate the influence of gusts on agents, and through the dynamic semi-physical simulation software, adjust the control strategy of agents in real time to compensate for the interference caused by gusts; Fourth, for sudden communication delays, adopt an agent cluster clustering strategy that combines the master-slave and distributed methods, and transmit the planned trajectory point data packets in batches at different times.

Images