Unmanned aerial vehicle semi-physical simulation test platform and method based on digital twinning

Abstract

The invention belongs to the technical field of digital twinning and semi-physical simulation, and discloses an unmanned aerial vehicle semi-physical simulation test platform and method based on digital twinning, and the simulation test platform comprises a physical unmanned aerial vehicle which is a real unmanned aerial vehicle and flies in a real target environment; wherein a first flight controller, a camera, a laser radar, an airspeed meter, a GPS positioning module, an IMU module and an airborne computer are carried on the physical unmanned aerial vehicle; the GPS positioning module and the IMU module are respectively connected with the first flight controller, and the first flight controller, the camera and the laser radar are respectively connected with the airborne computer; the physical host is deployed with a twin domain and comprises a twin environment and a twin unmanned aerial vehicle; the twin environment is a simulation environment the same as a real target environment, and the twin unmanned aerial vehicle is a digital simulation model of the physical unmanned aerial vehicle; the second flight controller is connected to the physical host and used for controlling the twin unmanned aerial vehicle; and the airborne computer and the physical host carry out information interaction through a wireless network.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital twin and semi-physical simulation, and in particular relates to a semi-physical simulation test platform and method for unmanned aerial vehicle (UAV) based on digital twin. Background Art

[0002] Drones are currently being integrated into various industries, such as express delivery, forestry inspections, river inspections, and aerial photography. When drones perform autonomous patrol missions, they typically fly according to pre-planned routes. Actual flight scenarios are often complex and uncertain, especially in low-altitude urban environments. A drone encountering a flying bird or floating garbage can easily cause a collision, potentially damaging the drone.

[0003] Digital twin technology has also seen rapid development in recent years. Leveraging data from physical models, sensor updates, and operational history, it integrates multidisciplinary, multi-physical, multi-scale, and simulation processes to map objects in virtual space, reflecting the entire lifecycle of corresponding physical equipment. Simulation and visualization in virtual space, coupled with 5G network signals for rapid communication between the virtual and real worlds, enable virtual-real mapping and mutual control. This emerging technology offers new insights into the development of simulation-based control for drones.

[0004] In the current public technical literature, the application publication number CN118259600A discloses a method of changing the built-in environmental interference model to receive external environment, integrating meteorological and geographical information, so that the UAV operator can visually feel the environmental changes. However, it only uses actual data to estimate the simulation posture, and the posture presented is still the software-estimated posture, which cannot fundamentally reduce the error with the real machine posture. The application publication number CN116048118A discloses a UAV formation virtual-real collaborative experimental platform and control method based on digital twins, in which the obstacles are preset and stationary, and the UAV must hover and wait before avoiding the obstacle. It does not have the randomness of the obstacle appearance and the UAV's emergency rapid response function, which actually simplifies the UAV flight environment. In the document "Li Can, Hou Xingming, Qi Qiming, et al. UAV parallel combat simulation based on digital twins. Firepower and Command Control, 2023, 48(08): 23-31", target recognition and tracking under full virtual conditions are realized, but real machine control experiments are not realized, and virtual-real collaborative control is not realized by using real machine data as input.

[0005] The above methods each have their own advantages and disadvantages, both in simulation and virtual-reality collaborative control. For UAV simulation, the authenticity is poor without actual environmental data. For virtual-reality collaborative control, the stability of control cannot be verified without real-machine comparison. Most UAV virtual-reality collaborations are full software-in-the-loop simulations, which lack the participation of real environments or real-machine flight comparisons in experimental effectiveness verification. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the existing technology and provide a semi-physical simulation test platform for UAVs based on digital twins, which combines digital twins and semi-physical simulation in the actual measurement and verification of real UAVs. The hardware-in-the-loop combines the real environment data collected by the real machine to generate more realistic attitude control signals, making the attitude control of the twin UAVs more realistic.

[0007] The present invention also provides a semi-physical simulation test method for drones based on digital twins, which combines the sensor information of real machines in the twin domain to construct a twin environment that simulates the real environment in real time, making the simulation of twin drones closer to reality, and realizing virtual-reality collaboration based on digital twins by establishing two-way information communication between the real machine and the twin drone.

[0008] The technical solution provided by the present invention is: A UAV hardware-in-the-loop simulation test platform and method based on digital twins, comprising: Physical drones, which are real drones flying in real target environments; The physical drone is equipped with a first flight controller, a camera, a laser radar, an airspeed meter, a GPS positioning module, an IMU module, and an onboard computer; the GPS positioning module, the IMU module, and the airspeed meter are respectively connected to the first flight controller, and the first flight controller, the camera, and the laser radar are respectively connected to the onboard computer; A physical host, on which a twin domain is deployed, wherein the twin domain includes a twin environment and a twin drone; The twin environment is a simulation environment identical to the real target environment, and the twin drone is a digital simulation model of the physical drone, flying in the twin environment. a second flight controller connected to the physical host, the second flight controller being used to control the twin drones; The onboard computer and the physical host exchange information via a wireless network.

[0009] A digital twin-based UAV hardware-in-the-loop simulation test method, using the digital twin-based UAV hardware-in-the-loop simulation test platform, includes: Build twin drones; The physical host sends a flight waypoint sequence to the onboard computer, and the onboard computer transmits the flight waypoint sequence to the first flight controller; The first flight controller enables the physical UAV to fly in a real target environment according to the flight waypoint sequence, while collecting real target environment information in real time through a camera and a lidar, and collecting position information and attitude information of the physical UAV in real time through a GPS positioning module and an IMU module; The physical host receives the real target environment information, the position information and the attitude information of the physical UAV, and constructs a twin environment in real time based on the real target environment information. At the same time, the position information and attitude information of the physical UAV are transmitted to the second flight controller. The second flight controller controls the flight of the twin UAV so that the twin UAV maps the flight status of the physical UAV. A moving virtual obstacle is constructed in the twin environment. The second flight controller determines whether there is a collision risk between the twin drone and the virtual obstacle based on the virtual obstacle information. If it is determined that there is a collision risk, the flight waypoint sequence is updated and the twin drone is controlled to fly according to the updated flight waypoint sequence. At the same time, the second flight controller transmits the updated flight waypoint sequence to the first flight controller via the physical host and the onboard computer, and controls the physical drone to map the flight status of the twin drone.

[0010] Preferably, the method for constructing a twin environment is: Perform color correction and lens distortion correction on the photos taken by the camera, and fuse the corrected photos with the point cloud data collected by the lidar to generate dense point cloud data with texture; Generate a high-fidelity mesh model using the dense point cloud data and export it to fbx format; After optimizing and repairing the texture of the FBX mesh model, adjust the model scale, import the regional DEM data to generate the terrain base, and align the building model to the terrain according to GPS coordinates; Set the sun angle according to the latitude and longitude and time of the real area, and integrate the Ultra Dynamic Sky plug-in to achieve rain and snow effects.

[0011] Preferably, the method for determining whether there is a collision risk between the twin drones and the virtual obstacle is: Obtain the location information and moving speed of virtual obstacles; Calculating future moments The distance between the twin drone and the virtual obstacle ; ; If there is a future moment , making , it is judged that there will be a collision between the twin drones and the virtual obstacle; in, Requirements: ; Where, Indicates the current moment, is the predicted duration set based on the system response time, is the position vector of the twin UAV in the twin environment, is the position vector of the virtual obstacle in the twin environment; is the relative velocity vector between the twin UAVs and the virtual obstacle.

[0012] Preferably, the process of mapping the flight status of the twin drone by the physical drone further includes: The wind force and wind speed in the real target environment are collected through the airspeed meter and transmitted to the twin environment; the second flight controller obtains the wind force and wind speed information, calculates the wind compensation amount based on the wind force and wind speed information; and converts the wind compensation amount into control instructions to control the flight posture of the twin UAV, and at the same time maps the twin UAV flight posture to the physical UAV.

[0013] Preferably, the calculation formula for the wind compensation amount is: ; Where, is the air density, is the resistance coefficient, represents the projection area of ​​the twin UAVs on the plane perpendicular to the wind direction, is the wind speed vector in the twin UAV body coordinate system.

[0014] Preferably, if it is determined that there is a risk of collision between the twin drone and the virtual obstacle, it is determined whether the twin drone can make a local detour; Among them, when ,and When the twin UAVs are judged to be able to make a partial detour, the second flight controller updates the waypoints according to the partial detour method; Where, Indicates the collision radius of the virtual obstacle, represents the safety radius of the twin drones, Indicates the twin UAV heading deviation angle, Indicates the maximum allowed heading deviation angle of the twin drones; If it is determined that the twin UAVs cannot perform local detours, the second flight controller updates the waypoints according to the global path replanning method.

[0015] Preferably, the UAV hardware-in-the-loop simulation test method based on digital twins further includes: The virtual camera in the twin domain acquires target image data and transmits the target image data to the second flight controller. The second flight controller performs target recognition based on the target detection algorithm and controls the twin UAV to track the target. At the same time, the target tracking status is mapped to the physical UAV through the onboard computer and the first flight controller.

[0016] The beneficial effects of the present invention are: This invention uses digital twin technology to proportionally model the physical parameters of physical drones and to proportionally model real scenes in real time. It combines dynamic weather simulation with random obstacle generation to significantly improve the simulation environment's restoration of complex low-altitude scenes, making the test results closer to actual flight conditions.

[0017] The semi-physical simulation test method for UAVs based on digital twins provided by the present invention realizes bidirectional data mapping and counter-control between the physical domain and the twin domain based on ROS wireless network communication, uses real sensor data and real-time wind disturbances to drive the virtual model with physical rules, and simultaneously detects obstacles in the virtual domain and generates obstacle avoidance instructions, which are fed back to the physical UAV in real time, forming a closed-loop control and significantly reducing the attitude adjustment error between the virtual and real machines. The collision risk is predicted in real time through lidar data and kinematic models, and combined with hardware-in-the-loop (HIL) simulation technology, the UAV can quickly generate obstacle avoidance instructions when encountering sudden interference without hovering and waiting, thereby enhancing the emergency response capability and flight safety in complex environments. Hardware-in-the-loop (HIL) simulation is combined with virtual target recognition and tracking technology to test high-risk tasks (such as dense obstacle avoidance and extreme weather tracking) in a twin environment, without having to worry about the physical UAV colliding with the detection target. It provides an efficient and low-risk solution for the development, testing and optimization of UAV systems.

[0018] The present invention uses a real flight control system to control the twin UAV through a hardware-in-the-loop setting. The binary code execution of the real flight control system interacts with the sensor hardware. By injecting the airspeed meter and IMU data in the real environment into the HIL simulation in real time, key physical interferences such as sensor noise and communication delay are accurately reproduced, which greatly improves the simulation environment's restoration of real flight conditions. Fault injection and response testing based on real flight control hardware (such as disabling the TECS wind compensation module) provides a fault reproduction capability that is crucial in the aviation field and effectively verifies the system robustness under extreme conditions. The control instructions verified in the simulation environment can be directly applied to the physical UAV, eliminating the essential difference between software simulation and hardware execution, and ensuring the consistency of virtual and real control.

[0019] The present invention integrates APM flight control, ROS protocol and multiple sensors (such as GPS, IMU, LiDAR), supports mainstream simulation engines (UE Chaos, JSBSim), ensures seamless interaction between hardware and simulation environment, and verifies the stability of control algorithms in actual hardware. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a framework diagram of the UAV semi-physical simulation test platform based on digital twins described in the present invention.

[0021] Figure 2 Schematic diagram of the APM hardware-in-the-loop simulation environment described in the present invention.

[0022] Figure 3 This is a flow chart of the obstacle collision detection algorithm described in the present invention.

[0023] Figure 4 This is a flow chart of the hybrid obstacle avoidance algorithm of global path replanning and local obstacle avoidance in the present invention.

[0024] Figure 5 This is a flowchart of the local detour obstacle avoidance described in the present invention.

[0025] Figure 6 This is the global path replanning and obstacle avoidance flow chart of the present invention.

[0026] Figure 7 This is a flowchart of the virtual-real collaborative target recognition and tracking described in the present invention. DETAILED DESCRIPTION

[0027] The present invention will be described in further detail below in conjunction with the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0028] The present invention provides a UAV semi-physical simulation test platform based on digital twin. Figure 1As shown, the digital twin-based drone hardware-in-the-loop simulation test platform comprises a physical domain and a twin domain. The twin domain, built using the UE Unreal Engine, includes a twin drone, a twin environment, virtual obstacles, and a virtual-real data interaction module. The physical domain includes a physical drone, a first flight controller, a GPS positioning module, a camera, a lidar, an airspeed meter, an IMU sensor, an onboard computer (Jetson Nano), a physical host (computer), and a second flight controller. Both the first and second flight controllers utilize APM flight control (hardware). The physical drone is a real drone, flying in a real target environment. The camera and lidar are directly connected to the onboard computer via a serial port protocol. The other sensors (GPS positioning module, IMU module, airspeed meter) are first connected to the first flight controller via a serial port protocol, and then the first flight controller is connected to the onboard computer via a serial port protocol. The onboard computer utilizes Wi-Fi to communicate with the physical host using the MAVLink communication protocol within the same local area network. The twin domain is deployed on the physical host. The twin environment is a simulated environment identical to the real target environment, modeled proportionally based on the real scenario. The twin drone is a high-precision digital model of the real drone. AirSim integrates numerous sensor functions internally, calling the corresponding APIs of the sensors on the physical drone as the corresponding virtual sensors in the twin domain. Here, the APIs corresponding to the GPS positioning module, camera, lidar, airspeed meter, and IMU sensor are enabled to enable the virtual sensors. A second flight controller is connected to the physical host and is used to control the twin drone.

[0029] The data between the physical domain and the twin domain exchange information through a wireless network; the virtual-reality interaction module is a communication hub connecting the physical domain and the twin domain, and is used to realize two-way, low-latency transmission of real data and twin data.

[0030] The ground station (Misson Planner) is software deployed on a physical host. It is the key human-computer interaction interface and mission management hub for drone mission planning, monitoring, and command issuance. The ground station's main functions include: 1) Mission Planning: Receives user-defined waypoint sequences ([WP1, WP2 … WPN]) and uploads them to the primary flight controller via the MAVLink protocol. 2) Status Monitoring: Receives and displays real-time status information from the physical drone (via its onboard computer) and the twin domain (e.g., drone position, attitude, battery voltage, and sensor readings). 3) Command Issuance: When needed (e.g., for global path replanning), receives new waypoint sequences ([WP1, WP_new1 … WP_newM]) generated by the twin domain and transmits them via MAVLink to the physical drone's onboard computer (Jetson Nano) or directly to the primary flight controller, guiding the drone to execute the updated route. 4) Parameter Configuration: Parameters for the secondary flight controller (APM) (e.g., enabling HIL mode (HIL_MODE=1) and setting the serial communication protocol (SERIALx_PROTOCOL=2). As the operations center of the physical domain, the ground station is the key link connecting user intent, physical drone execution, and twin domain simulation verification.

[0031] The virtual-reality interaction module in the onboard computer converts the physical domain ROS messages (sensor information format) into MAVLink protocol standard messages in real time, and communicates with the physical host under the same local area network; the virtual-reality interaction module in the physical host parses the messages and converts the real wind field data collected by the airspeed meter into AirSim wind field parameters (airsim_client.simSetWind()) in real time for the second flight controller to process and generate MAVLink control instructions; then the generated MAVLink control instructions (such as SET_ATTITUDE_TARGET) are converted into ROS control topics (such as / sim_controller / cmd) to drive the twin environment physics engine and realize two-way protocol parsing.

[0032] On the twin domain side: access the twin environment by calling the AirSim API to obtain the wind field and obstacle status, and access the twin drone model through the UE blueprint system.

[0033] This data interaction based on the ROS framework ensures the real-time transmission of sensor data, control commands, and environmental information. It also guarantees the compatibility and parsing efficiency of cross-domain commands, which is the key foundation for closed-loop control.

[0034] The twin domain can provide visualization and virtual-reality collaboration capabilities, specifically including virtual-reality collaborative obstacle avoidance and virtual-reality collaborative target recognition and tracking. When generating twin obstacles for virtual-reality collaborative obstacle avoidance testing, the physical host and the second flight controller issue control instructions based on virtual obstacle information and real-world environment information, thereby achieving virtual-reality collaborative obstacle avoidance. When performing virtual-reality collaborative target recognition and tracking, the twin machine can obtain images of the simulated environment and perform target recognition and tracking using the YOLO model. The second flight controller (APM) generates motor outputs based on YOLO control instructions and sensor information, thereby completing virtual-reality collaborative target recognition and tracking of the physical machine mapping the twin machine.

[0035] The present invention also provides a semi-physical simulation test method for UAV based on digital twins, and the specific implementation process is as follows.

[0036] S1. Build a twin environment and twin drone model In order to reflect the authenticity of the simulation platform and the accuracy of the control, the real target environment of the physical drone flight and the physical drone will be finely modeled.

[0037] To build a high-fidelity twin environment model, the aircraft uses a high-resolution camera to capture high-resolution images, while also carrying a lightweight LiDAR (Light Detection and Ranging) sensor to generate millimeter-level precision point cloud data. After acquiring the high-precision data, preprocessing is required before further model construction. Adobe Lightroom is used to color correct and correct for lens distortion in photos, which are then fused with the LiDAR point cloud using RealityCapture to generate a dense, textured point cloud. This dense point cloud is imported into RealityCapture to generate a high-fidelity mesh model, which is then exported to .fbx format. After further texture optimization and restoration, the model is imported into Unreal Engine (UE), where the model scale is adjusted. Using the Landscape tool, regional DEM data is imported to generate a terrain base, and the building model is aligned to the terrain using GPS coordinates. The sun angle is set based on the real-world latitude, longitude, and time of day. The Ultra Dynamic Sky plug-in is integrated to create rain and snow effects, which affect the twin drone's dynamics.

[0038] Create a CAD file in SolidWorks and fine-tune the exterior shell and visible components. Use the Quixel Megascans library to match real-world materials. When exporting to .fbx format, ensure the scale and units match the UE Unreal Engine default. The AirSim original model's default location is: AirSim\Unreal\Environments\Blocks\Content\VehicleAdv\Car\Sedan\SedanSK.uasset. Copy the .fbx file to this location in UE Unreal Engine and configure its physics properties. Add a "Physics Asset" and create capsule colliders for each propeller and fuselage, matching the real-world physical outlines. Set the mass distribution and center of gravity to match the real-world drone's position. Adjust the drag coefficient in "Project Settings → Physics." Replace the original AirSim Blueprint, maintaining the same skeleton structure and slot names (e.g., Thruster_1, Camera_Base) to ensure compatibility with the inherited AirSim control logic. Adjust the thrust parameters and modify the RotorThrust and RotorTorque in the AeroBody component to match the real-world motor parameters.

[0039] S2, second flight controller (APM) hardware-in-the-loop simulation environment construction The purpose of the hardware-in-the-loop (HIL) of this invention is not to verify the performance of the flight controller in wind disturbances or complex environments, but to make the HIL simulation more realistic. By injecting the airspeed meter and IMU data in the real environment into the HIL simulation in real time, the second flight controller can autonomously adjust the control instructions based on the sensor input under real disturbances, so that the twin drone mapping is closer to the real behavior of the physical drone.

[0040] Hardware-in-the-loop (HIL) simulation environments such as Figure 2 As shown, configure the firmware parameters for the secondary flight controller (APM). Use Mission Planner to flash the ArduCopter firmware to the flight controller hardware, selecting a version that supports HIL. Set the MissionPlanner parameters to: HIL_MODE = 1: Enable HIL simulation mode; SERIALx_PROTOCOL = 2 (MAVLink protocol) to specify the serial communication port; and SIM_ENABLE = 1: Enable simulation data input.

[0041] The physical drone is equipped with Jetson Nano as an onboard computer, responsible for collecting various sensor data, such as: airspeed meter (measuring true airspeed) and IMU (collecting three-axis acceleration and angular velocity).

[0042] Establish bidirectional communication between the onboard computer (Jetson Nano) and the physical host based on ROS. Configure the ROS network. Install ROS on the Jetson Nano and the physical host. Set ROS_MASTER_URI to the IP address of the physical host and ensure that both are on the same network.

[0043] The onboard computer (Jetson Nano) reads sensor data (IMU, GPS, camera, etc.) through a ROS node and encapsulates the data, preferably using protobuf to reduce transmission overhead. Based on the ROS publish-subscribe model, a ROS node is initialized, named 'nano_ros_node' (unique), and a ROS handle is instantiated. The publisher ' / real_drone / sensors' is instantiated to publish sensor data, and the receiver ' / sim_controller / cmd' is instantiated to subscribe to control commands from the physical host to adjust the attitude and achieve virtual-real collaboration.

[0044] The AirSim physical host initializes a ROS node, names it 'host_ros_node' (unique), and instantiates a ROS handle. It also instantiates the publisher ' / sim_controller / cmd' to publish control commands, instantiates the receiver ' / sim_controller / cmd' to subscribe to the Jetson Nano sensor data, and converts the received sensor data into a format recognizable by AirSim to complete sensor mapping.

[0045] Add high-precision timestamps to messages and predict network delays in control algorithms (such as using sliding window filtering) for time synchronization and delay compensation.

[0046] The physical host converts the received sensor data into a format that AirSim can recognize.

[0047] AirSim integrates numerous sensor functions. By calling the corresponding APIs for the sensors onboard the physical drone, you can enable virtual sensors. Here, we enable the APIs for the GPS positioning module, camera, lidar, airspeed meter, and IMU sensors as corresponding virtual sensors in the virtual domain.

[0048] To simulate a dynamic wind field, inject real airspeed data: client.simSetAirspeed(real wind speed value); AirSim wind disturbance modeling, using the AirSim API to update the wind field in real time to achieve dynamic wind field simulation. Convert the real anemometer data to the MAVLink HIL_SENSOR message format.

[0049] When twin drones fly in the AirSim virtual environment (twin environment), the TECS (Total Energy Control System) module in the secondary flight controller (APM) connected to the physical host computer loses its wind compensation function due to a lack of airspeed input. Modifying the secondary flight controller (APM) firmware disables the built-in TECS wind compensation module, bypassing physical sensor limitations. AirSim-calculated wind vectors are directly injected into the secondary flight controller via MAVLink #500 messages, ultimately correcting attitude based on a wind drag torque model.

[0050] The AirSim wind field model on the twin environment side generates the wind field, and the WindSim engine calculates the precise wind vector and encapsulates it as a MAVLink#500 message and sends it to the second flight controller.

[0051] Since the onboard airspeed meter of the second flight controller fails in the HIL environment, the compensation arbitrator will disable the TECS wind compensation inside the flight controller and use the virtual wind field data from AirSim for compensation instead.

[0052] The core idea of ​​the attitude correction based on the wind resistance torque model is to calculate the torque generated by the wind on the drone, and then generate a feedforward compensation signal to be superimposed on the output of the attitude controller to offset the influence of the wind. The wind resistance torque model mainly considers the force and torque generated by the wind in the body coordinate system. It can be approximated as: ; Where, is the air density, is the resistance coefficient, represents the projection area of ​​the twin UAVs on the plane perpendicular to the wind direction, is the wind speed vector in the twin UAV body coordinate system.

[0053] The moment is caused by the misalignment between the wind force point and the center of gravity. Assume that the positional deviation between the center of pressure (CP) and the center of gravity (CG) is , then the wind resistance moment is: ; According to the kinetic equation: ; in, is the moment of inertia matrix, is the angular rate, represents the angular acceleration, is the control torque (output by the controller), is the wind resistance torque.

[0054] In order to offset the wind resistance torque, we hope that the control torque can produce a The opposite torque, ;in, is the torque calculated by the feedback controller (PID) in the second flight controller.

[0055] Feedforward compensation The compensation amount is added to the output of the PID controller in the second flight controller to obtain the corrected total torque output of the servo. , and finally convert the total torque into motor output command; Therefore, this control command implicitly compensates for wind forces, counteracting wind disturbances by adjusting attitude and thrust. The wind in the simulation environment continuously acts on the drone's dynamic model. AirSim drives the drone according to the APM's control commands. The resulting twin drone's attitude is determined by both the APM's control commands (which already include wind resistance adjustments) and the physical effects of wind disturbances, completing a high-fidelity mapping of the physical drone.

[0056] S3. Communication link configuration The second flight controller (APM) is connected to the physical host computer running the simulation software via USB. Configure the UDP port (default 14550) to ensure a two-way communication link between the APM and the simulator.

[0057] S4. UAV flight test process The drone's cruise state is defined as three states: normal cruise, obstacle testing, and target identification and tracking. When the physical drone is in normal cruise mode, the UE Unreal Engine transmits sensor data (such as IMU and GPS) collected by the physical drone's onboard computer to the secondary flight controller (APM) via the HIL_STATE_QUATERNION message. The secondary flight controller (APM) then runs autonomous control logic (executing pre-set waypoint missions). Based on the sensor data, it autonomously generates a SET_ATTITUDE_TARGET message and sends it to the UE Unreal Engine to control the drone's attitude, enabling the twin drone to mirror the physical drone in real time within the twin environment. When the twin drone's virtual radar detects an obstacle, the twin drone's information, combined with real-world environmental information collected by the physical drone, serves as input to the secondary flight controller (APM). The control decisions made by the secondary flight controller (APM) after parsing the data serve as instructions for subsequent attitude updates of the twin and physical drones in the UE Unreal Engine, enabling the twin drone to control the physical drone.

[0058] The specific process of the UAV flight test is as follows: (1) Initial task setting At time 0 of the test, the ground station uploads the cruise track ([WP1, WP2 …WPN]) consisting of N waypoints of the flight mission to the physical drone. Represents the latitude and longitude information of the waypoint. The waypoint is transmitted to the physical drone via the ground station using the MAVLink protocol. The physical drone starts and performs the flight mission. The twin drone is mapped in real time, and the twin drone status is defined: The position vector of the twin drone in the twin environment: (Unit: m); Velocity vector of the twin drone in the twin environment: (Unit: m / s) Safety radius of twin drones: (Unit: m, this radius includes the physical size of the drone itself, as well as a small additional safety buffer distance) Generate obstacles in the twin environment and make them move in a fixed direction. Define the obstacle state: Position vector of the virtual obstacle in the twin environment: (Unit: m); Velocity vector of the virtual obstacle in the twin environment: Unit: m / s Collision radius of virtual obstacles: (Unit: m, which can be understood as the size range of the obstacle) The twin drones monitor their surroundings through virtual radar. The radar detection range radius is defined as (Unit: m). When an obstacle enters this monitoring range (i.e. ), the system will start the collision detection algorithm (see the process Figure 3 ), to determine whether the twin drones are likely to collide with the obstacle in the future.

[0059] In the short forecast window Inside( For the current moment, is the predicted duration set according to the system reaction time), assuming that the drone and the obstacle maintain the current velocity vector and move in a uniform straight line. ; Relative displacement vector between the twin drone and the virtual obstacle ; ; Relative velocity vector between the twin drones and the virtual obstacle ; ; The distance between the twin drone and the virtual obstacle (scalar) ;

[0060] The condition for a collision to occur is: there is a future moment , making To efficiently detect collision risks, calculate the distance The minimum point of During the derivation process, we introduced a time offset variable Take the derivative and set it to zero to find the critical time ; ; Calculating the prediction window The actual minimum spacing within ; like (The minimum value appears in the past): The current distance is increasing, take ; like :but ; like (The minimum value appears after the future prediction window): then take ; Collision detection: like and (i.e. the minimum distance appears within the effective prediction window), then it is determined that there is a collision risk (estimated collision time ).

[0061] above Indicates the minimum distance between the twin drones and the virtual obstacle, Indicates the critical time point at which the calculated distance reaches the minimum. represents the predicted collision time, Represents a time offset variable introduced to simplify the derivation process.

[0062] After a potential collision is detected, the flight path needs to be dynamically adjusted to avoid obstacles while staying as close to the original mission route as possible. Figure 4 As shown in the figure, it is a hybrid algorithm process that combines global path replanning and local obstacle avoidance: Determine whether a local detour is possible: The judgment condition is , and the deviation angle ; Where, Indicates the collision radius of the virtual obstacle, represents the safety radius of the twin drones, Indicates the heading deviation angle of the drone. Indicates the maximum allowed heading deviation angle of the drone (limited by the rotation capability of the drone); Local detour obstacle avoidance Figure 5 As shown, the algorithm here is not specifically limited and any algorithm can be embedded and used. The dynamic window method (DWA) can be preferably used. Its core idea is to dynamically evaluate all possible motion trajectories in the short term within the range allowed by the physical limitations of the drone and select the safe path with the highest comprehensive score. First, a dynamic window (physical constraint filtering) is constructed to calculate the instantaneous achievable speed set based on the current state of the drone and the hardware limitations. ; ; Where, is the forward speed of the UAV (m / s), is the UAV steering angular velocity (rad / s), is the acceleration (m / s²), is the angular acceleration (rad / s²); the subscripts min and max indicate the minimum and maximum values ​​of each parameter, respectively.

[0063] Dynamic window Each speed pair within Predicting the future The motion trajectory in time and detects whether the trajectory collides with a virtual obstacle: ,in, Indicates the minimum distance between the trajectory and the obstacle; The best speed instruction is then selected by a weighted scoring function: ; in, Indicates the cosine value of the deviation angle between the end point of the trajectory and the original heading. , , Are weight coefficients, used to optimize To maintain the direction of the route and optimize To maximize travel speed and optimize To stay away from obstacles; Finally, the speed pair with the highest score is selected as the control instruction to update the speed instruction: ; At the same time, the AirSim API in the twin environment updates the wind field in real time, and the wind force continues to act on the drone dynamics model. The second flight controller combines the wind field information with the updated speed instructions and gives the final motor control instructions based on the above wind resistance torque model.

[0064] AirSim receives the instructions and issues control instructions through ROS to the onboard computer, which then transmits them to the first flight controller. The first flight controller controls the physical drone according to the received instructions to complete virtual-reality collaboration and achieve virtual-reality collaborative local detour and obstacle avoidance.

[0065] If no collision risk is detected, the aircraft returns to the original route.

[0066] When there is a risk of collision but local detour is not possible, such as Figure 6 As shown, global path replanning (RRT*-Connect) is optimally performed. When a drone encounters a large obstacle (such as a building or mountain) that it cannot circumvent, it must completely change its original route. Global path replanning (RRT*-Connect) is used to quickly generate a new, safe, and as short a route as possible within the twin environment, while remaining close to the original plan.

[0067] In this embodiment, the RRT*-Connect method, an algorithm for robot path planning, which is an improved version of the Rapidly-exploring Random Tree (RRT) algorithm, is used to perform global path replanning.

[0068] First define the search space: abstract environment map (including dynamic obstacle information) Where, It is the twin environment map boundary of the drone activity, for example, a 100m×100m×50m airspace; Represents three-dimensional real space, that is, the space composed of three real coordinate axes (x, y, z).

[0069] Starting tree The root node is the current drone position , target tree The root node is the next waypoint of the original path ; A target point is randomly selected (80% probability of throwing it in the direction of the end point, 20% pure random) and a bidirectional RRT tree is grown. Each tree performs a search for the nearest neighbor. If there is no collision obstacle, a new node may be generated. This is repeated until the two trees are adjacent to each other, which is the initial exploration path.

[0070] After finding the path, optimize it. Because the initial exploration path is very messy with many forks, you need to remove unnecessary inflection points on the path like stretching a rubber band, while ensuring that there are no collisions.

[0071] Optimization principles: Keep the total distance as short as possible (avoid detours) The new route should be as close to the original planned route as possible (minimize deviations) The cost function is designed as follows: ; in, is the total path length, For new path With the original path Hausdorff distance (path similarity), Is the weight coefficient, used to adjust the total length of the path and path similarity The importance of these two factors in the total cost. Engineers adjust these two factors based on actual needs (whether shortening the distance or minimizing deviation from the original plan is more important).

[0072] Finally, the optimized path is generated The optimized path is converted into waypoints WP_new ([WP1, WP_new1…WP_newM]). The new waypoints are then transmitted from the ground station to the host computer. The physical drone no longer requires precise control; it simply flies according to the new waypoints. At this point, the twin drones mirror the physical drones. After completing obstacle avoidance, the system detects no collision risk and returns to the original route. This completes obstacle detection and virtual-reality collaborative obstacle avoidance.

[0073] Start target recognition and tracking mode Figure 7 As shown, in this mode, the AirSim virtual camera is used to acquire target image data, and the image information is transmitted to YOLO for target recognition and tracking. The YOLO model calculates the target deviation and gives control instructions. The APM hardware combines sensor information to analyze and track the control instructions, and issues motor control instructions. AirSim combines the control instructions with the dynamic wind field to complete attitude adjustment. At the same time, the control instructions are issued to the onboard computer through ROS to complete virtual-real collaborative target tracking. A detailed description of this method is as follows: AirSim virtual camera configuration, configure camera parameters (image type, resolution, FOV, position, etc.) in settings.json: Image type: "ImageType": 0 0=Scene view (RGB image), 1=Depth (depth map), 2=Segmentation (semantic segmentation map); Resolution: "Width": 640, "Height": 480; FOV (Field of View): "FOV_Degrees": 90; Camera installation position "X": 0.5, "Y": 0.0, "Z": 0.1 (body coordinate system).

[0074] YOLO uses the AirSim Python API to obtain images, preferably single-frame RGB images, that is, ImageType: 0 mentioned above; request RGB images, bind the camera name, and after obtaining the response, use OpenCV to convert the binary data into a numpy array.

[0075] The YOLO model takes an RGB image as input and outputs detection results (including bounding boxes, categories, and confidence levels). The results are returned as a DataFrame containing the coordinates and category of each detection.

[0076] The YOLO model selects specific categories and calculates the pixel deviation between the target center and the image center. The pixel deviation is converted into an attitude command and sent to the flight controller via SET_ATTITUDE_TARGET.

[0077] The AirSim API updates the wind field in real time within the simulation environment, and wind forces continue to act on the drone's dynamic model. The APM in HIL mode continuously receives sensor data (HIL_SENSOR messages). When it receives a SET_ATTITUDE_TARGET message from the YOLO model, it interprets the command based on the sensor information and generates motor outputs. These outputs are then returned via MAVLink to drive the drone in AirSim.

[0078] AirSim receives commands and issues control commands to the host computer through ROS to complete collaborative target recognition and tracking.

[0079] The digital twin-based UAV hardware-in-the-loop simulation test platform and test method designed by this invention first uses digital twin technology to model real-world scenarios and UAV physical parameters. Combined with dynamic weather simulation and random obstacle generation, this significantly improves the simulation environment's fidelity to complex low-altitude scenarios. Bidirectional data mapping and control are achieved between the physical and twin domains based on the MAVLink protocol. Real sensor data and real-time wind disturbance physics drive the virtual model. LiDAR data and kinematic models are used to predict collision risks in real time. Combined with hardware-in-the-loop (HIL) simulation technology, this allows the UAV to quickly generate obstacle avoidance commands when encountering sudden interference, eliminating the need for hovering and waiting. This enhances emergency response capabilities and flight safety in complex environments.

[0080] In existing technologies, software simulations either simplify or ignore microscopic features such as the Pixhawk flight controller's hardware watchdog mechanism and ADC sampling jitter. However, drone control failures in wind disturbances often stem from these details. For example, in the Boeing 737 MAX series of accidents, subsequent simulations failed to replicate the fatal interaction between sensor failures and the MCAS system, demonstrating the limitations of pure software simulation.

[0081] This method uses hardware-in-the-loop (HIL) to set up a real second flight controller (hardware) to control the twin drone. Simulation combined with virtual target recognition and tracking technology allows high-risk tasks (such as dense obstacle avoidance and extreme weather tracking) to be tested in a separate twin environment, without the risk of the physical drone colliding with the detection target. This provides an efficient and low-risk solution for the development, testing, and optimization of drone systems. The value of hardware-in-the-loop (HIL) lies primarily in the following three areas: first, the fidelity of physical effect simulation. Through real-time data interaction between real flight control hardware and the simulation environment (see the instruction manual for setting up the hardware-in-the-loop simulation environment for the second flight controller (APM)), sensor noise (such as IMU sensor drift and airspeed meter turbulence noise) and communication delay (MAVLink protocol transmission delay) can be accurately reconstructed. Pure software simulation can only rely on idealized mathematical models and cannot reproduce the random interference characteristics at the hardware level. Second, the reliability of development and verification. There is an essential difference between the binary code execution of real flight control and software simulation. Finally, fault reproduction capability. Fault injection and response testing based on real flight control hardware (such as disabling the TECS wind compensation module) provides fault reproduction capabilities that are crucial in the aviation field, effectively verifying system robustness under extreme conditions. Control instructions verified in the simulation environment can be directly applied to physical drones, eliminating the essential differences between software simulation and hardware execution, and ensuring consistency between virtual and real control.

[0082] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A UAV hardware-in-the-loop simulation test platform based on digital twin, characterized by: include: Physical drones, which are real drones flying in real target environments; The physical drone is equipped with a first flight controller, a camera, a laser radar, an airspeed meter, a GPS positioning module, an IMU module, and an onboard computer; the GPS positioning module, the IMU module, and the airspeed meter are respectively connected to the first flight controller, and the first flight controller, the camera, and the laser radar are respectively connected to the onboard computer; A physical host, on which a twin domain is deployed, wherein the twin domain includes a twin environment and a twin drone; The twin environment is a simulation environment identical to the real target environment, and the twin drone is a digital simulation model of the physical drone, flying in the twin environment. a second flight controller connected to the physical host, the second flight controller being used to control the twin drones; The onboard computer and the physical host exchange information via a wireless network.

2. A UAV hardware-in-the-loop simulation test method based on digital twins, characterized in that: The UAV hardware-in-the-loop simulation test platform based on digital twin as claimed in claim 1 comprises: Build twin drones; The physical host sends a flight waypoint sequence to the onboard computer, and the onboard computer transmits the flight waypoint sequence to the first flight controller; The first flight controller enables the physical UAV to fly in a real target environment according to the flight waypoint sequence, while collecting real target environment information in real time through a camera and a lidar, and collecting position information and attitude information of the physical UAV in real time through a GPS positioning module and an IMU module; The physical host receives the real target environment information, the position information and the attitude information of the physical UAV, and constructs a twin environment in real time based on the real target environment information. At the same time, the position information and attitude information of the physical UAV are transmitted to the second flight controller. The second flight controller controls the flight of the twin UAV so that the twin UAV maps the flight status of the physical UAV. A moving virtual obstacle is constructed in the twin environment. The second flight controller determines whether there is a collision risk between the twin drone and the virtual obstacle based on the virtual obstacle information. If it is determined that there is a collision risk, the flight waypoint sequence is updated and the twin drone is controlled to fly according to the updated flight waypoint sequence. At the same time, the second flight controller transmits the updated flight waypoint sequence to the first flight controller via the physical host and the onboard computer, and controls the physical drone to map the flight status of the twin drone.

3. The UAV hardware-in-the-loop simulation test method based on digital twin according to claim 2 is characterized in that: The method for building a twin environment is: Perform color correction and lens distortion correction on the photos taken by the camera, and fuse the corrected photos with the point cloud data collected by the lidar to generate dense point cloud data with texture; Generate a high-fidelity mesh model using the dense point cloud data and export it to fbx format; After optimizing and repairing the texture of the FBX mesh model, adjust the model scale, import the regional DEM data to generate the terrain base, and align the building model to the terrain according to GPS coordinates; Set the sun angle according to the latitude and longitude and time of the real area, and integrate the Ultra Dynamic Sky plug-in to achieve rain and snow effects.

4. The UAV hardware-in-the-loop simulation test method based on digital twin according to claim 2 or 3 is characterized in that: The method for determining whether there is a collision risk between the twin drones and virtual obstacles is: Obtain the location information and moving speed of virtual obstacles; Calculating future moments The distance between the twin drone and the virtual obstacle ; ； If there is a future moment , making , it is judged that there will be a collision between the twin drones and the virtual obstacle; in, Requirements: ; Where, Indicates the current moment, is the predicted duration set based on the system response time, is the position vector of the twin UAV in the twin environment, is the position vector of the virtual obstacle in the twin environment; is the relative velocity vector between the twin UAVs and the virtual obstacle.

5. The UAV hardware-in-the-loop simulation test method based on digital twin according to claim 4 is characterized in that: The process of mapping the flight status of the twin drone by the physical drone also includes: The wind force and wind speed in the real target environment are collected through the airspeed meter and transmitted to the twin environment; the second flight controller obtains the wind force and wind speed information, calculates the wind compensation amount based on the wind force and wind speed information; and converts the wind compensation amount into control instructions to control the flight posture of the twin UAV, and at the same time maps the twin UAV flight posture to the physical UAV.

6. The UAV hardware-in-the-loop simulation test method based on digital twin according to claim 5 is characterized in that: The calculation formula of the wind compensation amount is: ； Where, is the air density, is the resistance coefficient, represents the projection area of ​​the twin UAVs on the plane perpendicular to the wind direction, is the wind speed vector in the twin UAV body coordinate system.

7. The UAV hardware-in-the-loop simulation test method based on digital twin according to claim 6 is characterized in that: If it is determined that there is a risk of collision between the twin drone and the virtual obstacle, it is determined whether the twin drone can make a local detour; Among them, when ,and When the twin UAVs are judged to be able to make a partial detour, the second flight controller updates the waypoints according to the partial detour method; Where, Indicates the collision radius of the virtual obstacle, represents the safety radius of the twin drones, Indicates the twin UAV heading deviation angle, Indicates the maximum allowed heading deviation angle of the twin drones; If it is determined that the twin UAVs cannot perform local detours, the second flight controller updates the waypoints according to the global path replanning method.

8. The UAV hardware-in-the-loop simulation test method based on digital twin according to claim 7 is characterized in that: Also includes: The virtual camera in the twin domain acquires target image data and transmits the target image data to the second flight controller. The second flight controller performs target recognition based on the target detection algorithm and controls the twin UAV to track the target. At the same time, the target tracking status is mapped to the physical UAV through the onboard computer and the first flight controller.

Images