A Target Detection and Obstacle Avoidance Method for Quadrotor UAVs Oriented to Digital Twins

Abstract

The present invention relates to a target detection and obstacle avoidance method for a quadcopter drone oriented to digital twins, which is applied during the flight of a quadcopter drone, so that the quadcopter drone can quickly make corresponding evasive actions when it detects the type of obstacle target in an unfamiliar environment. By carrying various sensors and data transmission equipment, the quadcopter drone can interact with its twin drone in a virtual scene in real time and efficiently, achieving a certain degree of spatiotemporal consistency, and accurately mapping the flight status of the real quadcopter drone; a digital twin platform containing an efficient target detection algorithm and an obstacle avoidance algorithm is introduced into a ground computer to achieve high-precision human-computer interaction, realize multi-sensor fusion for information processing, achieve remote high-precision processing of drone information, realize safe and rapid obstacle avoidance of the quadcopter drone, and make the drone more efficient.

Description

Technical Field

[0001] The present invention belongs to the field of digital twin technology and drones, and relates to a target detection and obstacle avoidance method for a four-rotor drone oriented to digital twins. Background Art

[0002] Quadrotors are compact, high-performance vertical take-off and landing (VTOL) drones. They feature a simple structure, flexible operation, and a high payload capacity, making them valuable for both military and civilian applications. Quadrotors can be flown manually using a remote controller or hovered and tracked using a ground station and advanced APIs like MAVSDK. Because they can carry a variety of equipment to complex locations and perform tasks difficult to accomplish manually, multirotor drones are widely used in military and civilian applications, such as high-altitude inspections, forest fire detection, and disaster relief supply delivery. During quadrotor flight, specific target reconnaissance missions require high-precision and high-speed target detection algorithms. Furthermore, obstacle avoidance in the flight path is a crucial issue.

[0003] With the advancement of computer simulation technology, methods for modeling environments and drones are becoming increasingly sophisticated. To achieve real-time monitoring and prediction of drone flight processes, many researchers have begun to adopt digital twinning. Digital twinning involves digitally creating a virtual representation of a physical entity. Using historical data, real-time data, and algorithmic models, it simulates, verifies, predicts, and controls the entire lifecycle of the physical entity. Using digital twinning technology to model the flight environment and build an environmental digital twin platform allows for precise analysis of the environment's impact on drone flight. Using digital twinning technology to model quadcopter drones and build quadcopter twins allows for precise tracking of the drone's flight status and safety.

[0004] Although many target detection algorithms and obstacle avoidance algorithms have achieved very good results in experimental verification, these are all drone aerial target detection and obstacle avoidance conducted in real environments, and there are problems such as poor obstacle avoidance effect and difficulty in setting up simulation scenes for low-altitude target detection algorithms. Summary of the Invention

[0005] Technical problems to be solved

[0006] In order to avoid the shortcomings of the existing technology, the present invention proposes a digital twin-oriented quadrotor drone target detection and obstacle avoidance method, which uses digital twin technology to complete the construction of a digital twin platform and control the quadrotor drone on the ground to perform specific target detection and obstacle avoidance.

[0007] Technical Solution

[0008] A digital twin-oriented quadrotor drone target detection and obstacle avoidance method is characterized by using various sensors onboard the quadrotor drone to obtain information about obstacles in front of the drone, processing the received sensor data to detect and avoid obstacles according to the type of obstacle. The steps are as follows:

[0009] Step 1. Build an environment model based on Unreal Engine UE: Build a twin model of the real stadium in UE based on the environment and size of the real stadium;

[0010] Step 2: Build a quadrotor drone twin model based on AirSim:

[0011] Define the inertial coordinate system, i.e. the static coordinate system - the earth coordinate system, and the non-inertial coordinate system, i.e. the dynamic coordinate system - the body coordinate system;

[0012] Assumptions for the quadrotor drone: a uniform and symmetrical rigid body, with no change in mass and moment of inertia, only gravity and propeller pull, and one propeller axis rotating counterclockwise and the other rotating clockwise;

[0013] Build a quadrotor drone twin model in AirSim:

[0014] Dynamic model: The input is the thrust and torque provided by the propeller, and the output is the speed and angular velocity of the quadrotor;

[0015] Kinematic model: The input is the output of the dynamic model, that is, the speed and angular velocity of the quadrotor, and the output is the position and attitude of the quadrotor;

[0016] Step 3: Use the YOLOv5s target detection algorithm to perform target detection on the quadrotor drone aerial images;

[0017] The YOLOv5s target detection algorithm is an improved YOLOv5 target detection algorithm, and the improvement strategy is:

[0018] 1. The anchor box K-Means algorithm is changed to K-Means++;

[0019] 2. Add a small target detection layer and perform detection after splicing the shallow feature map with the deep feature map;

[0020] 3. Test using the α-IoU loss function;

[0021] Step 4: The quadcopter uses the MavLink message format and RPC protocol to communicate with the twin drone built on AirSim. The PX4 flight controller on the quadcopter sends the drone's flight status and position information to the ground computer of the twin drone built on AirSim. The quadcopter's sensor data is sent to the kinematic model of the quadcopter twin on the digital twin platform, including the dynamic model, through digital and image transmission.

[0022] Step 5: The digital twin platform uses the YOLOv5s target detection algorithm from step 3 to detect targets in the quadrotor drone's aerial images. It also uses a three-dimensional obstacle avoidance algorithm based on depth images to calculate an obstacle avoidance path. The digital twin platform sends the obstacle avoidance path to the real quadrotor drone by sending MavLink commands to control the real quadrotor drone to avoid obstacles.

[0023] The kinematic model of the quadrotor drone is:

[0024]

[0025] in, is the translational acceleration on the three axes, Indicates the roll angle, pitch angle and yaw angle of the drone, U1 represents the total thrust, U2 represents the thrust when the drone achieves roll motion, U3 represents the thrust when the drone achieves pitch motion, and U4 represents the thrust when the drone achieves yaw motion. m represents the mass of the drone, g is the acceleration of gravity, l represents the length of the arm, J represents the moment of inertia of the motor, and I xx , I yy , I zz where p, q, and r represent the roll, pitch, and yaw angular rates of the quadrotor, respectively.

[0026] The digital twin simulation end adopts the UE engine and AirSim simulation platform.

[0027] Beneficial effects

[0028] The present invention proposes a digital twin-oriented quadcopter drone target detection and obstacle avoidance method, which is applied during the flight of the quadcopter drone, so that the quadcopter drone can quickly make corresponding evasive actions when it detects the type of obstacle target in an unfamiliar environment. By carrying various sensors and data transmission equipment, the quadcopter drone can interact with the twin drone in the virtual scene in real time and efficiently, achieving a certain degree of spatiotemporal consistency, and accurately mapping the flight status of the real quadcopter drone; a digital twin platform containing efficient target detection algorithms and obstacle avoidance algorithms is introduced into the ground computer to achieve high-precision human-computer interaction, realize multi-sensor fusion for information processing, achieve remote high-precision processing of drone information, realize safe and rapid obstacle avoidance of the quadcopter drone, and make the drone more efficient.

[0029] Compared with the prior art, the present invention has the following technical features:

[0030] 1. Based on digital twin technology, the control capability of drones is further improved. In some complex terrains and severe weather conditions, digital twin technology is used to analyze various flight data and sensor data of real drones to achieve feedback control of the virtual twin drone on the real drone.

[0031] 2. Based on digital twin technology, real-time interaction between the real quadcopter and its virtual twin is achieved, including positive control and negative feedback. Through an auxiliary decision-making platform based on the twin drone, quadcopter flight becomes safer and more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a schematic diagram of the framework structure of the model system of the present invention;

[0033] Figure 2 For the stadium model;

[0034] Figure 3 It is a twin drone model;

[0035] Figure 4 This is the structural diagram of the improved YOLOv5 network. DETAILED DESCRIPTION

[0036] The present invention will now be further described with reference to the embodiments and accompanying drawings:

[0037] The present invention combines the UE4 rendering engine, the AirSim simulation platform, the improved YOLOv5 target detection algorithm and the three-dimensional obstacle avoidance algorithm based on depth images, and finally uses the native MavLink command to realize the flight control of the quadrotor drone.

[0038] The digital twin-oriented quadrotor drone target detection and obstacle avoidance method is characterized by real-time interaction and control of the digital twin-oriented quadrotor drone. It can be used for the quadrotor drone to obtain information about obstacles in front of the drone based on various sensors carried by the quadrotor drone in complex environments and severe weather conditions. The simulation software carried by the ground computer displays the drone's flight status in real time and processes the received sensor data to achieve rapid detection and obstacle avoidance of specific obstacle types.

[0039] Step 1: Build an environment model based on Unreal Engine UE

[0040] Environmental model modeling mainly includes the modeling of the selected specific environment - the stadium. Unreal Engine, abbreviated as UE, is a very famous game engine. It is a complete set of game development tools. It has been widely used in various fields such as game development, architectural design and rendering, car display, broadcasting and live events, film and television rendering, training and simulation. Unreal Engine also has an editor function, with a photo-realistic visual rendering level, supports dynamic physical simulation effects, and can produce lifelike animations. Its rich and powerful data interface can bring unlimited creative freedom to professionals from all walks of life. The present invention measures the environment and size of a real stadium and builds a twin model in UE.

[0041] Step 2: Build a quadrotor drone twin model based on AirSim

[0042] Build a quadrotor drone twin model in AirSim. The quadrotor drone twin model mainly performs dynamic modeling of the real quadrotor drone and annotates the established drone model.

[0043] Before building the model, it is necessary to define the inertial coordinate system (static coordinate system) - the earth coordinate system, and the non-inertial coordinate system (dynamic coordinate system) - the body coordinate system. The purpose of building a quadrotor model is to analyze the changes in the position and attitude of the quadrotor when it is subjected to external forces and external torques. Among them, the input of the dynamic model is the thrust and torque provided by the propeller, and the output is the speed and angular velocity of the quadrotor; the input of the kinematic model is the output of the dynamic model, that is, the speed and angular velocity of the quadrotor, and the output is the position and attitude of the quadrotor. To facilitate model building, the following assumptions are made about the quadrotor drone:

[0044] (1) The quadrotor drone is a uniformly symmetrical rigid body.

[0045] (2) The mass and moment of inertia of the quadrotor drone do not change

[0046] (3) The geometric center of the quadrotor coincides with its center of gravity

[0047] (4) The quadcopter is only subject to gravity and propeller pull

[0048] (5) One propeller shaft rotates counterclockwise and the other shaft rotates clockwise

[0049] Constructing a drone twin model and an environmental model from multiple dimensions. The premise for building a multidimensional model is to understand the data and constraint relationships at various physical levels between the drone and the environment, so that the model can accurately depict the physical entities. Furthermore, these physical data and constraint relationships must be further integrated, and finally, the correctness and effectiveness of the model must be verified through experiments.

[0050] The kinematic model of the drone is expressed as follows:

[0051]

[0052] in, is the translational acceleration on the three axes, Indicates the roll angle, pitch angle and yaw angle of the drone, U1 represents the total thrust, U2 represents the thrust when the drone achieves roll motion, U3 represents the thrust when the drone achieves pitch motion, and U4 represents the thrust when the drone achieves yaw motion. m represents the mass of the drone, g is the acceleration of gravity, l represents the length of the arm, J represents the moment of inertia of the motor, and I xx , I yy , I zz where p, q, and r represent the roll, pitch, and yaw angular rates of the quadrotor, respectively.

[0053] Step 3: Improved YOLOv5 target detection algorithm

[0054] YOLOv5 is a new generation of target detection algorithm in the YOLO series. The YOLOv5 target detection network provides four algorithms: YOLOv5s, YOLOv5m, YOLOv51 and YOLOv5x. Among them, the YOLOv5s algorithm is the network with the smallest depth and the smallest feature map width in the YOLOv5 series, while the three algorithms YOLOv5m, YOLOv51 and YOLOv5x are all products that are continuously deepened and widened on the basis of YOLOv5s. Since the detection task has high requirements on the detection speed, the present invention uses YOLOv5s with the fastest detection speed to perform target detection in quadcopter aerial images. The target detection algorithm is mainly evaluated by three indicators: intersection of union (IoU), detection speed (frames per second, FPS), and mean average precision (mAP). The specific improvement strategy is:

[0055] (1) Changing the anchor box K-Means algorithm to K-Means++

[0056] Before YOLOv5s uses K-Means clustering, the first step is to initialize k cluster centers. This is also the reason why the K-Means clustering algorithm suffers from a significant flaw: its convergence is heavily dependent on the initialization of the cluster centers. K-Means++ can effectively alleviate this problem, thereby improving detection accuracy and effectiveness to a certain extent. The K-Means++ algorithm essentially modifies the first step of the K-Means algorithm. This optimization ensures that the randomly selected center points no longer simply approach the local optimal solution, but rather approach the global optimal solution as much as possible. Therefore, K-Means++ can cluster better prior boxes.

[0057] (2) Adding a small target detection layer

[0058] One reason YOLOv5s performs poorly in detecting small objects is that the sample size of small objects is small. YOLOv5's downsampling factor is large, making it difficult for deeper feature maps to learn the characteristics of small objects. Therefore, we propose adding a small object detection layer to concatenate shallower feature maps with deeper feature maps for detection. This layer allows the network to focus more on detecting small objects, improving detection performance.

[0059] (3) Loss function uses α-IoU

[0060] YOLOv5s uses the CIOU loss function. Its advantages are: CIOU adds loss to the detection box scale, including both length and width losses, based on DIOU. This ensures that the predicted box is more consistent with the ground-truth box. However, the aspect ratio description introduces some ambiguity in relative values and fails to consider the balance between easy and difficult samples. To address these issues, the α-IoU loss function was used for testing.

[0061] Step 4: Establish communication between the real quadcopter and the twin drones

[0062] The quadcopter drone communicates with its twin drone based on the MavLink message format and RPC protocol. The quadcopter's PX4 flight controller can send the drone's flight status and location information to the ground computer.

[0063] Step 5: Real-time interaction based on digital twin technology

[0064] The quadcopter communicates with its twin drone using the MavLink message format. The quadcopter's PX4 flight controller transmits the drone's flight status and position information to a ground computer, which then transmits images and other data to the digital twin platform via digital and image transmission. The digital twin platform then sends MavLink commands to control the real quadcopter and avoid obstacles.

[0065] Build a real and twin data communication interface, that is, transmit the flight status data, trajectory data and sensor data of the real drone to the digital twin platform through various communication methods, and the digital twin platform sends obstacle avoidance commands to the real drone.

[0066] The present invention designs a target detection and obstacle avoidance method for a quadcopter drone oriented to digital twins, which realizes the interaction between the rotor drone and the twin drone, achieving the goal of reflecting the real with the virtual and controlling the virtual with the real. It solves the problems of the low intelligence level of quadcopter drones, poor data fusion, and difficulty in obstacle avoidance in complex environments. Through the digital twin platform, the quadcopter drone can avoid obstacles safely and quickly, making the drone more efficient.

Claims

1. A digital twin-oriented quadrotor drone target detection and obstacle avoidance method, characterized by: The quadcopter uses its own sensors to obtain information about obstacles in front of the drone, processes the received sensor data, and detects and avoids obstacles. The steps are as follows: Step 1. Build an environment model based on Unreal Engine UE: Build a twin model of the real stadium in Unreal Engine UE based on the environment and size of the real stadium; Step 2: Build a quadrotor drone twin model based on AirSim: Define the inertial coordinate system, i.e. the static coordinate system - the earth coordinate system, and the non-inertial coordinate system, i.e. the dynamic coordinate system - the body coordinate system; Assumptions for the quadrotor drone: a uniform and symmetrical rigid body, with no change in mass and moment of inertia, only gravity and propeller pull, and one propeller axis rotating counterclockwise and the other rotating clockwise; Build a quadrotor drone twin model in AirSim: Dynamic model: The input is the thrust and torque provided by the propeller, and the output is the speed and angular velocity of the quadrotor; Kinematic model: The input is the output of the dynamic model, that is, the speed and angular velocity of the quadrotor, and the output is the position and attitude of the quadrotor; Step 3: Use the YOLOv5s target detection algorithm to perform target detection on the quadrotor drone aerial image; the YOLOv5s target detection algorithm is an improved YOLOv5 target detection algorithm, and the improvement strategy is: (1) The anchor box K-Means algorithm was changed to K-Means++; (2) Add a small target detection layer and perform detection after splicing the shallow feature map and the deep feature map; (3) Test using the α-IoU loss function; Step 4: The quadrotor drone uses the MavLink message format and RPC protocol to communicate with the twin drone built on AirSim. The PX4 flight controller on the quadrotor drone sends the drone's flight status and position information to the twin drone's ground computer built on AirSim. The quadrotor drone's sensor data is sent to the quadrotor drone's twin kinematic model on the digital twin platform via digital and image transmission. Step 5: The digital twin platform uses the YOLOv5s target detection algorithm used in step 3 to detect targets in the quadrotor drone aerial image. At the same time, it uses a three-dimensional obstacle avoidance algorithm based on depth images to calculate the obstacle avoidance path. The digital twin platform sends the obstacle avoidance path to the real quadrotor drone through MavLink commands, controlling the real quadrotor drone to avoid obstacles.

2. The digital twin-oriented quadrotor drone target detection and obstacle avoidance method according to claim 1 is characterized by: The digital twin simulation end uses Unreal Engine UE and AirSim simulation platform.

Images