A dynamic target recognition and tracking method based on monocular unmanned flight platform

Abstract

The application relates to a dynamic target recognition and tracking method based on a monocular unmanned flight platform, uses a monocular camera as the only visual sensor for target perception, and adopts real-time dynamic target tracking combining target recognition and motion control. The application uses a low-cost monocular camera as the only visual perception device, deploys a light-weight convolutional neural network model on a low-cost embedded edge computing device, combines an open-source autonomous high-performance unmanned flight motion platform, and realizes efficient and accurate target recognition and tracking of an unmanned aerial vehicle on an interesting target under the condition of a lower economic cost, which can be quickly deployed. Through experimental testing, the device operates normally, the function effect is good, and the application value of the application is high.

Description

Technical Field

[0001] This invention belongs to the field of target recognition and tracking, and relates to a dynamic target recognition and tracking method based on a monocular unmanned aerial platform. Background Technology

[0002] In recent years, with the rapid development of unmanned aerial vehicle (UAV) technology, small rotorcraft UAV platforms have established advantages in flexibility and speed of movement, and have demonstrated advantages in rapid deployment and ease of application. Based on these characteristics, small rotorcraft UAV platforms can leverage their mobility advantages in the field of target recognition and tracking, achieving good application results. For example, Chinese patent application CN112215074A uses the NVDIAXavierNx module and the YOLOv4 algorithm to identify similar targets in preprocessed images and videos. The identified similar targets are displayed on the ground station via image and data transmission. The ground station is used to select specific targets for further tracking and identification, and uploads the specific targets to the UAV. The system of this invention identifies specific targets. The process is relatively complex and costly. This invention combines a low-cost monocular vision sensor and an onboard computer, along with relatively mature image processing technology and the YOLOv5 neural network, to process data and control aircraft movement in real time during flight. The process is simple and clear, and it can achieve good dynamic visual target recognition and tracking results at a lower cost. Summary of the Invention

[0003] Technical problems to be solved

[0004] To avoid the shortcomings of existing technologies, this invention proposes a dynamic target recognition and tracking method based on a monocular unmanned aerial platform, providing new application technology support for related fields such as environmental exploration and search and rescue in hazardous environments.

[0005] Technical solution

[0006] A dynamic target recognition and tracking method based on a monocular unmanned aerial vehicle platform, characterized by the use of a basic monocular camera, and the tracking steps are as follows:

[0007] Step 1: During the target data acquisition phase, the unmanned aerial platform uses the ROS package to record data on the actual flight status;

[0008] Step 2: Play the recorded ROS package target vision data, crop the target image from a fixed frame, use LabelImg to annotate the cropped image, and create the training set and test set of the target vision data;

[0009] Step 3: Train the lightweight YoloV5 convolutional neural network model using the training set;

[0010] Step 4: Input the data collected by the unmanned aerial vehicle platform at the target data location into the trained YOLOv5 convolutional neural network model to perform target recognition and obtain the target's location information;

[0011] Step 5: Send the target's location information to the ROS framework message queue and continuously update the ROS message queue; In the UAV motion control ROS framework, create a node named yolov5_detect_publisher_node, and send the target's location information to the ROS message queue under the topic / setpoint / relative_pos through this node. recvData is the recognition result of the YOLO network model received by the program, and finally the im_p sent by the program is the target's location information;

[0012] Step 6: After receiving the target position information, the drone motion control terminal under the ROS framework calculates the relative position between the drone and the target;

[0013] Step 7: Based on the relative position of the UAV and the target, send motion control commands to the flight controller:

[0014] Using the relative position in step 6 as the deviation control quantity, the waypoint and speed control quantities are calculated using the PID control method as motion control commands. The motion control commands are sent from the embedded edge computing core NVIDIA Jetson Nano to the flight control board Cube, and implemented using the communication method encapsulated by Mavlink under ROS.

[0015] According to the Mavros communication protocol, waypoint setting commands are sent to the flight controller via the setpoint_raw message;

[0016] Step 8: The UAV achieves real-time dynamic target tracking based on motion commands: The UAV receives motion commands calculated from the continuously updated real-time target position, enabling the monocular unmanned flight platform to achieve real-time dynamic tracking of targets of interest.

[0017] The training set contains 856 photos, and the test set contains 166 photos.

[0018] The YOLOv5 convolutional neural network model includes four general modules: input, baseline network, Neck network, and Head output.

[0019] The method for calculating the relative position between the UAV and the target is as follows: the relative position error between the UAV and the target is used as a control variable for waypoint control, and the speed control variable is calculated based on this variable. PID control is then used to achieve stable and smooth movement of the UAV.

[0020] Beneficial effects

[0021] This invention proposes a dynamic target recognition and tracking method based on a monocular unmanned aerial vehicle (UAV) platform. It uses a monocular camera as the sole visual sensor for target perception and employs real-time dynamic target tracking that combines target recognition with motion control. Specifically, the UAV collects visual data of the target of interest, extracts and annotates the visual data in fixed frames, and creates training and testing sets. A suitable convolutional neural network model is selected for training and testing of target recognition performance. The trained network model is used for target recognition, and the target's position information is sent to a message queue within the ROS framework. The UAV control terminal within the ROS framework receives the target position information, calculates the position relative to the target, and sends motion control commands to the flight controller based on the relative position of the UAV and the target. The UAV then performs real-time dynamic target tracking according to these motion commands.

[0022] The beneficial effects of this invention are as follows: it addresses the current lack of high-performance, real-time dynamic tracking unmanned aerial vehicles (UAVs) capable of rapid deployment for tracking any target in the field of target recognition and tracking. In terms of hardware, this invention employs a low-cost monocular camera as the sole visual sensing device, combined with a powerful, compact quadcopter UAV platform and a small, low-cost embedded edge computing core, the NVIDIA Jetson Nano, which boasts considerable computing power. In terms of methodology, it utilizes the lightweight convolutional neural network model Yolov5 and the mature open-source flight control system PX4. Overall, it achieves high-accuracy target recognition and rapid-response real-time dynamic target tracking under low-cost conditions. This has practical application value in areas such as reconnaissance in hazardous environments and target search and rescue, and also promotes the development of related industries. Attached Figure Description

[0023] Figure 1 Target Dataset Creation

[0024] Figure 2 Network training parameters and results

[0025] Figure 3 Flower target recognition results

[0026] Figure 4 Send ROS to the target location

[0027] Figure 5 Relative position calculation

[0028] Figure 6 Waypoint instruction data format

[0029] Figure 7 ROS waypoint command issuance

[0030] Figure 8 ROS speed command release

[0031] Figure 9 Planning flight patrol missions

[0032] Figure 10 Enable recognition and tracking function

[0033] Figure 11 : Identify tracking program results

[0034] Figure 12 Tracking the scene Detailed Implementation

[0035] The present invention will now be further described in conjunction with the embodiments and accompanying drawings:

[0036] The technical solution of this invention is as follows:

[0037] After comparing different feasible technical solutions, the technical approach of this invention is determined to be a lightweight convolutional neural network combined with a quadcopter unmanned flight platform equipped with an embedded edge computing core and an open-source flight control core, while the visual perception module only uses a basic monocular camera. The specific solution is as follows:

[0038] a) Use drones to collect visual data of targets of interest.

[0039] To achieve better final target recognition results, data should be collected during the target data acquisition phase based on the actual flight state. The specific acquisition method is the ROS package recording method commonly used in robot systems.

[0040] b) Extract visual data from fixed frames and label them to create training and testing sets.

[0041] Play back the recorded ROS package target visual data, and use a Python program to extract images by fixing frames. Then, use LabelImg to label the dataset to create training and testing sets for the target visual data. The creation process is as follows: Figure 1 As shown, the final training set contained 856 photos, and the test set contained 166 photos.

[0042] c) Select a suitable convolutional neural network model for training and test the target recognition effect.

[0043] Considering the limited computing power of the NVIDIA Jetson Nano embedded edge computing core, we used a lightweight YOLOv5 convolutional neural network model for training. YOLOv5 is a single-stage object detection algorithm that balances accuracy and speed. The entire algorithm in YOLOv5 can be divided into four general modules: input, baseline network, Neck network, and Head output. The relevant theory is open source, and its details will not be elaborated further.

[0044] After training for 50 epochs, the weight file is obtained, and the training results are as follows. Figure 2 As shown in the test, the trained model can achieve high accuracy in identifying the specified target.

[0045] d) Use the trained network model for target recognition.

[0046] To identify a specific category of target, the operation involves adding the parameter `--classes[number]` to the onboard computer system terminal. The mechanism is that a fixed category parameter is input, and then the image is processed by a convolutional neural network to extract features. The head output shows all targets in the image and their confidence scores. If the command includes the `classes` parameter, only the specified category is selected, labeled, and returned. Taking flower image recognition as an example, the recognition result is as follows... Figure 3 As shown.

[0047] e) Send the target's location information to the ROS framework message queue.

[0048] The drone's motion control is performed within the ROS framework. To control the drone's motion and achieve real-time dynamic tracking of the target of interest, the target's position information needs to be continuously updated in the ROS message queue. The program source code is as follows: Figure 4 As shown, the program creates a node named yolov5_detect_publisher_node in the ROS framework for UAV motion control. This node sends the target location information to the ROS message queue under the topic / setpoint / relative_pos. The target location information originates from the recognition results of the YOLO network model and is sent to the program via local socket communication. Here, recvData represents the recognition results received by the program from the YOLO network model, and the final im_p sent by the program represents the target's location information.

[0049] f) Under the ROS framework, the UAV control terminal receives the target position information and calculates the position relative to the target.

[0050] After receiving the target position information, the drone motion control terminal under the ROS framework needs to calculate the relative position between the drone and the target. The source code is as follows: Figure 5 As shown, the program works by using the relative position error between the UAV and the target as a control variable for waypoint control. Based on this error, it calculates the speed control variable and employs PID control to achieve stable and smooth movement of the UAV. The program principle is a well-known technique in the field of automatic control and will not be elaborated further.

[0051] g) Send motion control commands to the flight controller based on the relative position of the UAV and the target.

[0052] Motion control commands are sent from the embedded edge computing core NVIDIA Jetson Nano to the flight control board Cube. Communication between the two utilizes the Mavlink communication protocol, popular in micro-rotor drones. Mavlink is an open-source communication protocol released by ETH Zurich, primarily used in micro-aircraft. This protocol encapsulates message frames with custom data structures; both ends of the communication process package and parse these messages according to the protocol to transmit the specified message content. Since the onboard program runs under the ROS framework, the actual communication method uses Mavros, a ROS wrapper for Mavlink.

[0053] According to the Mavros communication protocol, waypoint setting commands can be sent to the flight controller via the `setpoint_raw` message. Specifically, in the source code, the waypoint commands are sent to the flight controller by sending the position information calculated by the program to the ` / mavros / setpoint_raw / local` topic. This topic receives messages of type `PositionTarget` with data formatted as `positionTarget`. This message defines the waypoint and speed commands, as officially defined as follows: Figure 6 As shown.

[0054] As can be seen, the message defines the waypoint and velocity commands using the data types geometry_msgs / Point and geometry_msgs / Vector3, respectively. In the source code, the issued commands are as follows: Figure 7 Here, `self.setpoint_.position` represents the waypoint information, and `self.setpoint_.velocity` represents the velocity information. Finally, `self.setpoint_pub_.publish(self.setpoint_)` sends the waypoint command to the defined ` / mavros / setpoint_raw / local` topic, which in turn passes it to the flight controller.

[0055] As described in waypoint instructions, speed commands are also transmitted to the flight controller by sending the speed information calculated by the program to the ` / mavros / setpoint_raw / local` topic. The source code is as follows... Figure 8 The velocity command is self.setpoint_.velocity. self.setpoint_pub_.publish(self.setpoint_) sends the velocity command to the topic / mavros / setpoint_raw / local, thus enabling the velocity command to be issued to the flight controller.

[0056] h) The UAV achieves real-time dynamic target tracking based on motion commands.

[0057] In summary, the motion commands received by the UAV are calculated based on the constantly updated real-time target position. Therefore, based on the dynamic and accurate target identification by the convolutional neural network model, and after a series of processing steps as described above, the real-time dynamic tracking function of the monocular UAV platform for the target of interest can be realized.

[0058] The present invention is further illustrated below with specific implementation examples:

[0059] Step 1: Establish a flight patrol route in the target area, such as... Figure 9 As shown, flight patrol missions are planned and issued for aircraft at the QGC ground station, and the mission setting process is a known technology.

[0060] Step 2: In the embedded edge computing core, start the target recognition and tracking program, such as... Figure 10 As shown, this is to enable the unmanned aerial vehicle (UAV) to identify and track targets of interest. `detect.py` is the Yolov5 recognition program, `--weightsbest.py` is the pre-trained weight file, `--source 0` specifies that the program reads data from a monocular camera, and `--classes 16` specifies the targets the program needs to identify. `tracker.launch` is the UAV's motion control program startup file.

[0061] Step 3: The unmanned aerial vehicle takes off, detects, identifies, and tracks targets during the patrol. The target image recognition results are as follows: Figure 3 As shown, the predetermined flower target was identified, and the target identification result is as follows. Figure 11 As shown, the target of Flowe was identified in the terminal in the upper left corner, while the target location information sent by the recognition program was received in the terminal in the lower left corner. The tracking scene is as follows. Figure 12 As shown.

[0062] Step 4: After completing the target identification and tracking task, the unmanned aerial vehicle safely returns to the takeoff point, thus completing the mission.

[0063] Matters not covered in this invention are common knowledge.

[0064] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A dynamic target recognition and tracking method based on a monocular unmanned aerial platform, characterized in that... Using a basic monocular camera, the tracking steps are as follows: Step 1: During the target data acquisition phase, the unmanned aerial platform uses the ROS package to record data on the actual flight status; Step 2: Play the recorded ROS package target vision data, crop the target image from a fixed frame, use LabelImg to annotate the cropped image, and create the training set and test set of the target vision data; Step 3: Train the lightweight YoloV5 convolutional neural network model using the training set; Step 4: Input the data collected by the unmanned aerial vehicle platform at the target data location into the trained YOLOv5 convolutional neural network model to perform target recognition and obtain the target's location information; Step 5: Send the target's location information to the ROS framework message queue and continuously update the ROS message queue; In the UAV motion control ROS framework, create a node named yolov5_detect_publisher_node, and send the target's location information to the ROS message queue under the topic / setpoint / relative_pos through this node. recvData is the recognition result of the YOLO network model received by the program, and finally the im_p sent by the program is the target's location information; Step 6: After receiving the target position information, the drone motion control terminal under the ROS framework calculates the relative position between the drone and the target; Step 7: Based on the relative position of the UAV and the target, send motion control commands to the flight controller: Using the relative position in step 6 as the deviation control quantity, the waypoint and speed control quantities are calculated using the PID control method as motion control commands. The motion control commands are sent from the embedded edge computing core NVIDIA Jetson Nano to the flight control board Cube, and implemented using the communication method encapsulated by Mavlink under ROS. According to the Mavros communication protocol, waypoint setting commands are sent to the flight controller via the setpoint_raw message; Step 8: The UAV achieves real-time dynamic target tracking based on motion commands: The UAV receives motion commands calculated from the continuously updated real-time target position, enabling the monocular unmanned flight platform to achieve real-time dynamic tracking of targets of interest.

2. The dynamic target recognition and tracking method based on a monocular unmanned aerial platform according to claim 1, characterized in that: The training set contains 856 photos, and the test set contains 166 photos.

3. The dynamic target recognition and tracking method based on a monocular unmanned aerial platform according to claim 1, characterized in that: The YOLOv5 convolutional neural network model includes four general modules: input, baseline network, Neck network, and Head output.

4. The dynamic target recognition and tracking method based on a monocular unmanned aerial platform according to claim 1, characterized in that: The method for calculating the relative position between the UAV and the target is as follows: the relative position error between the UAV and the target is used as a control variable for waypoint control, and the speed control variable is calculated based on this variable. PID control is then used to achieve stable and smooth movement of the UAV.

Images