Target identification and tracking method of unmanned aerial vehicle
By integrating cameras, ultrasonic sensors, optical flow modules and GPS modules on the drone, combined with KCF algorithms and target template prediction technology, the problem that traditional methods are difficult to achieve drone target recognition and tracking in complex environments is solved, and efficient , robust and accurate target tracking.
Patent Information
- Application Number
- CN202411360449.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2025-01-03
AI Technical Summary
Traditional target recognition and tracking methods are difficult to apply to drone video tracking. Due to the large displacement of the target, vibration noise, dynamic background and blur, it is difficult to achieve reliable target recognition and tracking in complex environments.
The drone is equipped with a camera, ultrasonic sensor, optical flow module, GPS positioning module and wireless communication module, combined with KCF algorithm and target template prediction technology, and through real-time attitude adjustment and image processing, fast and accurate tracking of mobile targets is achieved.
It achieves efficient, robust, accurate and real-time tracking of drone targets in complex environments, and is suitable for fixed and mobile targets, reducing human intervention, improving the efficiency and safety of tracking, and being highly adaptable.
Smart Images
Figure CN119245608A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to a target recognition and tracking method for unmanned aerial vehicles. Background Art
[0002] As the technology of UAVs gradually matures, the application of UAVs in the civil field is becoming more and more common. Due to the low cost, high efficiency and strong flexibility of UAVs, the application fields of UAVs such as geological survey, disaster monitoring, meteorological observation, air traffic control, border patrol monitoring, communication relay, pesticide monitoring, etc. have been developed and deployed. In particular, people are paying more and more attention to UAV aerial image photography, trying to detect and track specific types of objects such as vehicles, people, including moving pedestrians in real time, autonomous navigation and autonomous landing. The most common systems include cameras (visible light cameras and infrared cameras) equipped with a processor in the aircraft and an onboard GPS.
[0003] Visual object detection is an important component of drone applications and is key to developing fully automated systems. However, the task of object detection is very challenging, and the imaging conditions on low-cost consumer drones are even more difficult: the images are jittery and blurry due to the motion of the drone. Lighting and atmospheric conditions may affect the visibility and contrast of pedestrians on the ground. In addition, physical objects (such as vegetation and buildings) may occlude humans, making reliable recognition in partial or full occlusion very challenging. Onboard cameras usually have relatively low resolution and the captured objects are usually small. Therefore, image processing methods are particularly important, especially for the identification and tracking of objects. However, traditional target recognition and tracking methods are for detection with fixed cameras, which are common in traffic monitoring and large parking lots. The algorithm has the characteristics of fixed camera, high image quality, clear target, low noise pollution, simple background, etc., and is easy to implement. Traditional background subtraction and optical flow methods are limited by fixed background, and UAV video becomes complicated due to the large displacement of the target, vibration noise, dynamic background, clutter and blur. Therefore, the algorithm for identifying dynamic targets using fixed cameras cannot be applied to UAV video tracking. To this end, we propose a target recognition and tracking method for UAV to solve the above problems. Summary of the invention
[0004] The purpose of the present invention is to provide a target recognition and tracking method for an unmanned aerial vehicle to solve the problems raised in the above background technology.
[0005] The technical solution of the present invention is: a target identification and tracking method for a drone, comprising a drone, a camera is arranged on the body of the drone, an ultrasonic sensor is arranged inside the body of the drone, the ultrasonic sensor signal is connected to an optical flow module, the inside of the drone is fixedly connected to a GPS positioning module, the internal signal of the drone is connected to a wireless communication module, the inside of the drone is arranged with a data processing unit, and the inside of the drone is arranged with a control mainboard.
[0006] Preferably, the control mainboard is provided with a dual-core CPU, a quad-core GPU and an 8G flash memory stick.
[0007] Preferably, the drone target identification and tracking method comprises the following steps: Step S1: After the drone takes off near the target, it turns on the altitude hold mode and rotates along the yaw axis to shoot the surrounding environment; Step S2: Use the KCF algorithm in the images of the drone's surrounding environment to find the target and approach the target in a straight line, select the target and start the tracking mode; Step S3: During the tracking process, when the tracked object is severely occluded, the target template is used to predict the target position in the next frame. When the target is removed from the current frame, steps S1 and S2 are repeated. Step S4: Calculate the center position of the target and the tilt angle of the drone in each frame, generate corresponding attitude adjustment commands, send them to the drone via Wi-Fi, and make appropriate attitude adjustments.
[0008] Preferably, after the target position is correctly located in the image, the distance between the UAV and the target is calculated based on the coordinate information of the target in the image combined with the camera focal length, camera viewing angle and aircraft altitude. The method for the UAV to track the target is to install the camera vertically downward at the center of the UAV to capture the image in real time. The horizontal displacement between the UAV and the target is calculated from the pixel position of the target. The position of the target pixel in the image is defined as ( , ), and the conversion formula between the target and the horizontal displacement (x, y) of the drone is: Assume [△ ,△ ,△ ] indicates the relative position of the target, ( , , ) represents the body coordinate axis, △φ represents the angle between the yaw axis and the center of the target frame, and the following reference is used to estimate the distance between the drone and the target, that is, the expected size of the target is 40cm×30cm, and the distance represented is 3m, ( , ) represents the upper left corner position of the target box returned by the tracking algorithm, ( , ) represents the size of the target marking box, and the target center position is normalized as: The calculation method for the correction of the drone's attitude and lateral motion freedom is as follows: = = - = - When | |≥25°, the tracking mode is turned on, that is, it follows the target and maintains a certain distance.
[0009] Preferably, z represents the altitude of the drone, f represents the focal length of the camera, β represents the pitch angle of the camera, α represents the angle between the line connecting the two points of the object and the optical axis of the camera, (uo,vo) represents the coordinates of the center point of the image, and , represents the center position of the image target, Indicates the size of the target marker box, where ψ and 0 are the angles between the pitch axis and the yaw axis and the velocity direction respectively. and , represents the camera's viewing angle in the horizontal and vertical directions, Indicates the reference angle of the yaw axis, which controls the rotation of the drone around the yaw axis.
[0010] The present invention provides a grass planting mosaic drone through improvement, which has the following improvements and advantages compared with the prior art: First, the present invention enables the drone to quickly and comprehensively scan the surrounding environment by turning on the fixed-altitude mode and rotating along the yaw axis to shoot, thereby improving the efficiency of target recognition. At the same time, the KCF (Kernelized Correlation Filters) algorithm can quickly locate the target in the picture, so that the drone can quickly approach the target and start the tracking mode. When the tracked object is severely occluded, the target template is used to predict the target position in the next frame. This method can overcome the challenges brought by occlusion to a certain extent and maintain the continuity and stability of tracking. When the target moves out of the current picture, the system can re-execute the take-off, scanning and recognition steps to ensure that the tracking process will not be interrupted. By calculating the center position of the target in each frame and the tilt angle of the drone, the system can accurately generate attitude adjustment commands and send them to the drone via Wi-Fi. Real-time and accurate attitude adjustment can ensure that the drone always maintains stable tracking of the target. Since the entire recognition, tracking and attitude adjustment process is carried out in real time, the drone can quickly respond to dynamic changes of the target and maintain continuous tracking of the target. This method is not only applicable to fixed targets, but also to moving targets. By adjusting the drone's attitude in real time, the drone can always aim at the target, whether the target is stationary or moving. The drone target recognition and tracking method has significant advantages such as high efficiency, robustness, accuracy, real-time and adaptability, and can provide strong support for drone target tracking in complex environments.
[0011] Second: The present invention, by using the KCF algorithm and other advanced image processing techniques, the drone can quickly and accurately identify and track the target, which is crucial in various applications, such as search and rescue, wildlife research, agricultural monitoring, etc. By combining the position information of the target in the image and the camera parameters, the drone can calculate the distance to the target, providing an accurate understanding of the relative position between the drone and the target, so that the drone can more accurately adjust its posture and position to track the target. During the tracking process, when the target is severely occluded, the drone can use the target template to predict the position of the target in the next frame. This prediction mechanism ensures that the drone can still maintain tracking of the target when occlusion occurs. When the target disappears completely from sight, the drone can re-search and locate the target. The drone can generate corresponding attitude adjustment commands based on the calculated target position and its own attitude, ensuring that the drone can approach or move away from the target in the correct direction and angle to achieve accurate tracking. Since Wi-Fi is used for communication, the drone can receive instructions from remote operators or autonomous systems and make adjustments as needed, allowing the drone to adapt to various complex environments and tracking needs. By integrating various algorithms and technologies, the drone can achieve a highly automated tracking process, reducing the need for human intervention and improving the efficiency and accuracy of tracking. When the angle difference between the drone and the target exceeds a certain threshold (such as 25°), the drone will turn on the tracking mode. This mechanism ensures that the drone always maintains a line of sight to the target during the tracking process, thereby improving the safety of tracking.
[0012] Third: The present invention, through the joint work of ultrasonic sensors and optical flow modules, can provide stable hovering and precise control of the UAV when it is moving at close range and slow speed. The ultrasonic sensor can measure the distance between the UAV and the ground or other objects, while the optical flow module calculates the relative speed and direction of the UAV by analyzing the images captured by the camera. The combination of these two modules enables the UAV to maintain stable flight indoors or in low-light environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] The present invention will be further explained below in conjunction with the accompanying drawings and embodiments: Figure 1 It is a schematic diagram of the structure of the UAV of the present invention; Figure 2 It is a flow chart of the KCF algorithm of the present invention; Figure 3 It is a distance schematic diagram of the present invention; Figure 4 It is a schematic diagram of the target location in the image. DETAILED DESCRIPTION
[0014] The present invention is described in detail below, and the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0015] The present invention provides a target identification and tracking method for a drone by improvement. The technical solution of the present invention is: like Figure 1 - Figure 4 As shown, a target recognition and tracking method for a drone includes a drone, a camera is arranged on the drone body, an ultrasonic sensor is arranged inside the drone body, the ultrasonic sensor signal is connected to an optical flow module, a GPS positioning module is fixedly connected inside the drone, the internal signal of the drone is connected to a wireless communication module, a data processing unit is arranged inside the drone, and a control mainboard is arranged inside the drone.
[0016] Furthermore, the control motherboard is equipped with a dual-core CPU, a quad-core GPU and an 8G flash memory stick. The dual-core CPU can process multiple tasks in parallel, ensuring that various sensor data, navigation instructions and control algorithms of the drone can be processed efficiently and in real time during flight, which is crucial to the stability and response speed of the drone. The quad-core GPU (graphics processing unit) is extremely efficient in processing image and video data. For the optical flow module on the drone, the GPU can quickly analyze the images captured by the camera, calculate the relative speed and direction of the drone, and thus provide accurate control signals. The 8G flash memory stick provides the drone with ample storage space for caching and processing various data, which is not only It can not only improve the data processing speed, but also ensure that the UAV can operate stably for a long time in complex environments. Combined with ultrasonic sensors and optical flow modules, stable hovering and precise control of the UAV at close range and slow movement can be achieved. It can accurately measure the distance between the UAV and the ground or other objects. By continuously detecting distance changes, the UAV can adjust its flight altitude to ensure collision avoidance in complex environments. By analyzing the images captured by the camera, the relative speed and direction of the UAV are calculated, which is crucial for the UAV to maintain stable flight in low-light or indoor environments. The data provided by the optical flow module is combined with the distance data of the ultrasonic sensor to achieve precise control of the UAV in various environments.
[0017] Furthermore, the drone target recognition and tracking method includes the following steps: Step S1: After the drone takes off near the target, it turns on the altitude-fixing mode and rotates along the yaw axis to shoot the surrounding environment. The altitude-fixing mode ensures that the drone maintains a constant altitude during flight, avoiding image jitter or loss of focus caused by altitude changes. This stability is essential for shooting high-quality, clear images. In the fixed-altitude mode, the pilot does not need to pay attention to the altitude adjustment of the drone all the time, and can focus more on shooting operations and heading control, which greatly simplifies the control process and improves shooting efficiency. By rotating the drone along the yaw axis, 360-degree panoramic shooting can be achieved. This shooting method can fully capture the environmental information around the target and provide rich materials for later analysis and processing. The height and position of the drone can be adjusted at any time, and the rotation shooting along the yaw axis provides greater flexibility. The photographer can quickly change the shooting angle and direction as needed to capture the most ideal picture. In the fixed-altitude mode, the drone can be maintained at a safe altitude range for shooting, avoiding the risk of collision with ground obstacles or other aircraft. At the same time, the stable flight altitude also helps to improve the safety during shooting. Combined with the efficient, accurate and flexible characteristics of drone aerial photography technology and the stability of the fixed-altitude mode, clearer, more stable and comprehensive aerial images can be shot. Step S2: Use the KCF algorithm in the images of the drone's surroundings to find the target and approach the target in a straight line, select the target and start the tracking mode; the KCF algorithm is an efficient target tracking algorithm. Its core idea is to use the diagonal property of the circulant matrix in Fourier space to convert the matrix operation into the dot product of the elements, thereby greatly reducing the complexity of the calculation. This algorithm enables the drone to achieve sub-millisecond running time on each frame of the image, ensuring real-time tracking of the target. The KCF algorithm is based on the idea of correlation filters. By representing the appearance of the target as the weight of the filter, it achieves accurate positioning of the target. This algorithm can accurately find and select the target in a complex drone flight environment. , which provides a solid foundation for subsequent tracking. During the training process, the KCF algorithm effectively improves the robustness of the algorithm by selecting the target area as a positive sample and the surrounding area of the target as a negative sample. At the same time, the algorithm also adapts to the changes in the target appearance by continuously updating the weight of the filter, thereby ensuring stability during the tracking process. By adopting the KCF algorithm, the drone can identify and track the target in real time, and then adjust its own flight trajectory according to the tracking results to achieve flight control in a straight line approaching the target. This real-time flight control can greatly improve the work efficiency of the drone and reduce the difficulty of operation. The KCF algorithm is not only applicable to the drone field, but also can be applied to autonomous driving, indoor robot navigation, security monitoring and other fields; Step S3: During the tracking process, when the tracked object is severely occluded, the target template is used to predict the target position in the next frame. When the target is removed from the current frame, steps S1 and S2 are repeated; see Figure 2 ; Step S4: Calculate the center position of the target and the tilt angle of the drone in each frame, generate corresponding attitude adjustment commands, send them to the drone via Wi-Fi and make appropriate attitude adjustments. Calculate the center position of the target and the tilt angle of the drone in each frame, generate corresponding attitude adjustment commands, send them to the drone via Wi-Fi for attitude adjustment. This can improve target tracking accuracy, enhance flight stability, improve data transmission efficiency, realize remote control and monitoring, and improve the intelligence level of drones. These benefits make the application of drones in aerial photography, reconnaissance, search and rescue and other fields more extensive and efficient.
[0018] Furthermore, after the target position is correctly located in the image, the distance between the drone and the target is calculated based on the coordinate information of the target in the image combined with the camera focal length, camera viewing angle, and aircraft altitude. The drone tracks the target by installing the camera vertically downward in the center of the drone to capture the image in real time. The horizontal displacement between the drone and the target is calculated from the pixel position of the target. The position of the target pixel in the image is defined as ( , ), and the conversion formula between the target and the horizontal displacement (x, y) of the drone is: Assume [△ ,△ ,△ ] indicates the relative position of the target, ( , , ) represents the body coordinate axis, △φ represents the angle between the yaw axis and the center of the target frame, and the following reference is used to estimate the distance between the drone and the target, that is, the expected size of the target is 40cm×30cm, and the distance represented is 3m, ( , ) represents the upper left corner position of the target box returned by the tracking algorithm, ( , ) represents the size of the target marking box, and the target center position is normalized as: The calculation method for the correction of the drone's attitude and lateral motion freedom is as follows: = = - = - When | |≥25°, the tracking mode is turned on, that is, the target is followed and a certain distance is maintained. By accurately calculating the distance between the drone and the target, the drone can accurately track and follow the target, especially in scenarios that require precise navigation or obstacle avoidance. The drone can adaptively adjust its flight attitude and speed according to the distance and relative position relationship between the target and the drone to maintain a constant distance or specific angle with the target. By capturing images in real time and calculating the target position, the drone can respond to the position change of the target in real time to ensure the stability and continuity of tracking. This vision-based tracking method reduces the reliance on human operation, allowing the drone to perform tasks more intelligently, improving operational efficiency and safety. The drone can adapt to targets of different sizes, shapes and speeds, and only needs to adjust the relevant parameters according to the actual size and expected distance of the target. This method can be combined with other sensors and algorithms to further improve the accuracy and stability of tracking, such as combining GPS, IMU and other sensors for attitude estimation and fusion. In scenarios where collision with the target needs to be avoided, accurate calculation of distance and position relationship can help the drone better plan the path and speed, thereby avoiding potential safety risks.
[0019] Furthermore, where z represents the height of the drone, f represents the focal length of the camera, β is the pitch angle of the camera, α is the angle between the line connecting the two points of the object and the optical axis of the camera, (uo,vo) represents the coordinates of the center point of the image, and , represents the center position of the image target, Indicates the size of the target marker box, where ψ and 0 are the angles between the pitch axis and the yaw axis and the velocity direction respectively. and , represents the camera's viewing angle in the horizontal and vertical directions, Indicates the reference angle of the yaw axis, which controls the rotation of the drone around the yaw axis.
[0020] The above description enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for target recognition and tracking of an unmanned aerial vehicle, comprising an unmanned aerial vehicle, characterized in that: A camera is arranged on the drone body, an ultrasonic sensor is arranged inside the drone body, the ultrasonic sensor signal is connected to an optical flow module, a GPS positioning module is fixedly connected inside the drone, an internal signal of the drone is connected to a wireless communication module, a data processing unit is arranged inside the drone, and a control mainboard is arranged inside the drone.
2. The target recognition and tracking method of a drone according to claim 1, characterized in that: The control mainboard is provided with a dual-core CPU, a quad-core GPU and an 8G flash memory stick.
3. The target recognition and tracking method of a drone according to claim 1, characterized in that: The UAV target identification and tracking method comprises the following steps: Step S1: After the drone takes off near the target, it turns on the altitude hold mode and rotates along the yaw axis to shoot the surrounding environment; Step S2: Use the KCF algorithm in the images of the drone's surrounding environment to find the target and approach the target in a straight line, select the target and start the tracking mode; Step S3: During the tracking process, when the tracked object is severely occluded, the target template is used to predict the target position in the next frame. When the target is removed from the current frame, steps S1 and S2 are repeated. Step S4: Calculate the center position of the target and the tilt angle of the drone in each frame, generate corresponding attitude adjustment commands, send them to the drone via Wi-Fi, and make appropriate attitude adjustments.
4. The target recognition and tracking method of a drone according to claim 1, characterized in that: After the target position is correctly located in the image, the distance between the drone and the target is calculated based on the target coordinate information in the image combined with the camera focal length, camera viewing angle, and aircraft altitude. The drone tracks the target by installing the camera vertically downward in the center of the drone to capture the image in real time. The horizontal displacement between the drone and the target is calculated from the pixel position of the target. The position of the target pixel in the image is defined as ( , ), and the conversion formula between the target and the horizontal displacement (x, y) of the drone is: ； Assume [△ ,△ ,△ ] indicates the relative position of the target, ( , , ) represents the body coordinate axis, △φ represents the angle between the yaw axis and the center of the target frame, and the following reference is used to estimate the distance between the drone and the target, that is, the expected size of the target is 40cm×30cm, and the distance represented is 3m, ( , ) represents the upper left corner position of the target box returned by the tracking algorithm, ( , ) represents the size of the target marking box, and the target center position is normalized as: ， ， ， The calculation method for the correction of the drone's attitude and lateral motion freedom is as follows: = ， = - ， = - ； When | |≥25°, the tracking mode is turned on, that is, it follows the target and maintains a certain distance.
5. The target recognition and tracking method of a drone according to claim 1, characterized in that: z represents the height of the drone, f represents the focal length of the camera, β is the pitch angle of the camera, α is the angle between the line connecting the two points of the object and the optical axis of the camera, (uo,vo) represents the coordinates of the center point of the image, and , represents the center position of the image target, Indicates the size of the target marker box, where ψ and 0 are the angles between the pitch axis and the yaw axis and the velocity direction respectively. and , represents the camera's viewing angle in the horizontal and vertical directions, Indicates the reference angle of the yaw axis, which controls the rotation of the drone around the yaw axis.