Unmanned aerial vehicle high-altitude pedestrian tracking system design method based on monocular vision
Through the improved Tiny-YOLOv3 algorithm and coordinate conversion technology, combined with multi-index optimization functions, a monocular visual drone high-altitude pedestrian tracking system was built, which solved the precise positioning and computational complexity of binocular visual ranging, and achieved high altitude Accurate positioning and tracking of pedestrians at long distances improves the real-time and adaptability of the system.
Patent Information
- Application Number
- CN202310645525.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-02
- Publication Date
- 2024-12-03
AI Technical Summary
In the existing drone target positioning technology, binocular visual distance measurement has problems such as limited precise positioning distance, large calculation amount, high cost, complex structure and easy error, and it is difficult to adapt to high-altitude long-distance tracking applications. Monocular vision Positioning can effectively avoid these problems but needs to be improved to improve accuracy and applicability.
The improved Tiny-YOLOv3 algorithm is used for target recognition, and the TensorRT accelerates model inference is used to remove camera distortions, and the accurate target positioning of a single-eye camera is achieved, and the target tracking is carried out through multi-index optimization function. Visual drone high-altitude pedestrian tracking system.
It realizes accurate positioning and tracking of pedestrians at high altitude and long distances, reduces computing costs and structural complexity, improves the real-time and adaptability of the system, and is suitable for applications in a wide range of scenarios.
Smart Images

Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
(I) Technical field
[0001] The invention relates to the technical field of unmanned aerial vehicle visual tracking, and constructs a monocular vision unmanned aerial vehicle pedestrian tracking system suitable for high altitude and long distance. (II) Background technology
[0002] With the rapid development of social economy, drones have been widely used in more and more scenarios in recent years. Target tracking technology based on drones has been a research hotspot in recent years, and several research institutions at home and abroad have also carried out corresponding research on this. After nearly 20 years of technical research and development, drone target tracking systems have developed into a huge system based on various visual positioning, multiple optimization algorithms, and multiple control methods. The accuracy, breadth, and speed of target tracking have also been greatly improved.
[0003] In order to obtain the precise position of the target being tracked, the drone needs to calculate and locate the target depth. Currently, the drone target positioning technologies mainly include the following:
[0004] 1) Target positioning based on LiDAR: LiDAR detects the target position by emitting laser beams and receiving echoes. LiDAR has the advantages of high measurement accuracy, good real-time performance, and strong anti-interference ability. However, due to the disorder, sparsity, and low acquisition quality of point cloud data, its development is relatively slow.
[0005] 2) Target positioning based on UWB: UWB signals have strong anti-interference ability, high positioning accuracy, and can penetrate a variety of materials, so they are suitable for unmanned platforms. The UWB positioning system consists of three parts: the base station, the target node to be positioned, and the host computer target position solution platform. However, due to its short positioning distance and susceptibility to interference from metal and liquid, it is usually used for positioning in indoor environments.
[0006] 3) Target positioning based on millimeter wave radar: Millimeter wave radar refers to radar with an operating wavelength in the range of 1 to 10 mm and a corresponding frequency of 30 to 300 GHz. Millimeter wave radar has the advantages of strong anti-interference ability, good concealment, and long measurable distance. However, it is difficult for millimeter wave radar to obtain the required high transmission power and corresponding low-loss transmission feeder, and the loss is large during long-distance transmission in the atmosphere.
[0007] 4) Vision-based target positioning: This technology uses a camera to shoot the target and determines the target's position and motion trajectory by processing and analyzing the image. It is currently the most widely used and is mainly divided into target positioning based on binocular vision or monocular vision.
[0008] The principle of binocular vision ranging is to first calculate the parallax of the two images to obtain the distance of the target in front, then process the image through the target detection algorithm, and then combine the coordinate system transformation to solve the world coordinate system position of the object. Although binocular stereo ranging has the advantages of directly obtaining depth information, high positioning accuracy, and good measurement effect, it still has some disadvantages. Among them, the precise positioning distance of binocular vision positioning is limited, which greatly limits its application scope; at the same time, it has a very large amount of calculation and requires very high performance of the computing unit; and it requires complex stereo calibration of the binocular vision sensor, which is very easy to produce errors, and the low matching rate of the images collected by the two cameras and the changes in the external parameters of the binocular camera may lead to reduced ranging accuracy. Another difficulty lies in the registration effect of binocular vision. Although the stereo matching algorithm can be used to improve the ranging accuracy during the ranging process, the principle of ranging itself requires that the error between the two lenses should be as small as possible. If the errors of the two lenses are large, then the difficulty of the algorithm for later adjustment will be much greater.
[0009] Monocular ranging can effectively avoid these problems. Moreover, this method uses only one camera, which is low in cost. It can obtain the depth information of the target in the image to a certain extent. The system structure is relatively simple, the operation speed is fast, the computing power requirement is not high, the cost is low, and the volume is small. The error of monocular ranging can be effectively processed by adjusting the algorithm in the later stage. Moreover, the monocular vision positioning distance is long, which is suitable for tracking applications at high altitude and long distance, and other scenes that do not require extremely high precision. This is also the research focus of the present invention.
[0010] Therefore, realizing high-altitude pedestrian positioning of UAVs based on monocular vision and further combining it with the current tracking algorithm can better solve the application bottleneck of UAVs in a wide range of scenarios and has good application value and engineering value. (III) Summary of the invention
[0011] The present invention aims to propose a design method for a high-altitude pedestrian tracking system of an unmanned aerial vehicle based on monocular vision, focusing on the positioning of moving pedestrians on the ground during the flight of the unmanned aerial vehicle.
[0012] The present invention comprises the following main steps:
[0013] The first step is to identify pedestrians. The improved Tiny-YOLOv3 algorithm is used for target identification. Based on the traditional Tiny-YOLOv3 algorithm, in order to ensure that the detection algorithm meets the real-time requirements, TensorRT is used to accelerate the model inference and improve the speed of target detection.
[0014] The second step is to locate the target using the pixel coordinates obtained by target recognition. This step is the key and main content of the present invention. To obtain the coordinates of the pedestrian in the three-dimensional coordinate system, it is necessary to perform coordinate conversion on the pixel coordinate system, obtain the coordinates of the target in the camera coordinate system through the intermediate coordinate system image coordinate system, and then convert the coordinates into the coordinates in the body coordinate system and the world coordinate system. In the process of coordinate conversion, the camera has no inclination, the camera has an inclination, the pedestrian is located on the central axis of the camera screen, and the pedestrian is not located on the central axis of the camera screen. The pedestrian height prior condition of 1.7m is introduced to deduce the distance, and the effect of accurate target positioning can be achieved using only a monocular camera. At the same time, in response to different problems in specific applications, software programs are used to remove camera distortion, perform distance filtering, etc. to ensure the stability and practicality of the algorithm.
[0015] In the third step, for the target tracking problem, the existing target tracking algorithm is used to propose an optimization function containing multiple indicators, and an optimization model is established. Taking multiple indicators such as tracking effect, energy consumption, and obstacle avoidance as considerations, the strategy of rolling and real-time execution of path planning during flight is implemented to achieve the target tracking effect.
[0016] The above three steps have built a complete UAV high-altitude pedestrian tracking system based on monocular vision. The purpose is to form a comprehensive system with various technical advantages, solve existing problems such as binocular vision positioning, and contribute to the vigorous development of UAV technology.
[0017] The beneficial results of the present invention include: for the specific application of tracking pedestrians using drones, starting from the relative position relationship between the target and the camera, the various postures of the drone and the onboard camera are comprehensively considered, and the accurate target positioning of pedestrians is achieved through coordinate system conversion and solution, and combined with the target tracking algorithm, the whole process from identifying pedestrians, locating pedestrians to tracking pedestrians is completed. The whole process completely relies on the monocular camera, and while ensuring the positioning accuracy, it has the following characteristics: the algorithm is simple and has higher real-time performance; the positioning distance is long, which is suitable for tracking high-altitude and long-distance pedestrian targets; it has strong adaptability and a wider range of application scenarios. (IV) Description of the drawings
[0018] Figure 1 Basic relationship diagram of position solution;
[0019] Figure 2 Coordinate system transformation diagram;
[0020] Figure 3 Schematic diagram of relative positions when the camera has no tilt and the pedestrian is located on the central axis of the camera image;
[0021] Figure 4 Schematic diagram of the relative positions when the camera has no tilt and the pedestrian is in any position;
[0022] Figure 5 Schematic diagram of the camera's position when it is tilted;
[0023] Figure 6 Graph showing the relationship between the distance between the camera and pedestrians;
[0024] Figure 7 Experimental curve of high-altitude pedestrian positioning and ranging of UAV based on monocular vision;
[0025] Figure 8 The overall workflow diagram of the present invention; (V) Specific implementation methods
[0026] The present invention relates to a design method of a high-altitude pedestrian tracking system for an unmanned aerial vehicle based on monocular vision, focusing on the positioning of target pedestrians moving on the ground during the flight of the unmanned aerial vehicle. The contents are described as follows in conjunction with the accompanying drawings:
[0027] In terms of hardware configuration, the present invention selects NVIDIA Jetson TX2 microcomputer as the onboard computer, uses flight control Pixhawk4 as the basic controller of the UAV platform, and adopts Leixun C-RTK positioning module as the UAV positioning sensor to ensure that the UAV has stable flight performance, load capacity and computing power. In target recognition, a deep learning algorithm, the improved Tiny-YOLOv3 algorithm, is used for target recognition. On the basis of the traditional Tiny-YOLOv3 algorithm, in order to ensure that the real-time performance of the detection algorithm meets the requirements, TensorRT is used to accelerate the model inference and improve the speed of target detection.
[0028] The following focuses on the method of using monocular vision to locate target pedestrians:
[0029] like Figure 1 As shown in the figure, pedestrian positioning requires the target position to be solved, that is, the target coordinates in the actual space are obtained according to the target position in the camera image, which involves the following Figure 2 The coordinate system conversion process shown in Figure 2. The position solution process is the process of recursive conversion between coordinate systems. The specific transformation process is described as follows.
[0030] 1) Conversion from pixel coordinate system {pixel} to image coordinate system {image}
[0031] a. Get the target pixel coordinates
[0032] The pixel coordinate system {pixel} is a two-dimensional plane coordinate system. Looking from the small hole to the projection surface, the upper left corner of the projection surface is the origin O pix, the uv axis coincides with both sides of the projection surface. The target detection algorithm is used to obtain the target detection frame, and the pixel coordinates (u1, v1) and (u2, v2) of the upper left corner and lower right corner edge points of the target are obtained.
[0033] b. Solve the image coordinates
[0034] The image coordinate system {image} is a two-dimensional plane coordinate system, which is introduced here as an intermediate coordinate system.
[0035] The transformation from the pixel coordinate system {pixel} to the image coordinate system {image} is shown in formula (1). There are four parameters, dx, dy, u0, and v0, which can be directly obtained by camera calibration. dx and dy represent the physical size of each pixel in the x and y directions of the image plane, respectively.
[0036]
[0037] The coordinates (x′1, y′1) and (x′2, y′2) of the person in the image coordinate system {image} are obtained by the above formula.
[0038] 2) Conversion from image coordinate system {image} to camera coordinate system {camera}
[0039] a. Direct conversion of coordinate system
[0040] {camera} is the camera coordinate system, the origin is located at the pinhole, the z-axis coincides with the optical axis, and the x-axis coincides with the optical axis. c Axis and y c The axis is parallel to the projection plane. In this system, the camera is tilted downward at an angle α.
[0041] The conversion from {image} to {camera} is shown in formula (2), where f is the focal length of the camera.
[0042]
[0043] Given the focal length of the camera f and (x′1, y′1) and (x′2, y′2), we can substitute into formula (2) to get x c1 / z c1 ,y c1 / z c1 and x c2 / z c2 ,y c2 / z c2 . To completely solve (x c1 ,y c1 , z c1 ) and (x c2 ,y c2 , z c2 ) six parameters, introducing additional conditions for pedestrian tracking.
[0044] For the distance measurement of pedestrians by a monocular camera, the pedestrian's prior height h=1.7m is preset.
[0045] Next, we will start from the relative position relationship between the camera and the pedestrian and consider the distance measurement between the camera and the target in different situations. First, consider the most special case, that is, the camera has no tilt angle and the target is located on the central axis of the camera screen, such as Figure 3 If the upper and lower edges of the target are connected to the upper and lower vertical points of the target in the camera image, the two connecting lines must intersect on the camera optical axis, and the intersection point is the origin O of the camera coordinate system. From the similarity relationship of triangles, the straight-line distance z′ from the camera to the target can be obtained c As shown in formula (3).
[0046]
[0047] Since the {image} to {camera} axes are in the same direction, ∠γ1 and ∠γ2 can be calculated at {image} with the help of the camera focal length f.
[0048]
[0049] Furthermore, if Figure 4 As shown, the above conditions are relaxed: the camera has no tilt angle and the target position is arbitrary. Since the difference between the height h of the person in reality and the vertical coordinate of the target in the camera image, which the calculation depends on, is reflected on the y' axis, it is only necessary to project the real target and the pixel target onto the camera optical axis, that is, ignore the offset in the x' direction and only consider the y' direction. Therefore, this extension does not change the straight-line distance z' from the camera to the person c , the above conclusion still holds.
[0050] In practical applications, high-altitude drones need to shoot distant ground targets, and the camera needs to be tilted downward, with the tilt angle α. And the target position is arbitrary, such as Figure 5 As shown in the figure, the camera coordinate system is rotated -α degrees around the x-axis compared to when the camera has no tilt.
[0051] Because the above two cases use the condition that the camera optical axis is perpendicular to the height of the person and the height of the object to derive the triangle similarity relationship, but this condition is no longer true when the camera has an inclination angle and the optical axis is tilted. Therefore, the camera coordinate system {camera′} is introduced when the camera has no inclination angle, with its origin at the pinhole and the coordinate axis z′ c Parallel to the ground.
[0052] At this time, z′ c The axis is perpendicular to the height of the person, and the upper and lower vertical points of the target are at z′ c The projection on the axis is the same. It is stipulated that the positive direction of the intermediate variables ∠1 and ∠2 is the same as the camera tilt angle α, and the two angles are the same as z′ cThe axis angles are ∠γ1+α and ∠γ2+α respectively. After the angles are corrected, the above conclusions still hold.
[0053] At this time, the person is vertical to the axis z' c The distance difference is the height h of the person, and the two points above and below the person are perpendicular to the axis z′ c The intercept on is the straight-line distance between the camera and the person.
[0054]
[0055] Quantitative relationship such as Figure 6 As shown, it can be solved as follows:
[0056]
[0057] This method can be used to solve the coordinates of the person in the camera coordinate system {camera} (x c1 ,y c1 , z c1 ) and (x c2 ,y c2 , z c2 ).
[0058] b. Remove camera distortion
[0059] In the process of using a monocular camera, camera distortion is inevitable, so a software program is needed to remove the distortion. The principle is shown in formula (7).
[0060]
[0061] Among them, k1, k2, k3, p1, and p2 are all obtained by camera internal calibration and are the intrinsic parameters of the camera. The value of r can be calculated by formula (8).
[0062]
[0063] In the process of transforming from the image coordinate system to the camera coordinate system, it is necessary to use formula (7) to perform distortion correction.
[0064] c. Distance filtering
[0065] Both the body and the target are in dynamic change. At the same time, the lag of the detection program and its own errors will also lead to unstable detection results. In order to obtain a relatively smooth and practical distance value, the distance needs to be sampled at a high frequency and sliding filtered.
[0066] The UAV distance measurement program uses a sampling frequency of 5 Hz, that is, five distance data are collected per second, and the sliding window size is 5, that is, the five nearest distance values are taken for filtering each time.
[0067] 3) Conversion from camera coordinate system {camera} to body coordinate system {body}
[0068] {body} is the body coordinate system, which is fixed on the aircraft or plane, and its origin is located at the center of mass of the aircraft, x B The y axis is the longitudinal axis of the body, with the positive direction pointing toward the head; B The axis is perpendicular to the aircraft reference plane and points to the right of the aircraft, z B Axis and x B Axis and y B The axes form a right-handed coordinate system, with positive pointing downward.
[0069] In this system, the camera maintains an inclination angle of α downward, and α is between 20° and 30°. The target coordinates (x B ,y B , z B ) can be directly obtained from formula (9).
[0070]
[0071] 4) Conversion from body coordinate system {body} to world coordinate system {world}
[0072] The transformation from the body coordinate system to the world coordinate system is a rigid body transformation, which means that the object is not deformed. The rigid body transformation mainly relies on rotation transformation and translation transformation. The rotation transformation involves the three angles of the drone: pitch angle θ, yaw angle ψ and tilt angle Since the camera is fixed to the drone, the camera's real-world angle is affected by these angles. The following discusses the impact of the three angles on the positioning results.
[0073] ① The influence of the pitch angle θ on the positioning result is reflected in the camera inclination angle, which is opposite to the positive direction of α, so the final angle between the camera optical axis and the ground is α′=α-θ.
[0074] ② Because the yaw angle ψ does not change the angle between the camera's vertical and horizontal directions and the ground, it does not change the relative relationship between the camera and the person.
[0075] ③ Tilt angle This causes the person's height to tilt when shooting, so that the person's preset height h is no longer the difference between the vertical coordinates of the upper and lower points. Since the drone will hardly flip significantly in the steady state in this experiment, the tilt angle The impact on shooting positioning can be ignored. It is only involved in the conversion from the body coordinate system {body} to the world coordinate system {world}, and does not affect the conversion process of the camera coordinate system {camera}.
[0076] The world coordinate system {world} corresponds to the navigation coordinate system NED coordinate system and ENU coordinate system. Assume that the position of the drone in the NED coordinate system is (x0, y0, z0), and the pitch angle, tilt angle, and yaw angle are θ, ψ, Then the rotation matrix R from the body coordinate system to the NED coordinate system is X It is expressed as formula (10).
[0077]
[0078] The coordinates in the ENU coordinate system (x E ,y E , z E ) can be directly obtained from formula (11).
[0079]
[0080] The above fully explains the whole process from using a monocular camera to obtain the target pedestrian pixel points, to modeling and deducing the positional relationship between targets of different forms and the camera, and then transforming the coordinate system to finally complete the pedestrian target positioning.
[0081] After using a monocular camera to identify and locate the target, the coordinates of the target in the world coordinate system are obtained. Then, the existing target tracking algorithm is used to solve the problem of target tracking. The main steps to achieve target tracking are: under the constraints of the drone's state constraints, control input constraints, obstacle avoidance constraints, etc., an optimization function containing multiple indicators is proposed, an optimization model is established, and performance indicators such as control effect, cost, and energy consumption are comprehensively considered, and fuzzy processing is performed in the form of membership function. In this way, the strategy of rolling real-time execution of path planning is implemented in the actual control process to achieve the effect of target tracking.
Claims
1. A design method for a high-altitude pedestrian tracking system for an unmanned aerial vehicle based on monocular vision, characterized in that: The following steps are involved: S1. Capturing an image by using a monocular camera onboard a drone, and obtaining a target recognition result of the image by image target recognition, wherein the target recognition result is information about pedestrians in the image; S2. By using a monocular ranging method based on a pinhole model, the height of the pedestrian is preset, and the different positional relationships between the pedestrian and the camera are considered to determine the distance information between the pedestrian and the camera in the target recognition result, and then the accurate target positioning of the pedestrian is achieved through coordinate system conversion and calculation; S3. By combining with the target tracking algorithm, the target tracking effect is achieved.
2. The design method of the unmanned aerial vehicle high-altitude pedestrian tracking system based on monocular vision according to claim 1 is characterized in that: In step S1, the improved Tiny-YOLOv3 algorithm is used for target recognition, and TensorRT is used to accelerate the model inference to improve the speed of target detection.
3. The design method of the unmanned aerial vehicle high-altitude pedestrian tracking system based on monocular vision according to claim 1 is characterized in that: The specific steps of S2 are: S21, completing the conversion from the pixel coordinate system to the image coordinate system by obtaining the target pixel coordinates and then solving the image coordinates; S22, by presetting the height of the pedestrian as 1.7 meters and the positional relationship between the camera and the pedestrian, a camera distortion removal algorithm and a distance filtering algorithm are applied to complete the conversion from the image coordinate system to the camera coordinate system; S23, completing the conversion from the camera coordinate system to the body coordinate system by constructing a coordinate conversion matrix; S24. Consider the influence of the different relative positions of the airborne camera, the UAV and the pedestrian on the pitch angle, yaw angle and tilt angle, comprehensively consider the influence of the pitch angle, yaw angle and tilt angle on the coordinate transformation, construct the coordinate transformation matrix, and complete the transformation from the body coordinate system to the world coordinate system.
4. The design method of the unmanned aerial vehicle high-altitude pedestrian tracking system based on monocular vision according to claim 1 is characterized in that: In step S3, under the constraints of the drone’s state constraints, an optimization model is established, and performance indicators such as control effect, cost, and energy consumption are comprehensively considered. The path planning strategy is implemented in real time during the actual control process to achieve the target tracking effect.
5. The design method of the unmanned aerial vehicle high-altitude pedestrian tracking system based on monocular vision according to claim 1 or 3 is characterized in that: The conversion from the image coordinate system to the camera coordinate system in step S22 is specifically completed as follows: the height of the pedestrian is preset to be 1.7 meters. In the three cases where the camera has no tilt and the target is located on the central axis of the camera screen, the camera has no tilt and the target position is arbitrary, and the camera has a tilt and the target position is arbitrary, a monocular ranging model is constructed based on the pinhole model. The straight-line distance from the camera to the target pedestrian can be obtained based on the geometric relationship, and then the camera distortion removal algorithm and the distance filtering algorithm are used to complete the conversion from the image coordinate system to the camera coordinate system.
6. The design method of the unmanned aerial vehicle high-altitude pedestrian tracking system based on monocular vision according to claim 1 or 3 is characterized in that: The conversion from the body coordinate system to the world coordinate system in step S24 is completed specifically by adding the effects of the pitch angle, the tilt angle, and the yaw angle accordingly when calculating the distance between the pedestrian and the camera, thereby completing the conversion from the body coordinate system to the world coordinate system.
