Vision-based unmanned aerial vehicle forward-looking target tracking method, device and storage medium

Abstract

The application relates to the technical field of target tracking, in particular to a visual-based unmanned aerial vehicle front-view target tracking method, which comprises the following steps: acquiring an unmanned aerial vehicle front-view image; acquiring the three-dimensional position of a tracking target center point in a camera coordinate system; converting a point cloud map into an octree map; acquiring the position of the target center point in a world coordinate system; acquiring the complexity of the unmanned aerial vehicle front-view scene; and selecting different tracking modes according to the complexity of the unmanned aerial vehicle front-view scene to determine the unmanned aerial vehicle to track the target. The application can realize real-time and reliable tracking of the front-view target by the unmanned aerial vehicle, and further improve the autonomous ability of the unmanned aerial vehicle.

Description

Technical Field

[0001] The present invention relates to the field of target tracking technology, and in particular to a vision-based unmanned aerial vehicle forward-looking target tracking method, device and storage medium. Background Art

[0002] With the continuous development of drone technology, the use of drones to perform various complex missions has attracted widespread attention. Target tracking, as a drone task, plays a vital role in both civilian and military applications. In aerial photography, using target tracking technology to track the photographer can reduce operating costs. In the military, using target tracking technology to track specific targets can provide more information and dynamics. Visual sensors offer the advantages of low cost, low mass, and high information content, making them widely used in navigation, identification, and tracking. UAV target tracking can be categorized into forward-looking target tracking and downward-looking target tracking. In downward-looking target tracking, the target is often on the ground, so altitude sensors such as barometers and lasers can be used to provide an approximate depth estimate. However, in forward-looking target tracking, the target's depth from the drone must be obtained through other means. Furthermore, UAV target tracking must ensure stable, real-time visual features and a secure tracking path.

[0003] The technologies involved in forward-looking target tracking for drones include visual target detection and tracking, research on its real-time performance, guidance and control law design, binocular depth measurement, and path planning and obstacle avoidance. Traditional target tracking techniques use image processing algorithms to segment the target and then employ a Kalman filter to track it. However, these target tracking algorithms are only suitable for simple tracking scenarios and suffer from poor robustness, making tracking loss a common occurrence. A lightweight, real-time, robust solution with autonomous obstacle avoidance is the primary technical challenge facing drone target tracking. Summary of the Invention

[0004] In order to solve the above technical problems: traditional target tracking technology uses image processing algorithms to segment the target to be tracked, and then uses a Kalman filter to track the target. However, this type of target tracking algorithm is only suitable for simple tracking scenarios, has poor robustness, and is prone to tracking loss. The present invention provides a vision-based forward-looking target tracking method for unmanned aerial vehicles. The method uses the parallax of a binocular camera to obtain the depth information of the target; uses a visual simultaneous positioning and mapping algorithm to determine the position and posture of the unmanned aerial vehicle; establishes a dense point cloud map based on the depth map and the position of the unmanned aerial vehicle and converts it into an octree map; determines the tracking strategy based on the octree map and the position of the target; if there is no obstacle between the unmanned aerial vehicle and the target, a direct tracking solution is adopted; if there is an obstacle between the unmanned aerial vehicle and the target, a different path planning and tracking solution is adopted. The present invention can realize real-time and reliable tracking of forward-looking targets by unmanned aerial vehicles, thereby improving the autonomous capabilities of unmanned aerial vehicles.

[0005] The first object of the present invention is to provide a vision-based forward-looking target tracking method for a UAV, which is used for tracking a forward-looking target by a UAV, comprising:

[0006] Use the binocular camera on the drone to obtain the drone's front view image;

[0007] Obtain a depth map based on the parallax of the binocular camera on the drone;

[0008] The target to be tracked is selected through human-computer interaction, and the target in the left image is tracked using a filtering algorithm; and a rectangular frame is obtained to select the tracked target in the image;

[0009] Obtain the three-dimensional position of the center point of the tracking target in the camera coordinate system based on the tracking target and the depth map in the left eye image;

[0010] According to the tracked target, the left image and the UAV's own posture information are used as input sources, and the UAV's posture information is obtained using the simultaneous localization and mapping algorithm;

[0011] Obtain a point cloud map without tracking targets based on the drone's position information and depth map; convert the point cloud map into an octree map;

[0012] Obtain the position of the target center point in the world coordinate system based on the target's three-dimensional position in the camera coordinate system and the drone's posture information;

[0013] Obtaining the complexity of the UAV forward-view scene according to the position of the tracking target center point in the world coordinate system and the octree map; wherein the complexity of the UAV forward-view scene includes whether the UAV forward-view scene is a complex scene or a simple scene;

[0014] According to the complexity of the UAV's forward-looking scene, different tracking methods are selected to determine the target tracking of the UAV.

[0015] In one embodiment, the selecting of different tracking methods includes:

[0016] If the UAV's forward-looking scene is judged to be a simple scene, the target tracking guidance control law is used to track the target;

[0017] If the forward-looking scene of the UAV is judged to be a complex scene, the path planning algorithm and the path tracking algorithm are used to track the target.

[0018] In one embodiment, the path planning algorithm uses an artificial water flow method for path planning.

[0019] In one embodiment, if the drone's forward-view scene is determined to be a complex scene, target tracking is performed according to the following steps:

[0020] Path planning is performed using the artificial water flow method based on the UAV’s position information and octree map;

[0021] The vector from the current UAV to the nearest uncrossed path point and the vector formed by the nearest uncrossed path point and the next path point of the nearest uncrossed path point are weighted and the vector sum is calculated as the speed direction to be tracked, guiding the UAV to perform path tracking;

[0022] Re-plan the path and track the path according to the changes in the target point position.

[0023] In one embodiment, the process of tracking the target in the left-eye image using the filtering algorithm further includes:

[0024] The target detection algorithm is used to detect the target at regular intervals, and the detection frame with the highest intersection-over-union ratio of the target tracking result is selected as the correction frame. The tracking template is re-determined, and the target is tracked based on the correction frame.

[0025] In one embodiment, the three-dimensional position of the tracking target center point in the camera coordinate system is obtained according to the following steps:

[0026] Map the inscribed ellipse of the tracking target location into the depth map and determine all pixel values ​​within the ellipse;

[0027] Remove holes from the extracted pixel values, that is, remove values ​​with a depth of 0 and values ​​with a depth greater than a preset threshold;

[0028] Perform spatial median filtering on the pixel values ​​and obtain the current depth value of the tracked target by combining the target depth information of the previous frames in the video stream;

[0029] The center point of the tracking target in the pixel coordinate system is mapped to the normalized camera coordinate system through the intrinsic parameter matrix, and then mapped to the camera coordinate system through the depth value to obtain the three-dimensional position of the center point of the tracking target in the camera coordinate system.

[0030] In one embodiment, the point cloud map is obtained according to the following steps:

[0031] Remove the depth information of the tracking target in the depth map, use the remaining depth information in the depth map to map it to the camera coordinate system, and obtain the point cloud in the camera coordinate system;

[0032] The point cloud in the camera coordinate system is converted to the world coordinate system through the posture information of the drone, and filtered through a voxel filter to obtain a point cloud map.

[0033] In one embodiment, the complexity of the drone forward-view scene is obtained according to the following steps:

[0034] In the constructed octree map, the current position of the drone is connected to the position of the tracking target to form a line segment. With the line segment as the central axis, the specified distance between the beginning and the end of the line segment is removed, and the search radius is given to form a cylindrical area.

[0035] The distance relationship between the grid and the center of the sphere can be used to determine whether there is a grid inside the cylindrical area; if there is a grid inside, the current tracking scene is considered to be a complex scene; otherwise, the current tracking scene is considered to be a simple scene.

[0036] A second object of the present invention is to provide a vision-based forward-looking target tracking device for an unmanned aerial vehicle, comprising:

[0037] The target detection and positioning module is used to obtain the drone's front view image using the binocular camera on the drone; obtain a depth map based on the parallax of the binocular camera on the drone; select the target to be tracked through human-computer interaction, and use a filtering algorithm to track the target in the left eye image; and obtain a rectangular box in the image to select the tracked target;

[0038] A positioning and mapping module is used to obtain the three-dimensional position of the center point of the tracking target in the camera coordinate system based on the tracking target and the depth map in the left-eye image; based on the tracked target, the camera image and the drone's own posture information are used as input sources to obtain the drone's posture information using a simultaneous positioning and mapping algorithm; based on the drone's posture information and depth map, a point cloud map is obtained without the tracking target; the point cloud map is converted into an octree map; based on the three-dimensional position of the target in the camera coordinate system and the drone's posture information, the position of the target center point in the world coordinate system is obtained; based on the position of the tracking target center point in the world coordinate system and the octree map, the complexity of the drone's forward-view scene is obtained; wherein the complexity of the drone's forward-view scene includes whether the drone's forward-view scene is a complex scene or a simple scene;

[0039] The UAV target tracking module is used to select different tracking methods according to the complexity of the UAV's forward-looking scene to determine the target tracking of the UAV.

[0040] A third object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which executes the above method when the computer program is executed by a processor.

[0041] The present invention has at least the following beneficial effects:

[0042] This invention provides a vision-based forward-looking target tracking method for unmanned aerial vehicles (UAVs). This method first tracks a pedestrian target in an image. However, drift may occur during the tracking process. Therefore, target detection technology is used to detect the tracked target and correct for tracking errors. Simultaneously, a binocular camera uses parallax to determine a depth map. The tracking frame is matched with the depth map and filtered to obtain the target's depth information, thereby determining the target's position in the camera coordinate system. Furthermore, a simultaneous visual localization and mapping algorithm is used to obtain the UAV's pose information, which, combined with the depth information, enables dense mapping. The positions of the UAV and the target in the map, as well as map information, are used to determine whether obstacle avoidance is required during tracking. If an obstacle exists between the UAV and the tracked pedestrian, path planning and tracking are performed, achieving obstacle avoidance while tracking the pedestrian. If there are no obstacles between the UAV and the tracked pedestrian, target tracking is achieved by controlling the distance between the UAV and the target, the UAV's heading, and the UAV's altitude.

[0043] The present invention tracks pedestrians using correlation filtering; corrects the tracking results using the YOLOv4 algorithm; compresses the deep network model using a lightweight network model; uses the parallax of a binocular camera to obtain the depth information of the target; uses a visual simultaneous localization and mapping algorithm to determine the position of the drone; creates a dense point cloud map based on the depth map and the drone's position and converts it into an octree map; determines the tracking strategy based on the octree map and the target's position; adopts a direct tracking scheme if there are no obstacles between the drone and the target; and adopts different path planning and tracking schemes if there are obstacles between the drone and the target. This invention enables the drone to track forward-looking targets in real time and reliably, thereby improving the drone's autonomous capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A flow chart of a vision-based forward-looking target tracking method for an unmanned aerial vehicle (UAV) is provided as an embodiment of the present invention.

[0046] Figure 2 This is a framework diagram of the target detection algorithm correcting the target tracking result in an embodiment of the present invention.

[0047] Figure 3 This is a diagram showing the improvement of the backbone network during the YOLOv4-tiny lightweight process.

[0048] Figure 4 This is a diagram of the improved sub-network during the YOLOv4-tiny lightweight process.

[0049] Figure 5 Octree map created in real time for the drone during its movement.

[0050] Figure 6 Schematic diagram of the path tracking resultant velocity direction.

[0051] Figure 7 is the distance between the UAV and the pedestrian to be tracked.

[0052] Figure 8 The paths of pedestrians and drones to be tracked.

[0053] Figure 9 This is a schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention. DETAILED DESCRIPTION

[0054] To further illustrate the technical means and effectiveness of the present invention to achieve its intended objectives, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features, and effectiveness of a vision-based forward-looking target tracking method for drones proposed in accordance with the present invention. In the following description, references to "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0055] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0056] In order to solve the problem of tracking forward-looking targets during the execution of UAV missions, the present invention takes real rotorcraft UAVs as the research object and pedestrians as the tracking objects. It relies on visual perception, visual navigation, deep network model compression, multi-view geometry, and guidance and control technology to track and detect and correct pedestrian targets pre-selected in the image, and determine the relative position of pedestrians and UAVs based on binocular parallax information, finally completing real-time tracking of pedestrians.

[0057] The technical approach employed by this invention is to first track the pedestrian target in the image. However, drift may occur during the tracking process. Therefore, target detection technology is used to detect the tracked target and correct for tracking errors. Simultaneously, the binocular camera uses parallax to determine a depth map. Matching the tracking frame with the depth map and filtering it yields the target's depth information, thereby determining the target's position in the camera coordinate system. Furthermore, a simultaneous visual localization and mapping algorithm is used to obtain the drone's pose information, which, combined with the depth information, enables dense mapping. The positions of the drone and target within the map, as well as map information, are used to determine whether obstacle avoidance is required during tracking. If an obstacle exists between the drone and the tracked pedestrian, path planning and tracking are performed, achieving obstacle avoidance while tracking the pedestrian. If there are no obstacles between the drone and the tracked pedestrian, tracking is achieved by controlling the distance, heading, and altitude of the drone.

[0058] It should be noted that the tracking targets in the following implementation are mainly pedestrians.

[0059] The following describes in detail a specific solution of a vision-based forward-looking target tracking method for a UAV provided by the present invention with reference to the accompanying drawings.

[0060] See also Figure 1 FIG. 1 shows a block diagram of a vision-based forward-looking target tracking method for a UAV according to an embodiment of the present invention. The vision-based forward-looking target tracking method for a UAV is used for tracking a forward-looking target by a UAV, and includes:

[0061] S1. Use the binocular camera on the drone to obtain the front view image of the drone; obtain the depth map based on the parallax of the binocular camera on the drone;

[0062] In this embodiment, the left and right eyes of the binocular camera simultaneously capture images of the same scene, and a stereo matching algorithm is used to obtain a disparity map, and then a depth map.

[0063] Before obtaining the depth map, the binocular cameras must be calibrated to obtain the internal and external parameters of the two cameras. The images must be corrected based on the calibration results so that the left and right images are on the same plane. Pixel matching is then performed on the corrected images, and the depth map is finally calculated based on the matching results.

[0064] It should be noted that the image input to the algorithm comes from the forward-looking binocular camera installed on the drone.

[0065] S2. Select the target to be tracked through human-computer interaction, and use the filtering algorithm to track the target in the left image; and obtain a rectangular frame in the image to select the tracked target;

[0066] In this embodiment, the left eye image of the binocular camera is used for target tracking. The target tracking adopts a correlation filter type MOSSE algorithm, which can achieve ultra-high real-time tracking performance.

[0067] During tracking, the target is first selected through human-computer interaction. A filter is then trained using the pre-set target window to model its appearance. For tracking, the motion model is first used to find candidate target locations. The location with the largest response is then used as the new location to be tracked. The filter is then updated online based on the new tracking results.

[0068] To speed up tracking, the algorithm uses the time-frequency characteristics to convert the correlation operation into a product operation. The computational complexity of this process mainly comes from the forward and inverse fast Fourier transform. The time complexity of the algorithm is O(PlogP), where P is the number of pixels in the window. Assume that x represents the input, w represents the filter, and y represents the output. Represents related operations, then:

[0069]

[0070] The correlation theorem converts the correlation operation into a product operation:

[0071] Y=X·W *

[0072] Where X, Y, and W represent the Fourier transform of x, y, and w respectively. * represents the conjugate of W.

[0073] In this embodiment, the process of tracking the target in the left image using the filtering algorithm also includes:

[0074] At regular intervals, target detection is performed using a target detection algorithm. The detection frame with the highest intersection-over-union (IoU) of the target tracking result is selected as the correction frame. The tracking template is re-determined, and the target is tracked based on the correction frame. In this embodiment, the target detection algorithm used is the YOLO detection algorithm.

[0075] It mainly combines the YOLO detection algorithm with the MOSSE single target tracking algorithm, taking advantage of the advantages of both to achieve better tracking effects.

[0076] It's important to note that image-based object detection and visual single-target tracking algorithms are inherently complementary. Image-based object detection detects targets from an image perspective, and therefore requires a global search for the target. Visual single-target tracking algorithms are based on video streams. Based on the tracking results of the previous frame, they use a motion model to establish a correlation between the previous and next frames, thereby achieving tracking. It's important to note that the image used here is the left-eye image, while video images are also fed into the algorithm one by one.

[0077] See also Figure 2 The algorithm framework shown in this embodiment uses the detection results of the YOLO algorithm as observations to correct the tracking results of MOSSE in real time. Figure 2 The tracking thread is the main thread, and the detection thread is the observation thread. After the observation thread is started, the main thread is initially blocked. When the observation thread provides reliable data on the runway in the image—that is, when the detection confidence level is high enough over multiple consecutive frames and there are no significant jumps in the runway's position—the main thread completes initialization using the detected data. After initialization, the main thread continuously tracks the target, updating it when new detection results arrive, thereby addressing tracking drift caused by the limited accuracy of the tracking algorithm.

[0078] Since multiple pedestrian targets may be detected during the detection process, in order to determine the target to be tracked, it is necessary to calculate the intersection and union ratio of each detected target and the tracked target. The position of the target to be tracked is Pt, and all targets detected in the current frame are Pd. i (i=0,1,…,n-1), where n is the number of detected targets, and the intersection-over-union (IOU) of the tracked target and the i-th detection box i for:

[0079]

[0080] Then the target sequence number i is finally selected t It can be expressed as:

[0081]

[0082] Among them, max{} represents the maximum value operation; argmax{} represents the index corresponding to the element with the maximum value.

[0083] IOU0 represents the intersection-and-union ratio of the position of the target to be tracked and the first detected target. The variable names in the intersection-and-union ratio are generally sorted from 0, so the initial intersection-and-union ratio is represented as IOU0; if i t If it is -1, it means the tracking target is lost and needs to be reinitialized for tracking.

[0084] Once the main thread is initialized, it will not be blocked, so the speed of the observation-based track tracking algorithm is roughly the same as the speed of the tracking algorithm. However, this will cause the detection and tracking thread timestamps to be out of sync. This problem is not obvious when the detection speed is fast, but if the detection speed is slow, it may cause a certain lag in the tracking results.

[0085] In order to ensure that the detection algorithm can run in real time on the embedded system, a lightweight YOLOv4-Tiny network is designed in the embodiment.

[0086] There are two feasibility points for network lightweighting: (1) There is only one type of target in pedestrian target tracking, and its feature description itself is not particularly complex. Therefore, the feature extraction part of the network structure can be simplified to a large extent while basically ensuring performance; (2) The pedestrian target tracking task does not have particularly strict restrictions on target detection accuracy, and a slightly larger or smaller positioning box has no direct impact on the back-end pedestrian tracking.

[0087] In this embodiment, see Figure 3 As shown in the figure, the backbone feature extraction network of YOLOv4-tiny has been simplified. For the backbone network, the lightweight network designed in this embodiment has the following simplifications: (1) the number of convolutional layers after the first module is reduced from 32 to 24; (2) the second convolutional module is changed to a maximum pooling module; (3) the last convolution block in the original network is removed. Reducing the number of input channels will reduce the number of parameters, the amount of calculation, and the amount of memory access. Although the computational density of the convolutional layer is reduced, the overall computational density will increase. Therefore, when the feature expression is sufficient, reducing the number of input channels is the simplest and most direct way to improve the real-time performance of inference. In the initial stage of the feature extraction network, the width and height of the feature map are large. For a 3×3 convolution kernel, using a maximum pooling layer instead of a convolutional layer can greatly reduce the amount of calculation.

[0088] See also Figure 4 As shown in FIG, for the YOLOv4-tiny subnetwork, the lightweight subnetwork designed in this embodiment has the following simplifications: (1) the maximum pooling layer of the original network structure is removed, and in order to complete downsampling, the step size of the first convolution module in the subnetwork is set to 2; (2) the two outputs of the original YOLOv4-tiny backbone feature extraction network are both completed by the last TinyYoloBody, but since the sampling operation is moved up after the improvement, it is necessary to hand over the 26×26×256 feature layer output in the original feature extraction network to the penultimate TinyYoloBody for completion.

[0089] S3, obtaining the three-dimensional position of the center point of the tracking target in the camera coordinate system according to the tracking target in the left image and the depth map;

[0090] Specifically, the three-dimensional position of the tracking target center point in the camera coordinate system is obtained according to the following steps:

[0091] Map the inscribed ellipse of the tracking target location into the depth map and determine all pixel values ​​within the ellipse;

[0092] Remove holes from the extracted pixel values, that is, remove values ​​with a depth of 0 and values ​​with a depth greater than a preset threshold;

[0093] The pixel values ​​are spatially median filtered and combined with the target depth information of the previous frames in the video stream to obtain the current depth value of the tracked target; this ensures the spatiotemporal stability of the depth; among them, this depth information can be used to calculate the relative position of the target and the UAV, which is used for the design of the target tracking guidance law.

[0094] The center point of the tracking target in the pixel coordinate system is mapped to the normalized camera coordinate system through the intrinsic parameter matrix, and then mapped to the camera coordinate system through the depth value to obtain the three-dimensional position of the center point of the tracking target in the camera coordinate system.

[0095] In this embodiment, the depth information within the target range is determined by using the rectangular frame of the tracked target and the depth map, removing image depth holes, and combining the characteristics of pedestrians to determine the target depth value, thereby determining the three-dimensional position of the target center in the camera coordinate system; the details are as follows:

[0096] First, the inscribed ellipse of the target frame is given based on the axis-aligned target frame determined by image target tracking. Among them, the axis-aligned target frame is a rectangular frame, in which the axis-aligned target frame is the frame selected for the target at the beginning of tracking. In subsequent tracking, the algorithm will correct this rectangular frame so that it can still frame the target. Target tracking is reflected in the fact that this rectangular frame can always fit the target. Specifically, if the center point of the target frame is (x0 y0), the width and height of the target frame are w and h respectively, then the equation of the inscribed ellipse is:

[0097]

[0098] The depth values ​​of all pixels within the ellipse are extracted to form a sequence S[k]. Values ​​that are 0 or greater than a preset threshold T are discarded. The standard deviation of the remaining pixels is then calculated. If the standard deviation is greater than a specified threshold τ, the major and minor axes of the inscribed ellipse are reduced and recalculated. This cycle continues until the standard deviation is less than τ or the number of pixels in the ellipse is less than 80. Finally, the median of all pixels in the ellipse is taken as the temporary target depth value.

[0099] It should be noted that the preset threshold T and the specified threshold τ are set based on experience and need to consider the actual task

[0100] In order to further ensure the reliability of the target depth value, this embodiment provides a simple and operable time filtering solution. Since the depth of the target changes continuously, it can be considered that the target depth changes slightly between consecutive frames. This embodiment uses a sliding window method to detect whether the data is a jump value. First, the size of the sliding window windows_size is set, which is generally set to 15. The window is filled with data during initialization. After the window is filled with data, the mean mean and standard deviation std of the data in the window are calculated, and the tolerance threshold thres is set at the same time. When the new depth data data is estimated, the following judgment is made:

[0101] mean-thres×std <data<mean+thres×std

[0102] If this condition is met, the first data item in the sliding window is dequeued and the new data item is enqueued. Otherwise, the new data item is considered an outlier and unreliable. Furthermore, this solution uses several counting variables to ensure the reliability of the processing. If there is no positioning data for multiple consecutive frames, or if multiple consecutive frames of data fail to meet the above constraints, the data in the sliding window is considered unreliable and the entire queue must be cleared.

[0103] S4. Based on the tracked target, the image and the UAV's own posture information are used as input sources to obtain the UAV's posture information using a simultaneous localization and mapping algorithm; where the image refers to the left eye image;

[0104] Specifically, the detection result of the tracking target is used as a dynamic target. Point features within the area are not extracted. The camera image and the drone's own posture information are used as input sources, and the drone's posture information is obtained using a simultaneous localization and mapping algorithm. The detection result is a rectangular box that frames the pedestrian in the camera image.

[0105] It should be noted that the camera image refers to the left eye image; the drone's own attitude information is measured by the inertial measurement unit on the drone, which mainly includes the position and attitude of the drone relative to the map coordinate system.

[0106] In this embodiment, a simultaneous localization and mapping algorithm is used to estimate the camera's pose. The drone's pose is then determined based on the camera's mounting angle with the drone. The algorithm uses the coordinate system determined upon successful simultaneous localization and mapping initialization as the world coordinate system. Subsequent dense mapping and drone pose estimation are both based on this coordinate system. Simultaneous localization and mapping is a well-known SLAM technology.

[0107] S5. Obtain a point cloud map without the tracking target based on the UAV's position information and depth map; convert the point cloud map into an octree map;

[0108] The point cloud map is obtained by following the steps below:

[0109] Remove the depth information of the tracking target in the depth map, use the remaining depth information in the depth map to map it to the camera coordinate system, and obtain the point cloud in the camera coordinate system;

[0110] The point cloud in the camera coordinate system is converted to the world coordinate system through the posture information of the drone, and filtered through a voxel filter to obtain a point cloud map.

[0111] In this embodiment, in order to reduce the amount of calculation in the mapping process, the depth map is first downsampled. The downsampling ratio is adjusted according to the actual situation. The depth of the pixel point (u, v) in the pixel coordinate system after downsampling is recorded as depth(u, v). The point set after downsampling is S d , remove the pixels within the pedestrian detection frame in the depth map.

[0112] On this basis, the point cloud information can be converted to the camera coordinate system using the depth information and the camera intrinsic parameter matrix. Specifically, assuming that the camera intrinsic parameter matrix is ​​K:

[0113]

[0114] Among them, f x 、f y Respectively represent the corresponding scaling ratios between the x-axis and y-axis of the camera coordinate system and the x-axis and y-axis of the pixel coordinate system, c x 、c y Represents the offset between the origins of the two coordinate systems. Assume that the coordinates of the corresponding point of (u,v) in the camera coordinate system are P c =[x c y c z c ] T , where T represents the transpose of the vector, which is from a row vector to a column vector, so:

[0115]

[0116] The pixel set contained in all target boxes detected by pedestrians is S p , the final point cloud set obtained in the camera coordinate system is S c ,have:

[0117]

[0118] Then, according to the camera's pose and the relationship between the camera and the body coordinate system, the point cloud can be transformed into the world coordinate system. Specifically, by assuming that the pose of the image is R cw , t cw, where R cw Indicates the pose matrix of the frame image transformed from the world coordinate system to the camera system, t cw Represents the translation matrix, then point P w Projection P in the camera coordinate system c for:

[0119] P c =R cw P w +t cw

[0120] The final point cloud set in the world coordinate system is recorded as S w , we can get:

[0121]

[0122] The point cloud in the world coordinate system may have some noise. The restored point cloud is filtered to remove external points and obtain the point cloud map, which is then converted into an octree map, such as Figure 5 This map is denoted as M and is used for back-end path planning.

[0123] It should be noted that the origin of the body coordinate system is at the center of mass of the drone. The X-axis points toward the nose of the drone, the Z-axis is perpendicular to the X-axis and points downward on the plane's symmetry plane. The Y-axis points to the right of the plane, perpendicular to the OXZ plane. The body coordinate system conforms to the right-hand coordinate system rule (the thumb of the right hand points toward the positive X-axis, the index finger points toward the positive Y-axis, and the middle finger points toward the positive Z-axis, all perpendicular to each other). The body coordinate system uses the same definition as in flight control textbooks. Because the conversion relationship from the camera coordinate system to the world coordinate system cannot be directly obtained, this coordinate system is introduced only to indirectly obtain the conversion relationship from the camera coordinate system to the body coordinate system (camera installation relationship) and the body coordinate system to the world coordinate system (flight attitude).

[0124] S6. Obtain the position of the target center point in the world coordinate system based on the three-dimensional position of the target in the camera coordinate system and the posture information of the UAV;

[0125] In this embodiment, according to the three-dimensional position of the target in the camera coordinate system, that is, the three-dimensional coordinates Pc of the target center point in the camera coordinate system w , then the three-dimensional coordinates of the target center point in the world coordinate system are Po w It can be expressed as:

[0126]

[0127] Among them, R cw Indicates the pose matrix of the frame image transformed from the camera system to the world coordinate system, t cwRepresents the translation matrix.

[0128] Project the target center point to the ground, take the projection point as the center of the ground circle, and use r safe is the radius, h safe A cylinder is formed at the right height, and the inside of the cylinder is a safe zone where drones are prohibited from flying.

[0129] S7. Obtaining a degree of complexity of the forward-view scene of the UAV according to the position of the center point of the tracking target in the world coordinate system and the octree map; wherein the degree of complexity of the forward-view scene of the UAV includes whether the forward-view scene of the UAV is a complex scene or a simple scene;

[0130] The complexity of the drone's forward-looking scene is obtained by following the steps below:

[0131] In the constructed octree map, the current position of the drone is connected to the position of the tracking target to form a line segment. With the line segment as the central axis, the specified distance between the beginning and the end of the line segment is removed, and the search radius is given to form a cylindrical area. Among them, the position of the drone is visible in the octree map, the target position is set, and the position of the tracking target is also the position in the octree map.

[0132] The distance between the grid and the center of the sphere can be used to determine whether there is a grid in the cylindrical area; if there is a grid inside, the current tracking scene is considered to be a complex scene; otherwise, the current tracking scene is considered to be a simple scene. Among them, the grid is automatically generated in the octree map. The grid is the smallest unit of map division, usually a square or rectangular grid unit. The presence of a grid in a certain area indicates that this place is not empty and there are obstacles.

[0133] S8. Select different tracking methods according to the complexity of the UAV's forward-looking scene to determine the UAV's target tracking.

[0134] The selection of different tracking methods includes:

[0135] If the UAV's forward-looking scene is judged to be a simple scene, the target tracking guidance control law is used to track the target;

[0136] If the forward-looking scene of the UAV is judged to be a complex scene, the path planning algorithm and the path tracking algorithm are used to track the target.

[0137] The path planning algorithm adopts artificial water flow method to perform path planning.

[0138] Specifically, if the drone's forward-looking scene is determined to be a complex scene, target tracking is performed according to the following steps:

[0139] Path planning is performed using the artificial water flow method based on the UAV’s position information and octree map;

[0140] The vector from the current UAV to the nearest uncrossed path point and the vector formed by the nearest uncrossed path point and the next path point of the nearest uncrossed path point are weighted and the vector sum is calculated as the speed direction to be tracked, guiding the UAV to perform path tracking;

[0141] Re-plan the path and track the path according to the changes in the target point position.

[0142] It should be noted that, assuming that the three-dimensional coordinates of the drone in the world coordinate system at the current moment are P uav =[x uav y uav z uav ] T , the three-dimensional coordinates of the target center point in the world coordinate system Po w , we can give the line segment seg between the current position of the UAV and the target center point. uav and Po w As the center of the circle, two spheres are formed with a radius of distance d. The intersection points of the two spheres and the line segment seg are P u ' av and Po' w , with P u ' av and Po' w The connected line segment seg' serves as the central axis, and the radius r serves as the protection radius, forming a protection cylinder C. Regions below a certain threshold are removed from this cylinder to form region C'. This removal aims to prevent interference from ground targets. The threshold is set based on engineering experience, taking into account the area occupied by ground targets in different mission scenarios. The octree map grid points are then determined to exist within this region. If so, the scene is considered complex.

[0143] In this embodiment, different tracking methods are selected according to the complexity of the drone's forward-looking scene to determine the drone's target tracking, as follows:

[0144] S10.1. If the UAV's forward-looking scene is determined to be a simple scene, path planning is not performed, and the proposed distance, heading, and altitude control scheme is used for target tracking.

[0145] Specifically, the proposed distance, heading, and altitude control scheme is used for target tracking. Since the UAV does not rotate the roll angle during the tracking process, this condition is fully taken into account during the scheme design process. The position of the target center point in the camera coordinate system is Po c =[x c y c z c ] TAnd the position Po of the target center point in the pixel coordinate system p =[u p v p 1] T (homogeneous coordinates)

[0146] For the distance control scheme, it is mainly used to control the distance that needs to be maintained for forward tracking. According to:

[0147]

[0148] The distance between the UAV and the target can be obtained, that is, the target distance is:

[0149]

[0150] For the heading control scheme, in order to ensure that the coordinates of the pedestrian target center in the pixel coordinate system along the u-axis are always located at the center of the image. The control target center in the pixel coordinate system along the u-axis is always located at the center of the image, which is equivalent to the x-axis coordinate of the target center point in the camera coordinate system being 0. To ensure this condition, assuming that the current pitch angle of the drone is θ, the angle required to rotate is Δψ n According to the transformation of the horizontal heading coordinate system and the camera coordinate system and the definition of the yaw angle, we can know that:

[0151]

[0152] Thus we can get:

[0153]

[0154] That is, horizontal deflection Δψ is required n , using the north-east coordinate system as the navigation coordinate system, Δψ n The nose of the engine deflects to the right, Δψ n When the value is negative, the nose of the aircraft turns to the left. This angle is sent to the flight control system as the desired yaw angle change to ensure that the target is always located in the center of the image along the u-axis.

[0155] The height control scheme controls the coordinate of the target center in the v-axis direction to always be at the center of the image. This is because:

[0156]

[0157] Assuming that the roll angle of the drone is 0, the Z n Axis displacement Δz n for:

[0158]

[0159] That is, the vertical movement Δz in the horizontal heading coordinate systemn Δz n is the timing of the UAV moving downward, Δz n When the value is negative, the drone moves upward. This height change is sent to the flight control system as the desired height change to ensure that the target is always located in the center of the image along the v-axis.

[0160] S10.2. If the UAV's forward-looking scene is judged to be a complex scene, the artificial water flow method is used for path planning, and the proposed path tracking algorithm is used to track the planned path points.

[0161] First, the drone's flight path is planned using an octree map and artificial water flow methods. To ensure the target remains in view, the drone's nose is still locked onto the target using yaw control in S10.1. Yaw control involves using the calculated yaw angle transmitted to the drone for flight control. Path planning provides a series of path points, and the path tracking algorithm must ensure smooth and stable tracking. The path tracking algorithm proposed in this embodiment achieves a good balance between tracking performance and computational complexity, and is a tracking method that can be applied in engineering applications.

[0162] The schematic diagram of the path tracking algorithm proposed in this embodiment is as follows Figure 6 As shown, the tracking effect is as follows Figure 7 、 Figure 8 As shown. Since all path points can be obtained at once in the experiment, if the motion component in the path direction is taken into account, it is equivalent to the algorithm having a certain prediction effect. Based on this idea, this embodiment designs the following path tracking algorithm:

[0163] Assume that the point set queue of path planning is The current position of the drone is P uav =[x uav y uav z uav ] T If the drone has passed a point in the point set path, the point is popped out of the queue. For the i-th point in the point set queue If:

[0164]

[0165] Then pop the point from the queue, otherwise keep the waypoint.

[0166] After clearing the waypoints that have been crossed, assume that the coordinates of the first point in the queue (the waypoint closest to the drone) are The coordinates of the second point are Then the vector from the drone to the nearest point is The vector from the drone to the second point in the queue is The vector from the first waypoint to the second waypoint in the queue is

[0167] To enhance the practicality of the method, this embodiment considers the impact of the combined flight speed of the drone and the distance between the drone and the path point on path tracking. Assuming that the combined flight speed of the drone is VEL, the adaptive coefficient cof can be set to:

[0168]

[0169] where K p is the proportional coefficient, which is generally set between 0 and 1.

[0170] Using the adaptive coefficient, the final tracking speed is expressed as:

[0171]

[0172] This speed is fed into the control loop to track the speed, thereby completing path tracking.

[0173] The present invention provides a vision-based forward-looking target tracking device for an unmanned aerial vehicle, comprising:

[0174] The target detection and positioning module is used to obtain the drone's front view image using the binocular camera on the drone; obtain a depth map based on the parallax of the binocular camera on the drone; select the target to be tracked through human-computer interaction, and use a filtering algorithm to track the target in the left eye image; and obtain a rectangular box in the image to select the tracked target;

[0175] A positioning and mapping module is used to obtain the three-dimensional position of the center point of the tracking target in the camera coordinate system based on the tracking target and the depth map in the left-eye image; based on the tracked target, the camera image and the drone's own posture information are used as input sources to obtain the drone's posture information using a simultaneous positioning and mapping algorithm; based on the drone's posture information and depth map, a point cloud map is obtained without the tracking target; the point cloud map is converted into an octree map; based on the three-dimensional position of the target in the camera coordinate system and the drone's posture information, the position of the target center point in the world coordinate system is obtained; based on the position of the tracking target center point in the world coordinate system and the octree map, the complexity of the drone's forward-view scene is obtained; wherein the complexity of the drone's forward-view scene includes whether the drone's forward-view scene is a complex scene or a simple scene;

[0176] The UAV target tracking module is used to select different tracking methods according to the complexity of the UAV's forward-looking scene to determine the target tracking of the UAV.

[0177] See also Figure 9 As shown, Figure 9 This is a schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention.

[0178] It should be noted that Figure 9 This is a structural diagram of the hardware operating environment of the computer device.

[0179] See also Figure 9 As shown, the computer device may include: a processor 101, such as a CPU, a memory 105, a user interface 103, a network interface 104, and a communication bus 102; wherein the communication bus 102 is used to realize the connection and communication between these components. The user interface 103 may include a display screen, an input unit such as a keyboard, and the user interface 103 may optionally include a standard wired interface and a wireless interface. The network interface 104 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 105 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 105 may optionally be a storage device independent of the aforementioned processor 101.

[0180] Those skilled in the art will understand that Figure 9 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0181] See also Figure 9 As shown, the memory 105 as a storage medium may include an operating system, a network communication module, a user interface module and a computer program. Among them, the operating system is a program that manages and controls the hardware and software resources of a computer device, and the operation of the computer program and other software or programs.

[0182] exist Figure 9 In the computer device shown, the user interface 103 is mainly used to connect to the terminal and communicate data with the terminal; the network interface 104 is mainly used to communicate data with the background server; the processor 101 can be used to call the computer program stored in the memory 105.

[0183] In this embodiment, the computer device includes: a memory 105, a processor 101, and a computer program stored in the memory 105 and executable on the processor, wherein:

[0184] When the processor 101 calls the computer program stored in the memory 105, it performs the following operations:

[0185] Use the binocular camera on the drone to obtain the drone's front view image;

[0186] Obtain a depth map based on the parallax of the binocular camera on the drone;

[0187] The target to be tracked is selected through human-computer interaction, and the target in the left image is tracked using a filtering algorithm; and a rectangular frame is obtained to select the tracked target in the image;

[0188] Obtain the three-dimensional position of the center point of the tracking target in the camera coordinate system based on the tracking target and the depth map in the left eye image;

[0189] According to the tracked target, the left image and the UAV's own posture information are used as input sources, and the UAV's posture information is obtained using the simultaneous localization and mapping algorithm;

[0190] Obtain a point cloud map without tracking targets based on the drone's position information and depth map; convert the point cloud map into an octree map;

[0191] Obtain the position of the target center point in the world coordinate system based on the target's three-dimensional position in the camera coordinate system and the drone's posture information;

[0192] Obtaining the complexity of the UAV forward-view scene according to the position of the tracking target center point in the world coordinate system and the octree map; wherein the complexity of the UAV forward-view scene includes whether the UAV forward-view scene is a complex scene or a simple scene;

[0193] According to the complexity of the UAV's forward-looking scene, different tracking methods are selected to determine the target tracking of the UAV.

[0194] When the processor 101 calls the computer program stored in the memory 105, it also performs the following operations:

[0195] The following tracking methods are available:

[0196] If the UAV's forward-looking scene is judged to be a simple scene, the target tracking guidance control law is used to track the target;

[0197] If the forward-looking scene of the UAV is judged to be a complex scene, the path planning algorithm and the path tracking algorithm are used to track the target.

[0198] When the processor 101 calls the computer program stored in the memory 105, it also performs the following operations:

[0199] The path planning algorithm adopts artificial water flow method to perform path planning.

[0200] When the processor 101 calls the computer program stored in the memory 105, it also performs the following operations:

[0201] If the drone's forward-looking scene is judged to be complex, target tracking is performed according to the following steps:

[0202] Path planning is performed using the artificial water flow method based on the UAV’s position information and octree map;

[0203] The vector from the current UAV to the nearest uncrossed path point and the vector formed by the nearest uncrossed path point and the next path point of the nearest uncrossed path point are weighted and the vector sum is calculated as the speed direction to be tracked, guiding the UAV to perform path tracking;

[0204] Re-plan the path and track the path according to the changes in the target point position.

[0205] When the processor 101 calls the computer program stored in the memory 105, it also performs the following operations:

[0206] The process of tracking the target in the left image by using the filtering algorithm also includes:

[0207] The target detection algorithm is used to detect the target at regular intervals, and the detection frame with the highest intersection-over-union ratio of the target tracking result is selected as the correction frame. The tracking template is re-determined, and the target is tracked based on the correction frame.

[0208] When the processor 101 calls the computer program stored in the memory 105, it also performs the following operations:

[0209] The three-dimensional position of the center point of the tracking target in the camera coordinate system is obtained according to the following steps:

[0210] Map the inscribed ellipse of the tracking target location into the depth map and determine all pixel values ​​within the ellipse;

[0211] Remove holes from the extracted pixel values, that is, remove values ​​with a depth of 0 and values ​​with a depth greater than a preset threshold;

[0212] Perform spatial median filtering on the pixel values ​​and obtain the current depth value of the tracked target by combining the target depth information of the previous frames in the video stream;

[0213] The center point of the tracking target in the pixel coordinate system is mapped to the normalized camera coordinate system through the intrinsic parameter matrix, and then mapped to the camera coordinate system through the depth value to obtain the three-dimensional position of the center point of the tracking target in the camera coordinate system.

[0214] When the processor 101 calls the computer program stored in the memory 105, it also performs the following operations:

[0215] The point cloud map is obtained by following the steps below:

[0216] Remove the depth information of the tracking target in the depth map, use the remaining depth information in the depth map to map it to the camera coordinate system, and obtain the point cloud in the camera coordinate system;

[0217] The point cloud in the camera coordinate system is converted to the world coordinate system through the posture information of the drone, and filtered through a voxel filter to obtain a point cloud map.

[0218] When the processor 101 calls the computer program stored in the memory 105, it also performs the following operations:

[0219] The complexity of the drone's forward-looking scene is obtained by following the steps below:

[0220] In the constructed octree map, the current position of the drone is connected to the position of the tracking target to form a line segment. With the line segment as the central axis, the specified distance between the beginning and the end of the line segment is removed, and the search radius is given to form a cylindrical area.

[0221] The distance relationship between the grid and the center of the sphere can be used to determine whether there is a grid inside the cylindrical area; if there is a grid inside, the current tracking scene is considered to be a complex scene; otherwise, the current tracking scene is considered to be a simple scene.

[0222] In summary, the technical approach adopted by this invention is to first track the pedestrian target in the image. However, drift may occur during the tracking process. Therefore, target detection technology is used to detect the tracked target and correct for tracking errors. Simultaneously, the binocular camera uses parallax to determine a depth map. Matching the tracking frame with the depth map and filtering it yields the target's depth information, thereby determining the target's position in the camera coordinate system. Furthermore, a simultaneous visual localization and mapping algorithm is used to obtain the drone's pose information, which, combined with the depth information, enables dense mapping. The positions of the drone and target within the map, as well as map information, are used to determine whether obstacle avoidance is required during tracking. If an obstacle exists between the drone and the tracked pedestrian, path planning and tracking are performed, achieving obstacle avoidance while tracking the pedestrian. If there are no obstacles between the drone and the tracked pedestrian, tracking is achieved by controlling the distance between the drone and the target, the drone's heading, and the drone's altitude.

[0223] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. Furthermore, the foregoing descriptions of specific embodiments of this specification are provided. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential sequence shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0224] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0225] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A vision-based UAV forward-looking target tracking method, characterized in that: Used for UAV tracking of forward-looking targets, including: Use the binocular camera on the drone to obtain the drone's front view image; Obtain a depth map based on the parallax of the binocular camera on the drone; The target to be tracked is selected through human-computer interaction, and the target in the left image is tracked using a filtering algorithm; and a rectangular frame is obtained to select the tracked target in the image; Obtain the three-dimensional position of the center point of the tracking target in the camera coordinate system based on the tracking target and the depth map in the left eye image; According to the tracked target, the left image and the UAV's own posture information are used as input sources, and the UAV's posture information is obtained using the simultaneous localization and mapping algorithm; Obtain a point cloud map without tracking targets based on the drone's position information and depth map; convert the point cloud map into an octree map; Obtain the position of the target center point in the world coordinate system based on the target's three-dimensional position in the camera coordinate system and the drone's posture information; Obtaining the complexity of the UAV forward-view scene according to the position of the tracking target center point in the world coordinate system and the octree map; wherein the complexity of the UAV forward-view scene includes whether the UAV forward-view scene is a complex scene or a simple scene; According to the complexity of the UAV's forward-looking scene, different tracking methods are selected to determine the target tracking of the UAV; The complexity of the drone's forward-looking scene is obtained by following the steps below: In the constructed octree map, the current position of the drone is connected to the position of the tracking target to form a line segment. With the line segment as the central axis, the specified distance between the beginning and the end of the line segment is removed, and the search radius is given to form a cylindrical area. The distance relationship between the grid and the center of the sphere can be used to determine whether there is a grid inside the cylindrical area; if there is a grid inside, the current tracking scene is considered to be a complex scene; otherwise, the current tracking scene is considered to be a simple scene.

2. The vision-based forward-looking target tracking method for unmanned aerial vehicles according to claim 1, characterized in that: The following tracking methods are available: If the UAV's forward-looking scene is judged to be a simple scene, the target tracking guidance control law is used to track the target; If the forward-looking scene of the UAV is judged to be a complex scene, the path planning algorithm and the path tracking algorithm are used to track the target.

3. The vision-based forward-looking target tracking method for unmanned aerial vehicles according to claim 2, characterized in that: The path planning algorithm adopts artificial water flow method to perform path planning.

4. The vision-based forward-looking target tracking method for unmanned aerial vehicles according to claim 3, characterized in that: If the drone's forward-looking scene is judged to be complex, target tracking is performed according to the following steps: Path planning is performed using the artificial water flow method based on the UAV’s position information and octree map; The vector from the current UAV to the nearest uncrossed path point and the vector formed by the nearest uncrossed path point and the next path point of the nearest uncrossed path point are weighted and the vector sum is calculated as the speed direction to be tracked, guiding the UAV to perform path tracking; Re-plan the path and track the path according to the changes in the target point position.

5. The vision-based UAV forward-looking target tracking method according to claim 1, characterized in that: The process of tracking the target in the left image by using the filtering algorithm also includes: The target detection algorithm is used to detect the target at regular intervals, and the detection frame with the highest intersection-over-union ratio of the target tracking result is selected as the correction frame. The tracking template is re-determined, and the target is tracked based on the correction frame.

6. The vision-based forward-looking target tracking method for unmanned aerial vehicles according to claim 1, characterized in that: The three-dimensional position of the center point of the tracking target in the camera coordinate system is obtained according to the following steps: Map the inscribed ellipse of the tracking target location into the depth map and determine all pixel values ​​within the ellipse; Remove holes from the extracted pixel values, that is, remove values ​​with a depth of 0 and values ​​with a depth greater than a preset threshold; Perform spatial median filtering on the pixel values ​​and obtain the current depth value of the tracked target by combining the target depth information of the previous frames in the video stream; The center point of the tracking target in the pixel coordinate system is mapped to the normalized camera coordinate system through the intrinsic parameter matrix, and then mapped to the camera coordinate system through the depth value to obtain the three-dimensional position of the center point of the tracking target in the camera coordinate system.

7. The vision-based forward-looking target tracking method for unmanned aerial vehicles according to claim 1, characterized in that: The point cloud map is obtained by following the steps below: Remove the depth information of the tracking target in the depth map, use the remaining depth information in the depth map to map it to the camera coordinate system, and obtain the point cloud in the camera coordinate system; The point cloud in the camera coordinate system is converted to the world coordinate system through the posture information of the drone, and filtered through a voxel filter to obtain a point cloud map.

8. A vision-based UAV forward-looking target tracking device, characterized in that: include: The target detection and target positioning module is used to obtain the front view image of the drone using the binocular camera on the drone; Obtain a depth map based on the parallax of the binocular camera on the drone; The target to be tracked is selected through human-computer interaction, and the target in the left image is tracked using a filtering algorithm; and a rectangular frame is obtained to select the tracked target in the image; The localization and mapping module is used to obtain the three-dimensional position of the center point of the tracking target in the camera coordinate system based on the tracking target and the depth map in the left-view image. Based on the tracked target, the camera image and the drone's own posture information are used as input sources to obtain the drone's posture information using a simultaneous localization and mapping algorithm. Based on the drone's posture information and depth map, a point cloud map is obtained without the tracking target. Convert the point cloud map into an octree map; Obtaining the position of the target center point in the world coordinate system based on the three-dimensional position of the target in the camera coordinate system and the posture information of the drone; obtaining the complexity of the drone's forward-view scene based on the position of the target center point in the world coordinate system and the octree map; wherein the complexity of the drone's forward-view scene includes whether the drone's forward-view scene is a complex scene or a simple scene; The UAV target tracking module is used to select different tracking methods according to the complexity of the UAV forward-looking scene to determine the target tracking of the UAV; The complexity of the drone's forward-looking scene is obtained by following the steps below: In the constructed octree map, the current position of the drone is connected to the position of the tracking target to form a line segment. With the line segment as the central axis, the specified distance between the beginning and the end of the line segment is removed, and the search radius is given to form a cylindrical area. The distance relationship between the grid and the center of the sphere can be used to determine whether there is a grid inside the cylindrical area; if there is a grid inside, the current tracking scene is considered to be a complex scene; otherwise, the current tracking scene is considered to be a simple scene.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images