An unmanned aerial vehicle target tracking method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose a kind of unmanned vehicle target tracking method, device, electronic equipment and storage medium, involve target tracking technical field, comprising: obtaining the first image data collected by unmanned vehicle, determine the first region where target object is in first image and the second region associated with target object in second image;Obtain the first response peak in the corresponding response value of each pixel point in first region and the second response peak in the corresponding response value of each pixel point in second region, obtain the first tracking frame information of target object in first region and the second tracking frame information of target object in second region;Determine the first difference coefficient between first response peak and second response peak, determine the second difference coefficient between first tracking frame information and second tracking frame information;Determine the degree of occlusion of target object based on first difference coefficient and second difference coefficient, according to the degree of occlusion control instruction is sent to unmanned vehicle, realize unmanned vehicle real-time target object tracking.

Description

Technical Field

[0001] This invention relates to the field of target tracking technology, and in particular to a method, apparatus, electronic device, and storage medium for tracking unmanned aerial vehicles (UAVs). Background Technology

[0002] In recent years, drones have been widely used in agricultural plant protection, power line inspection, traffic patrol, security, and firefighting. Target tracking is an important research direction in computer vision, which involves accurately locating the position and trajectory of targets of interest in video sequences. Applying target tracking technology to drones helps improve their intelligence level.

[0003] In practical tracking applications, the target region of interest is often affected by environmental factors, leading to inaccurate algorithm calculations, unstable target tracking, and ultimately target loss. Directly using response peaks to determine target loss has a high false positive rate and lacks robustness. While re-detecting the target using continuous inter-frame difference methods can detect targets moving slowly, it is unsuitable for tracking fast-moving targets. Therefore, existing technologies struggle to accurately determine whether a target has been lost. Summary of the Invention

[0004] To address the existing technical problems, embodiments of the present invention provide a method, apparatus, electronic device, and storage medium for tracking unmanned aerial vehicles (UAVs).

[0005] To achieve the above objectives, the technical solution of this invention is implemented as follows:

[0006] In a first aspect, embodiments of the present invention provide a method for tracking a target using a drone, the method comprising:

[0007] Acquire first image data collected by the drone, the first image data including a first image and a second image; the second image is a frame following the first image.

[0008] Determine the first region where the target object is located in the first image and the second region associated with the target object in the second image; perform target tracking processing on the pixels in the first region and the second region to obtain the first response peak value in the response value corresponding to each pixel in the first region and the second response peak value in the response value corresponding to each pixel in the second region, and obtain the first tracking box information of the target object in the first region and the second tracking box information of the target object in the second region.

[0009] Determine a first difference coefficient between the first response peak and the second response peak, and determine a second difference coefficient between the first tracking box information and the second tracking box information;

[0010] The degree of occlusion of the target object is determined based on the first difference coefficient and the second difference coefficient. Control commands are sent to the drone according to the degree of occlusion of the target object. The control commands are used to adjust the flight state of the drone.

[0011] In the above scheme, sending control commands to the drone based on the degree of occlusion of the target object includes:

[0012] When the degree of occlusion of the target object is less than a first threshold, a first control command is sent to the drone. The first control command is used to instruct the drone to adjust its flight direction so that the drone can track the target object.

[0013] When the target object is occluded to a degree greater than or equal to the first threshold, a second control command is sent to the drone. The second control command is used to instruct the drone to maintain a hovering state so that the drone can continuously acquire image data in the hovering state.

[0014] In the above scheme, the first control command further includes target object motion information; the method further includes:

[0015] Obtain the coordinates of the first center point of the pixel corresponding to the first response peak in the pixel coordinate system; the pixel corresponding to the first response peak corresponds to the center point of the first tracking box;

[0016] Obtain the coordinates of the second center point of the pixel corresponding to the second response peak in the pixel coordinate system; the pixel corresponding to the second response peak corresponds to the center point of the second tracking box;

[0017] In the pixel coordinate system, determine the line connecting the coordinates of the first center point and the coordinates of the second center point, and obtain the angle between the line and the horizontal axis. The value of the angle is greater than or equal to 0 degrees and less than 90 degrees.

[0018] Based on the coordinates of the first center point, the coordinates of the second center point, and the included angle, the motion information of the target object is obtained.

[0019] The method in the above scheme further includes:

[0020] Obtain the second image data collected by the drone in a hovering state, and re-detect the target object in the second image data;

[0021] Obtain the object detection box in the third image from the second image data, and track the target object based on the object detection box.

[0022] In the above scheme, determining the first region where the target object in the first image is located and the second region associated with the target object in the second image includes:

[0023] Obtain the first region where the target object is located in the first image, and the first center position of the first region;

[0024] Determine the second center position in the second image that corresponds to the first center position, and determine the second region based on the second center position.

[0025] In the above scheme, determining the first difference coefficient between the first response peak and the second response peak includes:

[0026] The first difference coefficient is determined based on the difference between the first response peak value and the second response peak value and the coefficient of variation;

[0027] The value of the change coefficient is related to the comparison result of the first response peak and the second threshold and the comparison result of the third response peak and the second threshold. The third response peak is the maximum value of the response values ​​of each pixel in the region associated with the target object in the fourth image. The fourth image is the frame image following the first image.

[0028] In the above scheme, determining the second difference coefficient between the first tracking box information and the second tracking box information includes:

[0029] The area of ​​the first tracking frame is determined based on the first tracking frame information, and the area of ​​the second tracking frame is determined based on the second tracking frame information;

[0030] The second difference coefficient is determined based on the ratio between the area of ​​the first tracking frame and the area of ​​the second tracking frame.

[0031] Secondly, embodiments of the present invention also provide a drone target tracking device, the device comprising:

[0032] The first acquisition module is used to acquire first image data collected by the UAV, the first image data including a first image and a second image; the second image is a frame image after the first image; and to determine a first region where the target object in the first image is located and a second region associated with the target object in the second image.

[0033] The tracking module is used to perform target tracking processing on the pixels of the first region and the second region, respectively obtaining the first response peak value in the response value corresponding to each pixel in the first region and the second response peak value in the response value corresponding to each pixel in the second region, and respectively obtaining the first tracking box information of the target object in the first region and the second tracking box information of the target object in the second region;

[0034] The first determining module is used to determine a first difference coefficient between the first response peak and the second response peak, and to determine a second difference coefficient between the first tracking box information and the second tracking box information;

[0035] The control module is used to determine the degree of occlusion of the target object based on the first difference coefficient and the second difference coefficient, and to send control commands to the drone according to the degree of occlusion of the target object. The control commands are used to adjust the flight state of the drone.

[0036] In the above scheme, the control module further includes:

[0037] The first control submodule is used to send a first control command to the drone when the degree of occlusion of the target object is less than a first threshold. The first control command is used to instruct the drone to adjust its flight direction so that the drone can track the target object.

[0038] The second control submodule is used to send a second control command to the drone when the target object is occluded to a degree greater than or equal to the first threshold. The second control command is used to instruct the drone to maintain a hovering state so that the drone can continuously collect image data in the hovering state.

[0039] In the above scheme, the first control command also includes target object motion information;

[0040] The first control submodule is further configured to obtain the coordinates of the first center point of the pixel corresponding to the first response peak in the pixel coordinate system; the pixel corresponding to the first response peak corresponds to the center point of the first tracking frame; obtain the coordinates of the second center point of the pixel corresponding to the second response peak in the pixel coordinate system; the pixel corresponding to the second response peak corresponds to the center point of the second tracking frame; determine the line connecting the first center point coordinates and the second center point coordinates in the pixel coordinate system, and obtain the angle between the line and the horizontal axis, wherein the angle is greater than or equal to 0 degrees and less than 90 degrees; and obtain the motion information of the target object based on the first center point coordinates, the second center point coordinates and the angle.

[0041] In the above scheme, the device further includes:

[0042] The second acquisition module is used to acquire second image data collected by the UAV in a hovering state, and to re-detect the target object in the second image data;

[0043] The second determining module is used to obtain the object detection box in the third image in the second image data, and to track the target object based on the object detection box.

[0044] In the above scheme, the first acquisition module is used to obtain the first region where the target object is located in the first image, and the first center position of the first region; determine the second center position in the second image corresponding to the first center position, and determine the second region based on the second center position.

[0045] In the above scheme, the first determining module is used to determine the first difference coefficient based on the difference between the first response peak and the second response peak and the change coefficient; wherein, the value of the change coefficient is related to the comparison result of the first response peak and the second threshold and the comparison result of the third response peak and the second threshold, the third response peak being the maximum value among the response values ​​of each pixel in the region associated with the target object in the fourth image; the fourth image being the frame image following the first image.

[0046] In the above scheme, the first determining module is used to determine the area of ​​the first tracking frame based on the first tracking frame information, determine the area of ​​the second tracking frame based on the second tracking frame information, and determine the second difference coefficient based on the ratio between the area of ​​the first tracking frame and the area of ​​the second tracking frame.

[0047] Thirdly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the unmanned aerial vehicle target tracking method described in the first aspect of the present invention.

[0048] Fourthly, embodiments of the present invention also provide an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor, wherein, when the processor runs the computer program, it executes the steps of the UAV target tracking method described in the first aspect of the present invention.

[0049] The technical solution provided by this invention involves acquiring first image data collected by a drone, the first image data including a first image and a second image; the second image being a frame following the first image; determining a first region where a target object is located in the first image and a second region associated with the target object in the second image; performing target tracking processing on the pixels in the first region and the second region, respectively obtaining a first response peak value in the response values ​​corresponding to each pixel in the first region and a second response peak value in the response values ​​corresponding to each pixel in the second region, and respectively obtaining first tracking box information of the target object in the first region and second tracking box information of the target object in the second region; determining a first difference coefficient between the first response peak value and the second response peak value, and determining a second difference coefficient between the first tracking box information and the second tracking box information; determining the degree of occlusion of the target object based on the first difference coefficient and the second difference coefficient, and sending control commands to the drone according to the degree of occlusion of the target object, the control commands being used to adjust the flight state of the drone. This invention combines multi-frame response peaks and tracking box size change data to determine whether the target object is completely occluded, which can reduce the occlusion false positive rate and improve the robustness of the occlusion determination method; it generates UAV control information based on the target object's motion information, and then uses the network to transmit control commands to the UAV, realizing real-time target object tracking by the UAV. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0051] Figure 1 This is a schematic diagram of an unmanned aerial vehicle (UAV) target tracking system according to an embodiment of the present invention;

[0052] Figure 2 This is a flowchart illustrating the UAV target tracking method according to an embodiment of the present invention. Figure 1 ;

[0053] Figure 3 This is a flowchart illustrating the UAV target tracking method according to an embodiment of the present invention. Figure 2 ;

[0054] Figure 4 This is a schematic diagram of the structure of the UAV target tracking device according to an embodiment of the present invention;

[0055] Figure 5This is a schematic diagram of the hardware composition structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0057] The UAV target tracking method of this invention can be applied to Figure 1 The drone target tracking system shown includes a drone system, a drone cloud platform, and an information transmission system. The drone system transmits image data captured by a camera to the drone cloud platform via an onboard communication terminal, using either a public network (e.g., 4G / 5G mobile communication networks) or a private network, for monitoring and drone flight management. Specifically, the camera captures images of the target object, and the onboard communication terminal transmits the captured image data. The drone cloud platform processes the captured image data to obtain drone flight control commands, which are then sent to the drone via the communication system for flight control.

[0058] It should be noted that, Figure 1 The system structures shown are only some optional system structures for the UAV target tracking method applied in the embodiments of the present invention. The systems applied in the embodiments of the present invention are not limited to those shown. Figure 1 As shown.

[0059] Figure 2 This is a flowchart illustrating the UAV target tracking method according to an embodiment of the present invention. Figure 1 ;like Figure 2 As shown, the method includes:

[0060] Step 101: Acquire first image data collected by the drone, the first image data including a first image and a second image; the second image is a frame image after the first image; determine the first region where the target object in the first image is located and the second region associated with the target object in the second image;

[0061] Step 102: Perform target tracking processing on the pixels in the first region and the second region, respectively obtain the first response peak value in the response value corresponding to each pixel in the first region and the second response peak value in the response value corresponding to each pixel in the second region, and respectively obtain the first tracking box information of the target object in the first region and the second tracking box information of the target object in the second region;

[0062] Step 103: Determine the first difference coefficient between the first response peak and the second response peak, and determine the second difference coefficient between the first tracking box information and the second tracking box information;

[0063] Step 104: Determine the degree of occlusion of the target object based on the first difference coefficient and the second difference coefficient, and send control commands to the drone according to the degree of occlusion of the target object. The control commands are used to adjust the flight state of the drone.

[0064] The drone target tracking method of this embodiment is applied to a drone target tracking device. The drone target tracking device can be installed in an electronic device with processing capabilities, such as a personal computer, mobile terminal, or server, or implemented by a processor executing a computer program. For example, the electronic device is... Figure 1 The drone cloud platform shown.

[0065] In this embodiment, the drone is controlled to fly to a position where its camera system (or camera) can capture clear images including the target object, and the drone's camera system acquires images in real time. The drone cloud platform obtains the image data acquired by the drone (denoted as first image data), determines the target object to be tracked from the first image in the first image data, and determines the first region where the target object is located in the first image. Then, based on the first region in the first image, a second region associated with the target object in the next frame of the first image (i.e., the second image) is determined; wherein, the second region may be the region where the target object is located in the second image, or, since the target object may be moving quickly, the second region may only include part of the target object, or even not include the target object. For example, the second image is k frames away from the first image, and the value of k can be set according to the movement speed of the target object; the faster the target object moves, the larger the value of k.

[0066] In this embodiment, the Discriminative Scale Space Tracking (DSST) algorithm is used to obtain correlation filter response maps for the pixels in the first region and the second region, respectively. Each response value in the filter response map corresponds to a pixel in the first region and the second region, respectively. The response peak value (denoted as the first response peak value) and the response peak value (denoted as the second response peak value) of each pixel in the first region and the second region are obtained, respectively. The response peak value is the maximum value among the response values ​​in the filter response map. The pixel corresponding to the response peak value is the center position of the target object.

[0067] Using the pixel location corresponding to the first response peak as the center position of the target object in the first region, and the pixel location corresponding to the second response peak as the center position of the target object in the second region, first tracking box information of the target object in the first region and second tracking box information of the target object in the second region are obtained respectively. The first and second tracking box information include at least the dimensions of the tracking box of the target object in the first and second images, such as the width and height of the tracking box.

[0068] In this embodiment, a first difference coefficient, determined based on the first response peak and the second response peak, characterizes the degree of difference or change in the center position of the target object in the first image and the second image. A second difference coefficient, determined based on the first tracking box information and the second tracking box information, characterizes the degree of difference or change in the size of the target object's tracking box in the first image and the second image. The degree of occlusion of the target object is determined based on the change in the center position of the target object (first difference coefficient) and the change in the size of the target object's tracking box (second difference coefficient). The degree of occlusion indicates the extent to which the target object is occluded, or it can also characterize the presence, partial presence, or absence of the target object in the second image. Then, control commands are sent to the drone based on the degree of occlusion of the target object. The control commands are used to adjust the drone's flight state so that the drone can detect the target object more quickly.

[0069] By employing the technical solution of this invention, combining multi-frame response peaks and tracking box size change data to determine whether the target object is completely occluded, the occlusion false positive rate can be reduced and the robustness of the occlusion determination method can be improved; control commands are sent to the UAV based on the degree of occlusion of the target object, thereby realizing real-time target object tracking by the UAV.

[0070] In some optional embodiments of the present invention, determining the first region where the target object in the first image is located and the second region associated with the target object in the second image includes: obtaining the first region where the target object in the first image is located and the first center position of the first region; determining the second center position in the second image corresponding to the first center position, and determining the second region based on the second center position.

[0071] In this embodiment, for example, the UAV cloud platform reads a frame of image data from the first image data collected by the UAV, i.e., the first image. The operator can manually select the first area where the target object to be tracked is located in the first image using a rectangle. For example, this includes a horizontal crosshair marked with an X at the upper left corner of the rectangle. t-k The top left corner is labeled Y.t-k The width of the rectangle is denoted as W. t-k The height is denoted as H. t-k The first center position of the first region is obtained.

[0072] Read the second image after reading the first image, and determine the position in the second image relative to the first center. Corresponding second center position Then based on the second center position Obtain the second region associated with the target object. For example, this could be based on the second center location. Centered on a predetermined area, a second region is defined by a predetermined width and a predetermined height; wherein, the dimensions of the predetermined width and predetermined height can be the same as the width of the first region (e.g., W). t-k ) and height (e.g., H) t-k The width of the first region is the same as or greater than the width of the first region (e.g., W). t-k ) and height (e.g., H) t-k ).

[0073] In this embodiment, the first region and the second region can be used as search regions for target tracking processing. For example, the DSST algorithm can be used to track the pixels in the first region and the second region to obtain the first response peak in the first region and the second response peak in the second region, and to obtain the first tracking box information of the target object in the first region and the second tracking box information of the target object in the second region, respectively.

[0074] In some optional embodiments of the present invention, determining the first difference coefficient between the first response peak and the second response peak includes: determining the first difference coefficient based on the difference between the first response peak and the second response peak and a change coefficient; wherein the value of the change coefficient is related to the comparison result of the first response peak and the second threshold and the comparison result of the third response peak and the second threshold, the third response peak being the maximum value among the response values ​​corresponding to each pixel in the region associated with the target object in the fourth image; the fourth image being the frame following the first image.

[0075] In this embodiment, the next frame of the first image is denoted as the fourth image, and the maximum value of the response values ​​of each pixel in the region associated with the target object in the fourth image is denoted as the third response peak.

[0076] For example, the peak value of the first response is denoted as r. t-k The second peak response is denoted as r. t The peak value of the third response is denoted as r. t-k+1The first difference coefficient is determined based on the difference between the first response peak value and the second response peak value and the coefficient of variation. The first difference coefficient is denoted as R. t The coefficient of variation is denoted as α.

[0077]

[0078] When the values ​​of the first response peak and the third response peak are both less than or equal to the second threshold λ, the value of the change coefficient α is 1; otherwise, the value of the change coefficient α is 0.

[0079] In some optional embodiments of the present invention, determining the second difference coefficient between the first tracking frame information and the second tracking frame information includes: determining the area of ​​the first tracking frame based on the first tracking frame information, determining the area of ​​the second tracking frame based on the second tracking frame information, and determining the second difference coefficient based on the ratio between the area of ​​the first tracking frame and the area of ​​the second tracking frame.

[0080] In this embodiment, for example, the information of the first tracking frame includes the tracking frame width w. t-k and height h t-k The information of the second tracking box includes the tracking box width w. t and height h t Therefore, the area of ​​the second tracking box is w. t *h t The area of ​​the first tracking box is w t-k *h t-k The second difference coefficient is denoted as S. t .

[0081]

[0082] When the area w of the second tracking box t *h t The area w of the first tracking box t-k *h t-k When the ratio between them is less than the second threshold μ, the second difference coefficient S t The value of is 1; otherwise, the second difference coefficient S t The value of is 0, and the value of the second threshold μ can be set according to the actual scenario.

[0083] In this embodiment, for example, the degree of occlusion of the target object determined based on the first difference coefficient and the second difference coefficient can be represented by an occlusion determination function, the value of which characterizes the degree of occlusion of the target object. Let the occlusion determination function be denoted as f. t The occlusion determination function can be expressed by the following expression:

[0084] f t=S t ·R t =S t ·α·(r t-k -r t )

[0085] If the occlusion determination function f is calculated t If the value of the occlusion determination function is greater than or equal to the first threshold ν, the target object is determined to be completely occluded and target re-detection is required; if the calculated value of the occlusion determination function is less than the first threshold ν, the target object is determined not to be completely occluded and the drone is controlled to track the target object.

[0086] This invention also provides a method for tracking a target using a drone. This embodiment, based on the foregoing embodiments, specifically describes step 104 of the foregoing embodiments. In this embodiment, sending control commands to the drone based on the degree of occlusion of the target object includes two possible outcomes, such as... Figure 3 As shown, step 104 may include:

[0087] Step 104a: When the degree of occlusion of the target object is less than a first threshold, a first control command is sent to the drone. The first control command is used to instruct the drone to adjust its flight direction so that the drone can track the target object.

[0088] Step 104b: When the target object is occluded to a degree greater than or equal to the first threshold, a second control command is sent to the drone. The second control command is used to instruct the drone to maintain a hovering state so that the drone can continuously acquire image data in the hovering state.

[0089] The technical solution of this invention sends control commands to the drone based on the degree of occlusion of the target object. When the target object is not completely occluded, the drone is controlled to track the target object. When the target object is completely occluded, target re-detection is performed, thereby achieving real-time target tracking.

[0090] In some optional embodiments, the first control command further includes target object motion information; the method further includes: obtaining the coordinates of a first center point of the pixel corresponding to the first response peak in a pixel coordinate system; the pixel corresponding to the first response peak corresponds to the center point of the first tracking frame; obtaining the coordinates of a second center point of the pixel corresponding to the second response peak in the pixel coordinate system; the pixel corresponding to the second response peak corresponds to the center point of the second tracking frame; determining a line connecting the first center point coordinates and the second center point coordinates in the pixel coordinate system, obtaining the angle between the line and the horizontal axis, wherein the angle is greater than or equal to 0 degrees and less than 90 degrees; and obtaining the target object motion information based on the first center point coordinates, the second center point coordinates, and the angle.

[0091] In this embodiment, for example, the coordinates of the first center point of the pixel corresponding to the first response peak in the pixel coordinate system are obtained. The pixel corresponding to the first response peak corresponds to the center point of the first tracking box; the coordinates of the second center point of the pixel corresponding to the second response peak in the pixel coordinate system are obtained. The pixel corresponding to the second response peak corresponds to the center point of the second tracking box; in the pixel coordinate system, a line connecting the coordinates of the first center point and the coordinates of the second center point is determined, and the angle θ between the line and the horizontal axis is obtained. The angle θ satisfies the following expression; wherein the value range of the angle θ is greater than or equal to 0 degrees and less than 90 degrees:

[0092]

[0093] Determine the direction of movement of the target object, if The target object then moves to the right front of the drone; if The target object then moves to the right rear of the drone; if The target then moves to the left front of the drone; if The target object moves to the left rear of the drone; the angle between the target object's direction of movement and the horizontal direction is θ.

[0094] Based on the motion information of the target object, control commands are sent to the drone, instructing the drone to adjust its flight direction according to the motion direction of the target object, so that the drone can track the target object.

[0095] The technical solution of this invention uses the motion information of the target object to calculate the control command of the UAV, and then uses the network to transmit the control command to the airborne communication terminal on the UAV in real time. Then, the flight status is controlled by the UAV flight control system to realize the UAV automatically tracking the target.

[0096] In some optional embodiments, the method further includes: obtaining second image data collected by the UAV in a hovering state, re-detecting the target object in the second image data; obtaining an object detection box in a third image of the second image data, and tracking the target object based on the object detection box.

[0097] In this embodiment, when the degree of occlusion of the target object is greater than or equal to the first threshold, it indicates that the target object is completely occluded. In this case, the UAV cloud platform issues a second control command to make the UAV hover and allow the UAV to re-acquire image data.

[0098] For example, the drone cloud platform obtains three consecutive frames of images from the second image data captured by the drone in a hovering state, and converts the three frames from RGB format to grayscale images, denoted as I. T I T+1 I T+2 ;I T with I T+1 Subtract and binarize the results, and denote the result as I. b1 ;I T+1 with I T+2 Subtract and binarize the results, and denote the result as I. b2 ; to I b1 and I b2 Perform a bitwise AND operation, and denote the result as I. and ; to I and The opening operation is performed by first performing an erosion operation, followed by a dilation operation (the specific implementation process can be found in the general processing procedure for opening operations). The result is denoted as I. open ; in I open The algorithm detects the outline of an object. If the area of ​​the object's outline is within a given range, the outline is preserved, and the minimum bounding rectangle of the outline is obtained. i ,y i ,w i ,h i ) represents the bounding box of the i-th object, where x i The x-coordinate and y-coordinate of the top left corner of the rectangle i The ordinate of the top left corner of the rectangle, w i The width and h of the rectangle i This represents the height of the rectangle.

[0099] The center position of the object's rectangular frame is calculated and denoted as . connect Center position of the target object before it was lost Calculate the angle θ between the connecting line and the horizontal direction. i If |θ-θi If the value of |θ - θ is less than or equal to the set threshold ρ, proceed to the next step; if |θ - θ i If the value of | is greater than the set threshold ρ, then read the next frame image and repeat the above steps.

[0100] The scale of the selected object rectangles can be fine-tuned using the following method:

[0101]

[0102] Among them, w t-k-1 *h t-k-1 w is the area of ​​the tracking bounding box before the target object was lost. i *h i Let be the area of ​​the rectangle containing the i-th object.

[0103] If the value of ε is equal to 1, then the rectangle's size is not adjusted; if the value of ε is greater than 1, then w is set to... i =w t-k-1 h i =h t-k-1 .

[0104] Fine-tune the position of the selected object rectangle by moving the center of the object rectangle k pixels up, down, left, and right respectively, to obtain the coordinates of the four center positions:

[0105] At the four central locations, with (w i ,h i Four object detection boxes are obtained using a scale of 1, 2, and 3. The color histogram of each object detection box region is extracted and denoted as histogram. q q = 1, 2, 3, 4, at the initial center position of the object's bounding box, the tracking bounding box (w) before the target object was lost. t-k-1 h t-k-1 For the region, extract the color histogram when the target is not occluded, denoted as histogram. t-k-1 ;

[0106] Hist is calculated using the Barthel coefficient. q With hist t-k-1 The similarity between the two objects is calculated, and the object detection box with the highest similarity is denoted as (x). re ,y re ,w re ,h re ):w re =w i ,h re =h i ,

[0107] Set the object detection box (x re ,yre ,w re ,h re The region contained within is taken as the target object and the tracking continues.

[0108] Using the technical solution of this invention, in the case of target object loss, the three-frame difference method can be used to detect the lost target object. By combining the historical motion information of the target object and the detected target object detection box, the target object position is updated and the target object is continued to be tracked.

[0109] The UAV target tracking method of this invention will be described in detail below with reference to a specific example. The method includes:

[0110] Step 301: Acquire the first image data collected by the drone, the first image data including a first image and a second image; the second image is a frame image after the first image.

[0111] Step 302: Obtain the first region where the target object is located in the first image, and the first center position of the first region; determine the second center position in the second image corresponding to the first center position, and obtain the second region based on the second center position.

[0112] Step 303: Perform target tracking processing on the pixels in the first region and the second region, respectively obtain the first response peak value in the response value corresponding to each pixel in the first region and the second response peak value in the response value corresponding to each pixel in the second region, and respectively obtain the first tracking box information of the target object in the first region and the second tracking box information of the target object in the second region.

[0113] Step 304: Determine the first difference coefficient based on the difference between the first response peak and the second response peak and the change coefficient; wherein, the value of the change coefficient is related to the comparison result of the first response peak and the second threshold and the comparison result of the third response peak and the second threshold, the third response peak being the maximum value among the response values ​​of each pixel in the region associated with the target object in the fourth image; the fourth image is the frame image following the first image.

[0114] Step 305: Determine the area of ​​the first tracking frame based on the first tracking frame information, and determine the area of ​​the second tracking frame based on the second tracking frame information; determine the second difference coefficient based on the ratio between the area of ​​the first tracking frame and the area of ​​the second tracking frame.

[0115] Step 306: Determine the degree of occlusion of the target object based on the first difference coefficient and the second difference coefficient.

[0116] Step 307a: When the degree of occlusion of the target object is less than the first threshold, it is determined that the target object is not completely occluded.

[0117] Step 308a: Obtain the coordinates of the first center point of the pixel corresponding to the first response peak in the pixel coordinate system; the pixel corresponding to the first response peak corresponds to the center point of the first tracking box;

[0118] Obtain the coordinates of the second center point of the pixel corresponding to the second response peak in the pixel coordinate system; the pixel corresponding to the second response peak corresponds to the center point of the second tracking box;

[0119] In the pixel coordinate system, determine the line connecting the coordinates of the first center point and the coordinates of the second center point, and obtain the angle between the line and the horizontal axis. The value of the angle is greater than or equal to 0 degrees and less than 90 degrees.

[0120] Based on the coordinates of the first center point, the coordinates of the second center point, and the included angle, the motion information of the target object is obtained.

[0121] Step 309a: Send a first control command to the drone, the first control command being used to instruct the drone to adjust its flight direction so that the drone tracks the target object.

[0122] Step 307b: When the degree of occlusion of the target object is greater than or equal to the first threshold, it is determined that the target object is completely occluded.

[0123] Step 308b: Obtain the second image data collected by the UAV in hovering state, and re-detect the target object in the second image data.

[0124] Step 309b: Obtain the object detection box in the third image from the second image data, and track the target object based on the object detection box.

[0125] Here, steps 301 to 309 can be described in detail in the foregoing embodiments, and will not be repeated here. By employing the technical solution of this embodiment, combining multi-frame response peaks and tracking box size change data to determine whether the target object is completely occluded, the occlusion false positive rate can be reduced, and the robustness of the occlusion determination method can be improved. Based on the target object's motion information, UAV control information is generated, and then the control commands are transmitted to the UAV via the network to achieve real-time target object tracking by the UAV. In the case of target object loss, the three-frame difference method is used, combining the target object's historical motion information and the detected target object, to update the target object's position and continue tracking.

[0126] Based on the foregoing embodiments, this invention also provides a drone target tracking device. Figure 4 This is a schematic diagram of the structure of the UAV target tracking device according to an embodiment of the present invention, as shown below. Figure 4 As shown, the device includes:

[0127] The first acquisition module 201 is used to acquire first image data collected by the UAV, the first image data including a first image and a second image; the second image is a frame image after the first image; and to determine a first region where the target object in the first image is located and a second region associated with the target object in the second image.

[0128] The tracking module 202 is used to perform target tracking processing on the pixels of the first region and the second region, respectively obtaining the first response peak value in the response value corresponding to each pixel in the first region and the second response peak value in the response value corresponding to each pixel in the second region, and respectively obtaining the first tracking box information of the target object in the first region and the second tracking box information of the target object in the second region.

[0129] The first determining module 203 is used to determine a first difference coefficient between the first response peak and the second response peak, and to determine a second difference coefficient between the first tracking box information and the second tracking box information;

[0130] The control module 204 is used to determine the degree of occlusion of the target object based on the first difference coefficient and the second difference coefficient, and to send control commands to the drone according to the degree of occlusion of the target object. The control commands are used to adjust the flight state of the drone.

[0131] In some optional embodiments of the present invention, the control module 204 further includes:

[0132] The first control submodule is used to send a first control command to the drone when the degree of occlusion of the target object is less than a first threshold. The first control command is used to instruct the drone to adjust its flight direction so that the drone can track the target object.

[0133] The second control submodule is used to send a second control command to the drone when the target object is occluded to a degree greater than or equal to the first threshold. The second control command is used to instruct the drone to maintain a hovering state so that the drone can continuously collect image data in the hovering state.

[0134] In some optional embodiments of the present invention, the first control command further includes target object motion information;

[0135] The first control submodule is further configured to obtain the coordinates of the first center point of the pixel corresponding to the first response peak in the pixel coordinate system; the pixel corresponding to the first response peak corresponds to the center point of the first tracking frame; obtain the coordinates of the second center point of the pixel corresponding to the second response peak in the pixel coordinate system; the pixel corresponding to the second response peak corresponds to the center point of the second tracking frame; determine the line connecting the first center point coordinates and the second center point coordinates in the pixel coordinate system, and obtain the angle between the line and the horizontal axis, wherein the angle is greater than or equal to 0 degrees and less than 90 degrees; and obtain the motion information of the target object based on the first center point coordinates, the second center point coordinates and the angle.

[0136] In some optional embodiments of the present invention, the apparatus further includes:

[0137] The second acquisition module is used to acquire second image data collected by the UAV in a hovering state, and to re-detect the target object in the second image data;

[0138] The second determining module is used to obtain the object detection box in the third image in the second image data, and to track the target object based on the object detection box.

[0139] In some optional embodiments of the present invention, the first acquisition module 201 is used to obtain a first region in the first image where the target object is located, and a first center position of the first region; determine a second center position in the second image corresponding to the first center position, and determine the second region based on the second center position.

[0140] In some optional embodiments of the present invention, the first determining module 203 is used to determine the first difference coefficient based on the difference between the first response peak and the second response peak and a change coefficient; wherein the value of the change coefficient is related to the comparison result of the first response peak and the second threshold and the comparison result of the third response peak and the second threshold, the third response peak being the maximum value among the response values ​​of each pixel in the region associated with the target object in the fourth image; the fourth image being the frame following the first image.

[0141] In some optional embodiments of the present invention, the first determining module 203 is configured to: determine the area of ​​the first tracking frame based on the first tracking frame information; determine the area of ​​the second tracking frame based on the second tracking frame information; and determine the second difference coefficient based on the ratio between the area of ​​the first tracking frame and the area of ​​the second tracking frame.

[0142] In this embodiment of the invention, the device can be applied to an electronic device. The first acquisition module 201, tracking module 202, first determination module 203, and control module 204 in the device can all be implemented by a central processing unit (CPU), a digital signal processor (DSP), a microcontroller unit (MCU), or a field-programmable gate array (FPGA) in practical applications.

[0143] This invention also provides an electronic device. Figure 5 This is a schematic diagram of the hardware composition structure of an electronic device according to an embodiment of the present invention. For example... Figure 5 As shown, the electronic device 400 includes a processor 401 and a memory 402 for storing a computer program that can run on the processor 401, wherein the processor 401 executes the steps of the method described in the embodiments of the present invention when running the computer program.

[0144] Optionally, the electronic device 400 may also include at least one network interface 403. Various components in the electronic device 400 are coupled together via a bus system 404. It is understood that the bus system 404 is used to implement communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 The general designated all buses as Bus System 404.

[0145] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 402 described in this embodiment of the invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0146] The memory 402 in this embodiment of the invention is used to store various types of data to support the operation of the electronic device 400.

[0147] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of the present invention can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 402. Processor 401 reads the information in memory 402 and combines its hardware to complete the steps of the aforementioned method.

[0148] In an exemplary embodiment, the electronic device 400 may be implemented by one or more application-specific integrated circuits (ASICs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned method.

[0149] This invention also provides a computer-readable storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements the steps of the UAV target tracking method of this invention.

[0150] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0151] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0152] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0154] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0155] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0156] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0157] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0158] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for tracking targets using an unmanned aerial vehicle (UAV), characterized in that, The method includes: Acquire first image data collected by the drone, the first image data including a first image and a second image; the second image is a frame following the first image. Determine the first region where the target object is located in the first image and the second region associated with the target object in the second image; perform target tracking processing on the pixels in the first region and the second region to obtain the first response peak value in the response value corresponding to each pixel in the first region and the second response peak value in the response value corresponding to each pixel in the second region, and obtain the first tracking box information of the target object in the first region and the second tracking box information of the target object in the second region. A first difference coefficient is determined between the first response peak and the second response peak, and a second difference coefficient is determined between the first tracking box information and the second tracking box information; the determination of the first difference coefficient between the first response peak and the second response peak includes: determining the first difference coefficient based on the difference between the first response peak and the second response peak and a change coefficient; wherein, the value of the change coefficient is related to the comparison result of the first response peak and the second threshold and the comparison result of the third response peak and the second threshold, the third response peak being the maximum value among the response values ​​of each pixel in the region associated with the target object in the fourth image, and the fourth image being the frame image following the first image; The degree of occlusion of the target object is determined based on the first difference coefficient and the second difference coefficient. Control commands are sent to the drone according to the degree of occlusion of the target object. The control commands are used to adjust the flight state of the drone.

2. The method according to claim 1, characterized in that, Sending control commands to the drone based on the degree of occlusion of the target object includes: When the degree of occlusion of the target object is less than a first threshold, a first control command is sent to the drone. The first control command is used to instruct the drone to adjust its flight direction so that the drone can track the target object. When the target object is occluded to a degree greater than or equal to the first threshold, a second control command is sent to the drone. The second control command is used to instruct the drone to maintain a hovering state so that the drone can continuously acquire image data in the hovering state.

3. The method according to claim 2, characterized in that, The first control command further includes target object motion information; the method further includes: Obtain the coordinates of the first center point of the pixel corresponding to the first response peak in the pixel coordinate system; the pixel corresponding to the first response peak corresponds to the center point of the first tracking box; Obtain the coordinates of the second center point of the pixel corresponding to the second response peak in the pixel coordinate system; the pixel corresponding to the second response peak corresponds to the center point of the second tracking box; In the pixel coordinate system, determine the line connecting the coordinates of the first center point and the coordinates of the second center point, and obtain the angle between the line and the horizontal axis. The value of the angle is greater than or equal to 0 degrees and less than 90 degrees. Based on the coordinates of the first center point, the coordinates of the second center point, and the included angle, the motion information of the target object is obtained.

4. The method according to claim 2, characterized in that, The method further includes: Obtain the second image data collected by the drone in a hovering state, and re-detect the target object in the second image data; Obtain the object detection box in the third image from the second image data, and track the target object based on the object detection box.

5. The method according to claim 1, characterized in that, Determining the first region where the target object in the first image is located and the second region associated with the target object in the second image includes: Obtain the first region where the target object is located in the first image, and the first center position of the first region; Determine the second center position in the second image that corresponds to the first center position, and determine the second region based on the second center position.

6. The method according to claim 1, characterized in that, Determining the second difference coefficient between the first tracking box information and the second tracking box information includes: The area of ​​the first tracking frame is determined based on the first tracking frame information, and the area of ​​the second tracking frame is determined based on the second tracking frame information; The second difference coefficient is determined based on the ratio between the area of ​​the first tracking frame and the area of ​​the second tracking frame.

7. A drone target tracking device, characterized in that, The device includes: The first acquisition module is used to acquire first image data collected by the UAV, the first image data including a first image and a second image; the second image is a frame image after the first image; and to determine a first region where the target object in the first image is located and a second region associated with the target object in the second image. The tracking module is used to perform target tracking processing on the pixels of the first region and the second region, respectively obtaining the first response peak value in the response value corresponding to each pixel in the first region and the second response peak value in the response value corresponding to each pixel in the second region, and respectively obtaining the first tracking box information of the target object in the first region and the second tracking box information of the target object in the second region; The first determining module is used to determine a first difference coefficient between the first response peak and the second response peak, and to determine a second difference coefficient between the first tracking box information and the second tracking box information; The first determining module is specifically used to determine the first difference coefficient based on the difference between the first response peak and the second response peak and the change coefficient; wherein, the value of the change coefficient is related to the comparison result of the first response peak and the second threshold and the comparison result of the third response peak and the second threshold, the third response peak is the maximum value of the response values ​​of each pixel in the region associated with the target object in the fourth image, and the fourth image is the frame image after the first image; The control module is used to determine the degree of occlusion of the target object based on the first difference coefficient and the second difference coefficient, and to send control commands to the drone according to the degree of occlusion of the target object. The control commands are used to adjust the flight state of the drone.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 6.

Images