Unmanned aerial vehicle target tracking method and system based on hybrid algorithm

Abstract

The invention provides an unmanned aerial vehicle target tracking method and system based on a hybrid algorithm, and the method comprises the steps: carrying out the ID number matching of an unmanned aerial vehicle target based on a byterack algorithm of Yolov5 when a to-be-tracked unmanned aerial vehicle in a current frame image can be recognized through a Yolov5 algorithm, and continuously tracking the to-be-tracked unmanned aerial vehicle target; when the Yolov5 algorithm cannot identify the to-be-tracked unmanned aerial vehicle in the current frame image, the byterack algorithm is switched to the kcf algorithm, and the unmanned aerial vehicle target is predicted and tracked based on the kcf algorithm; when the kcf algorithm is switched to the Byterack algorithm, the target motion track information accumulated by the kcf algorithm is used as initialization data to be fused into the Byterack algorithm, so that the initialization time and errors of the Byterack algorithm after switching are reduced, and the target tracking reliability of the unmanned aerial vehicle is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unmanned aerial vehicle detection, and in particular to an unmanned aerial vehicle target tracking method and system based on a hybrid algorithm. BACKGROUND

[0002] With the rapid development of unmanned aerial vehicle technology, unmanned aerial vehicles are increasingly widely used in civilian and other fields. Unmanned aerial vehicles, with their flexible maneuverability and strong flight performance, have unique advantages and potential in reconnaissance, security monitoring, logistics distribution, agricultural inspection, and film shooting in many scenarios.

[0003] In the field of reconnaissance, traditional reconnaissance methods often have certain limitations when facing complex and variable battlefield environments. For example, although satellite reconnaissance methods have a wide coverage, their resolution and real-time performance cannot meet the precise reconnaissance needs of small targets in a specific area.

[0004] The currently widely used target tracking technologies mainly include deep learning algorithms and traditional tracking algorithms, each of which has certain limitations.

[0005] Deep learning-based tracking algorithms, such as YOLO (You Only Look Once) + bytrach, are trained based on large-scale data sets, learn rich target features, and have high detection accuracy. However, this method has limited application scenarios. When the target cannot be recognized by the trained model, such as a very regular person or vehicle, the problem of initializing tracking may occur. When the target is occluded, the target may be lost.

[0006] Traditional tracking algorithms, such as Kalman Filter and kcf (Kernel Correlation Filter), can predict and track targets to some extent, but they have weak target feature extraction and recognition capabilities in complex scenarios, and their performance is not good in multi-target tracking scenarios.

[0007] To solve the problem of low reliability of unmanned aerial vehicle target tracking caused by single type algorithms in the above prior art, the present application provides an unmanned aerial vehicle target tracking method and system based on a hybrid algorithm to solve the above problems. SUMMARY

[0008] The unmanned aerial vehicle target tracking method and system based on the hybrid algorithm can combine the bytetrack algorithm based on Yolo and the traditional tracking algorithm kcf, give full play to the advantages of both, make up for the defects of each other, realize more reliable and stable unmanned aerial vehicle target tracking, and meet the following requirements of the unmanned aerial vehicle in complex application scenarios: following and striking.

[0009] The first aspect of the application provides an unmanned aerial vehicle target tracking method based on a hybrid algorithm, comprising: detecting and identifying each frame of image including a to-be-tracked unmanned aerial vehicle based on a Yolov5 algorithm; obtaining an identification result of each frame of image including the to-be-tracked unmanned aerial vehicle; when the Yolov5 algorithm can identify the to-be-tracked unmanned aerial vehicle in the current frame of image, performing ID number matching on the unmanned aerial vehicle target based on the bytetrack algorithm of Yolov5, continuously tracking the to-be-tracked unmanned aerial vehicle target based on the ID of the unmanned aerial vehicle target, and outputting the position of the to-be-tracked unmanned aerial vehicle target; when the Yolov5 algorithm cannot identify the to-be-tracked unmanned aerial vehicle in the current frame of image, switching from the bytetrack algorithm to the kcf algorithm, performing prediction tracking on the unmanned aerial vehicle target based on the kcf algorithm, and outputting the position of the to-be-tracked unmanned aerial vehicle target; when the kcf algorithm switches to the bytetrack algorithm, the target motion trajectory information accumulated by the kcf algorithm is fused into the Bytetrack algorithm as initialization data, so as to reduce the initialization time and error of the Bytetrack algorithm after switching.

[0010] Optionally, the detection and identification of the to-be-tracked unmanned aerial vehicle based on the Yolov5 algorithm specifically comprises: generating a detection parameter according to the current illumination condition and the background feature complexity, and detecting and identifying the to-be-tracked unmanned aerial vehicle based on the generated detection parameter through the Yolov5 algorithm, wherein the detection parameter comprises an anchor box scale adjustment factor and a classification threshold.

[0011] Further, when the detection parameter is the anchor box scale adjustment factor, generating the detection parameter according to the current illumination condition and the background feature complexity specifically comprises: ; wherein, the anchor box scale adjustment factor is, the maximum value of the image brightness, the standard deviation of the image brightness; the background feature complexity index is, the first weight coefficient of the image brightness is, This is the first weighting coefficient for the complexity of background features.

[0012] Optionally, when the detection parameter is a classification threshold, the specific methods for generating the detection parameter based on the current lighting conditions and background feature complexity include: ; in, For classification threshold, It is the initial classification threshold. It is the average brightness value of the image. This is the second weighting coefficient for image brightness. This is the second weighting coefficient for the complexity of the background features.

[0013] Optionally, when the YOLOv5 algorithm can identify the drone to be tracked in the current frame image, the corresponding lighting conditions and the occlusion state of the drone to be tracked both meet the preset recognition conditions; when the YOLOv5 algorithm cannot identify the drone to be tracked in the current frame image, the corresponding lighting conditions or the occlusion state of the drone to be tracked do not meet the preset recognition conditions; wherein, the preset recognition conditions include lighting recognition conditions and occlusion state recognition conditions, wherein, the lighting recognition condition is that the brightness difference between the current frame image and the previous frame image is greater than a preset brightness threshold, or the color distribution difference between the current frame image and the previous frame image is greater than a preset color distribution difference threshold; the occlusion state recognition condition is that the overlap rate of the detection boxes is lower than a preset overlap rate threshold within a consecutive first preset number of cycles, and the number of detected target feature points of the drone to be tracked is less than a preset feature point number threshold.

[0014] Furthermore, the specific method for calculating the color distribution difference between the current frame image and the previous frame image is as follows: ; Where D is the color distribution difference between the current frame image and the previous frame image. This represents the first color histogram of the current frame image and the previous frame image. One bin index, This represents the total number of bins in the color histogram. This is the index of the i-th bin in the color histogram of the current frame image. This is the i-th bin index of the color histogram of the previous frame image. Each bin index corresponds to the frequency or normalized probability value of a color range.

[0015] Optionally, the method for calculating the overlap rate of the detection boxes is as follows: ; in, The overlap rate of the detection boxes that include the drone target to be tracked. and These represent the detection boxes of the drone target to be tracked in two adjacent frames. This represents the intersection area of ​​the detection boxes of the drone target to be tracked in two adjacent frames. This represents the area of ​​the union of the detection boxes of the drone target to be tracked in two adjacent frames.

[0016] Optionally, when switching from the kcf algorithm to the bytetrack algorithm, the corresponding lighting conditions and the occlusion state of the UAV to be tracked both meet the preset switching conditions; wherein, the preset switching conditions include that the number of consecutive second preset number of image frames is greater than the classification threshold, and the corresponding lighting conditions and the occlusion state of the UAV to be tracked both meet the preset recognition conditions.

[0017] Furthermore, the target motion trajectory information accumulated by the kcf algorithm is incorporated into the Bytetrack algorithm as initialization data, specifically as follows: For the position coordinate data in the target motion trajectory information, coordinate transformation is performed according to the image coordinate system of the Bytetrack algorithm; For the velocity information in the target motion trajectory information, the horizontal velocity component and the vertical velocity component are normalized respectively; For the motion direction information in the target motion trajectory information, the motion direction information is converted into angle values, and the converted angle values ​​are normalized. The position coordinate data after coordinate transformation of the third most recent preset number of frames of the KCF algorithm, the normalized horizontal velocity component, vertical velocity component, and motion direction angle value are used as initial data and incorporated into the state prediction model of the Bytetrack algorithm.

[0018] A second aspect of the present invention provides a UAV target tracking system based on a hybrid algorithm, comprising: The detection and recognition module uses the Yolov5 algorithm to detect and recognize each frame of the image, including the drone to be tracked. The acquisition module acquires the recognition results of each frame of image, including the drone to be tracked. The first output module, when the YOLOv5 algorithm can identify the drone to be tracked in the current frame image, performs ID number matching on the drone target based on the YOLOv5 bytetrack algorithm, continuously tracks the drone target to be tracked based on the drone target ID, and outputs the position of the drone target to be tracked. The second output module switches from the bytetrack algorithm to the kcf algorithm when the Yolov5 algorithm cannot identify the drone to be tracked in the current frame image. Based on the kcf algorithm, the drone target is predicted and tracked, and the position of the drone target to be tracked is output. The switching module fuses target motion trajectory information accumulated by the KCF algorithm into the Bytetrack algorithm as initialization data when the KCF algorithm switches to the Bytetrack algorithm, so as to reduce the initialization time and error of the Bytetrack algorithm after switching.

[0019] The technical scheme adopted by the present application includes the following technical effects: 1、When the Yolov5 algorithm can recognize the unmanned aerial vehicle to be tracked in the current frame image, the ID number matching of the unmanned aerial vehicle target is performed based on the Bytetrack algorithm of Yolov5, and the unmanned aerial vehicle target to be tracked is continuously tracked; when the Yolov5 algorithm cannot recognize the unmanned aerial vehicle to be tracked in the current frame image, the Bytetrack algorithm is switched to the KCF algorithm, and the unmanned aerial vehicle target is tracked based on the KCF algorithm; when the KCF algorithm is switched to the Bytetrack algorithm, the target motion trajectory information accumulated by the KCF algorithm is fused into the Bytetrack algorithm as initialization data, so as to reduce the initialization time and error of the Bytetrack algorithm after switching; the Bytetrack algorithm of Yolo is combined with the traditional tracking algorithm KCF, the advantages of both are fully utilized, and the defects of each are made up, so that the problem of low reliability of unmanned aerial vehicle target tracking in the prior art is solved, and the reliability of unmanned aerial vehicle target tracking is improved.

[0020] 2、In the technical scheme of the present application, detection parameters are generated according to the current illumination condition and the complexity of the background features, and the unmanned aerial vehicle to be tracked is detected and recognized by the Yolov5 algorithm based on the generated detection parameters, wherein the detection parameters include an anchor frame scale adjustment factor and a classification threshold, so that the Yolov5 algorithm adjusts the recognition parameters of the algorithm according to the change of the environment, and the reliability of unmanned aerial vehicle target detection and recognition is improved.

[0021] 3、In the technical scheme of the present application, when the KCF algorithm is switched to the Bytetrack algorithm, the corresponding illumination condition and the occlusion state of the unmanned aerial vehicle to be tracked both meet the preset switching condition; wherein the preset switching condition includes that the consecutive second preset number of image frames is greater than the classification threshold, and the corresponding illumination condition and the occlusion state of the unmanned aerial vehicle to be tracked both meet the preset recognition condition, so that the switching of the KCF algorithm and the Bytetrack algorithm has accurate judgment and evaluation conditions, and the reliability of unmanned aerial vehicle target detection and recognition is ensured.

[0022] 4、The technical scheme of the present application integrates the position coordinate data of the third preset number of frames of images converted by the kcf algorithm, the normalized horizontal direction velocity component, the vertical direction velocity component and the motion direction angle value into the state prediction model of the Bytetrack algorithm as initialization data, so that the Bytetrack algorithm can utilize the target motion trajectory information accumulated by the kcf algorithm to more accurately predict the position and motion state of the target during initialization, thereby reducing the initialization time and error and improving the robustness and accuracy of tracking.

[0023] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present application. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0025] Figure 1 A flowchart of the method of embodiment one in the present application scheme; Figure 2 Another flowchart of the method of embodiment one in the present application scheme; Figure 3 A timing diagram of the tracking algorithm in the method of embodiment one in the present application scheme; Figure 4 A flowchart of the flight tracking protocol in the method of embodiment one in the present application scheme; Figure 5 A flight tracking architecture diagram in the method of embodiment one in the present application scheme; Figure 6 A flight tracking information flow diagram in the method of embodiment one in the present application scheme; Figure 7 A structure diagram of the system of embodiment two in the present application scheme. DETAILED DESCRIPTION

[0026] For purposes of the present disclosure, like reference numbers in the drawings shall indicate analogous elements. The present disclosure provides a number of different embodiments or examples, which utilize different structures for implementing the present disclosure. In order to simplify the present disclosure, the description below describes certain examples with reference to particular components and configurations. Moreover, the present disclosure can refer to a number of different examples or embodiments. When certain details of these examples are described, it should be appreciated that not all examples can be comprised of all of the described components or use all of the described configurations. In addition, the present disclosure can use reference numbers and / or letters in different examples. Such repetition is for the sake of simplicity and clarity and does not indicate a relationship between the different embodiments and / or configurations discussed. It should be noted that the components illustrated in the figures are not necessarily drawn to scale. The present disclosure omits description of well-known components and processing techniques and processes to avoid unnecessarily obscuring the present disclosure.

[0027] Embodiment one Machine vision-based tracking algorithms are generally divided into two categories: target recognition tracking based on deep learning (such as Bytetrack) and traditional tracking algorithms based on filter analysis (KCF, MOSSE). The recognition tracking algorithm is very dependent on the detection and recognition of the AI inference model on the picture target, and different training models have very large recognition differences. And the traditional tracking algorithm has too high a requirement for hardware performance, and it is difficult to solve the problem of multi-target tracking. The unmanned aerial vehicle tracking method in the present application combines the advantages of the two algorithms, and ensures that the unmanned aerial vehicle target tracking is stable and consistent in various scenes and various task types.

[0028] As shown in Figures 1-3 The present application provides an unmanned aerial vehicle target tracking method based on a hybrid algorithm, comprising: S1, detecting and recognizing each frame of image including a to-be-tracked unmanned aerial vehicle based on a Yolov5 algorithm; S2, obtaining the recognition result of each frame of image including the to-be-tracked unmanned aerial vehicle; S3, judging whether the Yolov5 algorithm can recognize the to-be-tracked unmanned aerial vehicle in the current frame of image, if the judgment result is yes, executing step S4, if the judgment result is no, executing step S5; S4, performing ID number matching on the unmanned aerial vehicle target based on the Bytetrack algorithm of Yolov5, continuously tracking the to-be-tracked unmanned aerial vehicle target based on the ID of the unmanned aerial vehicle target, and outputting the position of the to-be-tracked unmanned aerial vehicle target; S5, switching from the Bytetrack algorithm to the Kcf algorithm, and performing prediction tracking on the unmanned aerial vehicle target based on the Kcf algorithm, and outputting the position of the to-be-tracked unmanned aerial vehicle target; S6, when the Kcf algorithm switches to the Bytetrack algorithm, the target motion trajectory information accumulated by the Kcf algorithm is fused into the Bytetrack algorithm as initialization data, so as to reduce the initialization time and error of the Bytetrack algorithm after switching.

[0029] In step S1, the detection and identification of the UAV to be tracked based on the Yolov5 algorithm specifically includes: Detection parameters are generated based on the current lighting conditions and background feature complexity. The UAV to be tracked is then detected and identified using the Yolov5 algorithm based on the generated detection parameters. The detection parameters include the anchor frame scale adjustment factor and the classification threshold.

[0030] Image acquisition: The drone is equipped with a high-definition camera and sensors, with a frame rate of no less than 30fps and a resolution of no less than 1080P. Sensor data acquisition: Equipped with an inertial measurement unit (IMU) and a BeiDou positioning system, the drone's acceleration and angular velocity are detected in real time, and the BeiDou positioning system provides accurate position information.

[0031] First, in the algorithm initialization phase, parameters are set and models are loaded for the Bytetrack algorithm, YOLOv5 model inference, and kcf tracking algorithm. The YOLOv5 model is trained with image data from high, medium, and low altitudes to ensure that the model can accurately associate with target objects under different viewpoints and scenes.

[0032] For the Yolov5 model, detection parameters are generated based on the current lighting conditions and background feature complexity, enabling the algorithm to accurately extract target features. This is done after obtaining the average brightness value under the lighting conditions. luminance standard deviation and background feature complexity index Then, the following specific calculation process can be used to generate suitable detection parameters to ensure that the algorithm can accurately detect targets in complex and ever-changing environments. Detection parameters may include anchor frame scale adjustment factors. and classification threshold , Adjusting the anchor box size in the YOLOv5 model to better adapt to object detection under different lighting and background conditions is crucial. Classification threshold Accurately classifying targets is crucial for the YOLOv5 model. To avoid misclassification due to lighting or background interference under different lighting and background conditions, the classification threshold needs to be adjusted appropriately.

[0033] When the detection parameter is an anchor frame scale adjustment factor, the specific methods for generating the detection parameter based on the current lighting conditions and background feature complexity include: ; in, This is the anchor frame size adjustment factor. This represents the maximum value of the image brightness; in common image systems, its value is typically set to 255. is the standard deviation of image brightness, which reflects the uniformity of illumination; is the background feature complexity index, is the first weight coefficient of image brightness, is the first weight coefficient of background feature complexity, and The value of directly affects the amplitude of anchor frame scale adjustment.

[0034] Wherein, the greater the value of the background feature complexity index B, the more complex the background, which can be classified according to the following standards: Low complexity (B = 0.3): uniform texture, less dynamic interference (such as farmland); Medium complexity (B = 0.6): there are some feature confusion or dynamic interference (such as suburban roads; High complexity (B = 0.9): multi-semantic interference, severe dynamic change (such as city center); a background complexity scoring standard can also be constructed or defined, and the current frame image is scored according to the scoring standard (0-1), and the score is taken as the background feature complexity index; it can also be other self-defined evaluation standards, which are not limited by the present application.

[0035] Wherein, when the detection parameter is a classification threshold, the detection parameter is generated according to the current illumination condition and the background feature complexity, and specifically includes: ; Wherein, is the classification threshold, is the initial classification threshold, is the average brightness value of the image, which reflects the overall illumination intensity of the image, is the second weight coefficient of image brightness (represented by image brightness standard deviation), used to measure the uniformity of the current picture illumination, The greater, the more uneven the illumination, the easier the detection is affected by noise / speckle, and the anchor frame needs to be adapted to the amplitude and the classification threshold to suppress false detection; is the second weight coefficient of background feature complexity (which can be represented by background feature complexity standard deviation or image average brightness), which plays a linear or sub-linear adjustment role in the self-adaptation of anchor frame scale and classification threshold. The background feature complexity standard deviation acquisition process can be: using the first frame image to construct an initial background feature model, which can be modeled at the pixel level; compare the real-time frame (current frame) with the background feature model pixel by pixel, calculate the difference value (such as gray change, histogram difference) of each pixel; take the square root of the difference value (i.e. standard deviation), get the overall background complexity value. The greater the value, the more dramatic the background change, the more obvious the noise interference.

[0036] In steps S2-S4, when the Yolov5 algorithm can identify the unmanned aerial vehicle to be tracked in the current frame image, the ID number of the unmanned aerial vehicle target is matched based on the bytetrack algorithm of Yolov5, the unmanned aerial vehicle target is continuously tracked based on the ID of the unmanned aerial vehicle target, and the position of the unmanned aerial vehicle target to be tracked is output. The data acquisition module in the unmanned aerial vehicle will continuously acquire images during the entire flight process, and send the images to the image processing module through the internal bus. The image processing module uses the Yolov5 algorithm to perform real-time target detection on the images taken by the unmanned aerial vehicle, identifies personnel, vehicles and other trained feature targets from the images, and presents the target detection frame in the real-time picture of the ground station software for the operator to quickly select the target. The detection result is continuously integrated into the Bytetrack algorithm in the system process, and Bytetrack numbers (ID) each detected unmanned aerial vehicle target through the Hungarian algorithm. After the operator selects the tracking target, the target ID will be used as the target ID for single continuous tracking by the Bytetrack algorithm.

[0037] When the operator needs the unmanned aerial vehicle to track a target in the picture, the tracking target is selected through the ground station software, and a tracking instruction is sent. After the data acquisition module receives the instruction to start tracking, it checks whether the target is identifiable, i.e., uses the Yolov5 model to detect the selected target area. When the detection result confidence is greater than 0.2, it is considered to be an identifiable target.

[0038] Bytetrack target tracking based on Yolov5: a multi-target tracking algorithm based on target detection bbox, which uses a Yolov5 target detection model trained on a large number of image data containing various targets (personnel, vehicles or other characteristic objects). Model pruning and quantization techniques are used to traverse the weight parameters in the Yolov5 network, set thresholds layer by layer , remove the neural network parameters that have less impact on the model performance (i.e., lower than the weight threshold The neuron and connection of the Yolov5 algorithm are reduced by a large margin, the quantity of model parameters and the quantity of calculation are reduced, and the quantization technology is used to convert the model weight and the activation value from the floating point operation to the integer operation, thereby further reducing the storage and calculation requirement of the model. The model can be run on the Riscv architecture domestic AI chip, and the KPU (Knowledge Process Unit) of the chip is used for AI inference acceleration. The Bytetrack algorithm constructs the motion trajectory of the target by associating the detection results between the front and rear image frames, and uses the Hungarian algorithm to accurately match the to-be-tracked unmanned aerial vehicle targets in different frames, so as to realize the continuous tracking of the to-be-tracked unmanned aerial vehicle target, and can memorize and associate the features of the to-be-tracked unmanned aerial vehicle target, distinguish the trajectories of multiple to-be-tracked unmanned aerial vehicle targets in the picture, and avoid trajectory confusion and loss. The Bytetrack algorithm based on the Yolov5 can realize the ID matching of multiple to-be-tracked unmanned aerial vehicle targets, and according to the accuracy and generalization ability of the Yolov5 in target detection, the to-be-tracked unmanned aerial vehicle target can be stably tracked in a clear and identifiable state.

[0039] In step S5, the algorithm is switched to the kcf algorithm by the Bytetrack algorithm, and the position of the to-be-tracked unmanned aerial vehicle target is output based on the kcf algorithm. The kcf (Kernel Correlation Filter) algorithm is based on kernel correlation filtering and has high efficiency in target position prediction. The possible position of the target in the current frame can be calculated according to the position of the target in the previous frame and the extracted target features. The kcf algorithm converts the tracking problem into a classification problem, constructs a filter, and calculates the filter coefficients in the frequency domain to maximize the similarity between the output response of the filter and the target. Fourier transform is used in the calculation process to convert the calculation from the spatial domain to the frequency domain, thereby greatly improving the calculation efficiency.

[0040] Preferably, when the Yolov5 algorithm can identify the to-be-tracked unmanned aerial vehicle in the current frame image, the corresponding illumination condition and the occlusion state of the to-be-tracked unmanned aerial vehicle both meet the preset identification condition; when the Yolov5 algorithm cannot identify the to-be-tracked unmanned aerial vehicle in the current frame image, the corresponding illumination condition or the occlusion state of the to-be-tracked unmanned aerial vehicle does not meet the preset identification condition; wherein the preset identification condition includes an illumination identification condition and an occlusion state identification condition, wherein the illumination identification condition is that the brightness difference between the current frame image and the previous frame image is greater than a preset brightness threshold, or the color distribution difference between the current frame image and the previous frame image is greater than a preset color distribution difference threshold; the occlusion state identification condition is that the overlap rate of the detection box in the continuous first preset number of periods is less than a preset overlap rate threshold, and the number of detected to-be-tracked unmanned aerial vehicle target feature points is less than a preset feature point number threshold. When the UAV target to be tracked is severely occluded or the light condition changes dramatically, detection failure may occur. In this case, the system uses the kcf tracker to make a short-term prediction of the target position, ensuring the continuity of tracking.

[0041] The overlap rate of the detection frame is calculated as follows: ; Wherein, is the overlap rate of the detection frame including the UAV target to be tracked, and represent the detection frame of the UAV target to be tracked in the adjacent two frames of images, represents the intersection area of the detection frame of the UAV target to be tracked in the two adjacent frames of images, represents the union area of the detection frame of the UAV target to be tracked in the adjacent two frames of images.

[0042] Through a large number of experimental tests, the threshold value of the target detection frame overlap rate is set to 0.3. If the overlap rate of the detection frame is less than 0.3 for 5 consecutive periods, and the number of detected target feature points is reduced by and the above (preset feature point number threshold) compared with the normal state, it is determined that the target is in a severely occluded state.

[0043] Light change determination: For the dynamic change of light condition, real-time analysis of image brightness histogram and color distribution characteristics is used for monitoring. By calculating the difference value of the current frame image and the previous frame image in brightness and color distribution, it is judged whether the light condition changes dramatically. For the calculation of the brightness difference value, the brightness value of the current frame image is , the brightness value of the previous frame image is , and the calculation formula of the brightness difference value is:

[0044] The preset dynamic change threshold of the brightness difference value is set to (the brightness value range is ).

[0045] The color distribution difference value between the current frame image and the previous frame image is calculated as follows: ; Wherein, D is the color distribution difference value between the current frame image and the previous frame image, represents the th bin index of the color histogram corresponding to the current frame image and the previous frame image, , is the total number of bins of the color histogram, is the i-th bin index of the color histogram of the current frame image, is the i-th bin index of the color histogram of the previous frame image, each bin index corresponds to a color range frequency or normalized probability value.

[0046] For the calculation of the color distribution difference value, the Bhattacharyya distance of the color histogram is calculated.

[0047] confidence is determined by the classification threshold and the confidence distribution of the model output; The self-adaptation of is adjusted by the standard deviation of image brightness. When the light changes suddenly or the background is complex ( >20 or D>0.2, or increases), the value of is increased to reduce false positives, resulting in effective samples passing through the threshold, and the detection enters a conservative mode.

[0048] In step S6, when the kcf algorithm is switched to the bytetrack algorithm, the corresponding light condition and the occlusion state of the unmanned aerial vehicle to be tracked both meet the preset switching condition; wherein, the preset switching condition includes that the classification threshold is greater than the second preset number of consecutive image frames, and the corresponding light condition and the occlusion state of the unmanned aerial vehicle to be tracked both meet the preset identification condition.

[0049] Switching rules: 1) Single trigger pre-switching: any serious condition (light condition and occlusion state condition of the unmanned aerial vehicle to be tracked) meets the preset switching condition, that is, enters the "unreliable detection state", executes the kcf algorithm, and the short-term prediction maintains the trajectory continuity (for example, the light changes suddenly when ΔL>20 or D>0.2, or the occlusion criterion IoU is less than 0.3 for 5 consecutive frames and the feature point reduction is greater than or equal to 70%).

[0050] 2) Simultaneous or continuous satisfaction of stable switching: when the light changes suddenly and the occlusion occurs at the same time, or the single serious condition continues to exist for multiple cycles, the prediction mode dominated by the kcf algorithm will be maintained, and the detection parameters will be set to be conservative (increase the classification threshold , expand the anchor frame size S, and reduce the NMS threshold), until the criterion is restored.

[0051] 3) Back switching condition (consistent with the state machine in the foregoing): the classification value of more than 3 consecutive image frames is greater than the classification threshold When the current detection frame is consistent with the kcf predicted frame (i.e. IoU ≥ 0.5, center deviation ≤ 10 pixels), and the light / shading criterion returns to the threshold (ΔL ≤ 20 and D ≤ 0.2, or IoU ≥ 0.5), it is switched back to the ByteTrack algorithm as the main algorithm, and the kcf algorithm is used to initialize the last 10 frames of trajectory to avoid ID drift.

[0052] Specifically, the target motion trajectory information accumulated by the kcf algorithm is integrated as initialization data into the ByteTrack algorithm, specifically: For the position coordinate data in the target motion trajectory information, coordinate conversion is performed according to the image coordinate system of the ByteTrack algorithm; For the speed information in the target motion trajectory information, the horizontal direction speed component and the vertical direction speed component are normalized respectively; For the motion direction information in the target motion trajectory information, the motion direction information is converted into an angle value, and the converted angle value is normalized; The coordinate-converted position coordinate data of the last third preset number of frames of images of the kcf algorithm, the normalized horizontal direction speed component, the vertical direction speed component, and the motion direction angle value are integrated as initialization data into the state prediction model of the ByteTrack algorithm.

[0053] When switching from the kcf algorithm to the ByteTrack algorithm, the system integrates the target motion trajectory information accumulated by the kcf algorithm as initialization data into the ByteTrack algorithm. Through data fusion, the information accumulated by the kcf algorithm during target tracking can be fully utilized, the initialization time and error of the ByteTrack algorithm after switching can be reduced, and the robustness and accuracy of tracking can be improved.

[0054] The extracted target motion trajectory information needs to be preprocessed to meet the input format requirements of the ByteTrack algorithm. For position coordinate data, the system will perform coordinate conversion according to the image coordinate system used by the ByteTrack algorithm to ensure data consistency. For speed information and motion direction information, the system will normalize them to a specific value range to facilitate fusion with other parameters in the ByteTrack algorithm. Specifically, for speed information, the following normalization formula is used:

[0055] wherein, and are the normalized horizontal and vertical direction speed components, and are the original speed components, , , , are the minimum and maximum values of the velocity component in the scene, respectively.

[0056] For the motion direction information, it is converted into an angle value and normalized to the range of .

[0057] After preprocessing, the target motion trajectory information will be integrated into the initialization process of the Bytetrack algorithm. The Bytetrack algorithm will establish a tracking model of the target according to the input target detection box and related feature information during initialization. In this invention, the target position information tracked by the kcf algorithm will be used as the update input of the Bytetrack algorithm until the system switches to the Bytetrack tracking algorithm, so as to integrate the results of the kcf algorithm into the state prediction model of the Bytetrack algorithm. In this way, the Bytetrack algorithm can utilize the target motion trajectory information accumulated by the kcf algorithm during initialization to more accurately predict the position and motion state of the target, thereby reducing the initialization time and error and improving the robustness and accuracy of tracking.

[0058] When the occlusion criterion is met (multi-cycle IoU <0.3 and feature point drop), detection confidence decay coexists with time sequence instability; at this time, the system will enter the "low confidence interval" and switch to the kcf algorithm short-term prediction to maintain trajectory continuity, and the detection parameters remain conservative until the occlusion is removed.

[0059] Switching and switching back: when stabilizes under conservative settings (continuous frames above threshold, and kcf algorithm prediction consistency passes), switch back to Bytetrack as the main algorithm; at the same time, the recent trajectory of the kcf algorithm (the trajectory of the last 10 frames of images) is used as the initialization input to avoid ID drift and reconstruction error.

[0060] The threshold of confidence 0.2 is used as the initial threshold for deployment to ensure basic detection of small / weak targets; in engineering implementation, this value is dynamically adjusted through field data and the above-mentioned adaptive factor to achieve the optimal balance of "detection rate-mis-detection rate-time sequence stability".

[0061] Based on the above strategy, if the target can be identified, the target ID is extracted to track the target alone, and the coordinate information of the target frame in the entire picture is output in real time. If the operator selects other targets that cannot be recognized by the yolov5 model or the identified target is lost due to factors such as reduced light intensity and shielding, the system will make an algorithm switching decision to use the kcf algorithm to continue the tracking task, and the tracking result will be used as the input of the bytetrack tracker to continue prediction during tracking. When the target is recognized again, the continuous tracking of the target is ensured.

[0062] As shown in Figure 4 , the results of the target tracking algorithm will be processed by the flight control unit to calculate the desired flight attitude and generate a standard crsf (crossfire radio system protocol) protocol for describing the attitude control of the unmanned aerial vehicle. The crsf protocol includes the pitch, yaw, and throttle size control information of the unmanned aerial vehicle. Finally, the PID algorithm and dshot600 protocol are used to control the motor system of the unmanned aerial vehicle.

[0063] As shown in Figure 5 , the camera carried by the unmanned aerial vehicle acquires environmental images, performs image acquisition, and then sends the acquired environmental images to the image processing unit. The image processing unit selects the unmanned aerial vehicle target to be tracked according to the ground station software, determines the corresponding tracking algorithm according to the Yolov5 algorithm recognition result or the preset recognition condition, continuously tracks the unmanned aerial vehicle target to be tracked, outputs the position of the unmanned aerial vehicle target to be tracked, and performs flight control on the unmanned aerial vehicle target to be tracked based on the position of the unmanned aerial vehicle target to be tracked. The ground station software runs on a computer, a tablet, or even a remote control device (the embodiment method of the present application can run on a terminal device where the ground station software is located, or can run on a remote control platform in communication with the terminal device where the ground station software is located). An intuitive interactive interface is provided for the operator. The operator can select the tracking target and the tracking mode on the software interface. The software sends control instructions to the unmanned aerial vehicle through a wireless communication module (Wi-Fi, 4G, 2.4GHz, etc.), and real-time tracking pictures are viewed. The ground station software stores task information by recording instruction issuing logs and recording the return pictures of the unmanned aerial vehicle, which facilitates the operator to view and analyze historical tracking tasks.

[0064] As shown in Figure 6As shown, the flight control includes following and striking, wherein the following: the unmanned aerial vehicle maintains the current flight height (h), and changes the flight direction and flight speed according to the target movement direction and the target frame size change. During this period, the pixel coordinates (x, y) of the target in the whole picture are obtained from the image processing module, and the current attitude (mainly the yaw angle w) of the unmanned aerial vehicle is obtained in real time. The desired attitude of the unmanned aerial vehicle is calculated by using the coordinate offset of the target in the picture, while maintaining the flight height (h). When the target moves away or approaches, the size change of the target frame is checked to control the flight direction of the unmanned aerial vehicle.

[0065] Striking: the unmanned aerial vehicle adjusts the flight height, so that the horizontal angle between the unmanned aerial vehicle and the target satisfies 70°-90°, when reaching the optimal impact angle, the fixed diving angle and flight speed are maintained to impact the target, and during the process, the target tracking is still continued through the image, and the aircraft attitude is fine-tuned to ensure accurate attack on the target.

[0066] When the Yolov5 algorithm can identify the unmanned aerial vehicle to be tracked in the current frame image, the ID number matching of the unmanned aerial vehicle target is carried out based on the bytetrack algorithm of Yolov5, and the unmanned aerial vehicle target to be tracked is continuously tracked; when the Yolov5 algorithm cannot identify the unmanned aerial vehicle to be tracked in the current frame image, the algorithm is switched from the bytetrack algorithm to the kcf algorithm, and the unmanned aerial vehicle target is tracked based on the kcf algorithm; when the kcf algorithm is switched to the bytetrack algorithm, the target motion trajectory information accumulated by the kcf algorithm is fused into the Bytetrack algorithm as initialization data, so as to reduce the initialization time and error of the Bytetrack algorithm after switching. By combining the bytetrack algorithm of Yolo with the traditional tracking algorithm kcf, the advantages of both are fully utilized, the defects of each are made up, the problem of low reliability of unmanned aerial vehicle target tracking caused by the prior art is solved, and the reliability of unmanned aerial vehicle target tracking is improved.

[0067] In the technical scheme of the application, detection parameters are generated according to the current illumination condition and the complexity of the background features, and the Yolov5 algorithm is used to detect and identify the unmanned aerial vehicle to be tracked based on the generated detection parameters, wherein the detection parameters include an anchor frame scale adjustment factor and a classification threshold, so that the Yolov5 algorithm adjusts the identification parameters of the algorithm according to the change of the environment, and improves the reliability of unmanned aerial vehicle target detection and identification.

[0068] In the technical scheme of the present application, when the kcf algorithm is switched to the bytetrack algorithm, the corresponding light conditions and the occlusion state of the unmanned aerial vehicle to be tracked both meet the preset switching conditions; wherein the preset switching conditions include that the consecutive second preset number of image frames is greater than the classification threshold, and the corresponding light conditions and the occlusion state of the unmanned aerial vehicle to be tracked both meet the preset identification conditions, so that the switching of the kcf algorithm and the bytetrack algorithm has accurate judgment and evaluation conditions, and the reliability of the unmanned aerial vehicle target detection and identification is ensured.

[0069] In the technical scheme of the present application, the position coordinate data of the coordinate conversion of the last third preset number of frames of images of the kcf algorithm, the normalized horizontal direction velocity component, the vertical direction velocity component and the motion direction angle value are taken as initialization data and integrated into the state prediction model of the Bytetrack algorithm, so that the Bytetrack algorithm can utilize the target motion trajectory information accumulated by the kcf algorithm during initialization, more accurately predict the position and motion state of the target, thereby reducing the initialization time and error, and improving the robustness and accuracy of tracking.

[0070] Embodiment two As Figure 7 shown, the technical scheme of the present application also provides an unmanned aerial vehicle target tracking system based on a hybrid algorithm, comprising: A detection and identification module 101 detects and identifies each frame of image including the unmanned aerial vehicle to be tracked based on the Yolov5 algorithm; An acquisition module 102 acquires the identification result of each frame of image including the unmanned aerial vehicle to be tracked; A first output module 103, when the Yolov5 algorithm can identify the unmanned aerial vehicle to be tracked in the current frame of image, performs ID number matching on the unmanned aerial vehicle target based on the Yolov5's bytetrack algorithm, continuously tracks the unmanned aerial vehicle target to be tracked based on the ID of the unmanned aerial vehicle target, and outputs the position of the unmanned aerial vehicle target to be tracked; A second output module 104, when the Yolov5 algorithm cannot identify the unmanned aerial vehicle to be tracked in the current frame of image, switches from the bytetrack algorithm to the kcf algorithm, performs prediction tracking on the unmanned aerial vehicle target based on the kcf algorithm, and outputs the position of the unmanned aerial vehicle target to be tracked; A switching module 105, when the kcf algorithm is switched to the bytetrack algorithm, integrates the target motion trajectory information accumulated by the kcf algorithm into the Bytetrack algorithm as initialization data, so as to reduce the initialization time and error of the Bytetrack algorithm after switching.

[0071] It should be noted that the implementation modes of the detection and recognition module 101, the acquisition module 102, the first output module 103, the second output module 104 and the switching module 105 in this embodiment correspond to the steps in Embodiment 1, and this embodiment will not be repeated here.

[0072] When the Yolov5 algorithm can recognize the unmanned aerial vehicle to be tracked in the current frame image, the ID number matching of the unmanned aerial vehicle target is performed based on the bytetrack algorithm of Yolov5, and the unmanned aerial vehicle target to be tracked is continuously tracked; when the Yolov5 algorithm cannot recognize the unmanned aerial vehicle to be tracked in the current frame image, the algorithm is switched from the bytetrack algorithm to the kcf algorithm, and the unmanned aerial vehicle target is tracked based on the kcf algorithm; when the kcf algorithm is switched to the bytetrack algorithm, the target motion trajectory information accumulated by the kcf algorithm is fused into the Bytetrack algorithm as initialization data to reduce the initialization time and error of the Bytetrack algorithm after switching. By combining the Yolo's bytetrack algorithm with the traditional tracking algorithm kcf, the advantages of both are fully utilized, the defects of each are made up, and the problem of low reliability of unmanned aerial vehicle target tracking in the prior art is solved, and the reliability of unmanned aerial vehicle target tracking is improved.

[0073] In the technical scheme of the application, the detection parameters are generated according to the current illumination condition and the complexity of the background features, and the Yolov5 algorithm is used to detect and recognize the unmanned aerial vehicle to be tracked based on the generated detection parameters, wherein the detection parameters include an anchor frame scale adjustment factor and a classification threshold, so that the Yolov5 algorithm adjusts the recognition parameters of the algorithm according to the change of the environment, and improves the reliability of unmanned aerial vehicle target detection and recognition.

[0074] In the technical scheme of the application, when the kcf algorithm is switched to the bytetrack algorithm, the corresponding illumination condition and the occlusion state of the unmanned aerial vehicle to be tracked both meet the preset switching condition; wherein the preset switching condition includes that the consecutive second preset number of image frames is greater than the classification threshold, and the corresponding illumination condition and the occlusion state of the unmanned aerial vehicle to be tracked both meet the preset recognition condition, so that the switching of the kcf algorithm and the bytetrack algorithm has accurate judgment and evaluation conditions, and the reliability of unmanned aerial vehicle target detection and recognition is ensured.

[0075] In the technical scheme of the present application, the position coordinate data of the last third preset number of frames of images after coordinate conversion of the kcf algorithm, the normalized horizontal direction velocity component, the vertical direction velocity component and the motion direction angle value are taken as initialization data and fused into the state prediction model of the Bytetrack algorithm, so that the Bytetrack algorithm can utilize the target motion trajectory information accumulated by the kcf algorithm to more accurately predict the position and motion state of the target during initialization, thereby reducing the initialization time and error and improving the robustness and accuracy of tracking.

[0076] Although the specific embodiments of the present application are described above with reference to the drawings, the description is not a limitation on the scope of protection of the present application, and those skilled in the art should understand that various modifications or changes made by those skilled in the art on the basis of the technical scheme of the present application without creative labor are still within the scope of protection of the present application.

Claims

1. A hybrid algorithm-based unmanned aerial vehicle target tracking method, characterized in that, The method comprises the following steps: detecting and identifying each frame of image including the unmanned aerial vehicle to be tracked based on the Yolov5 algorithm; obtaining the identification result of each frame of image including the unmanned aerial vehicle to be tracked; when the Yolov5 algorithm can identify the unmanned aerial vehicle to be tracked in the current frame of image, performing ID number matching on the unmanned aerial vehicle target based on the bytetrack algorithm of Yolov5, continuously tracking the unmanned aerial vehicle target to be tracked based on the ID of the unmanned aerial vehicle target, and outputting the position of the unmanned aerial vehicle target to be tracked; when the Yolov5 algorithm cannot identify the unmanned aerial vehicle to be tracked in the current frame of image, switching from the bytetrack algorithm to the kcf algorithm, performing prediction tracking on the unmanned aerial vehicle target based on the kcf algorithm, and outputting the position of the unmanned aerial vehicle target to be tracked; when the kcf algorithm switches to the bytetrack algorithm, the target motion trajectory information accumulated by the kcf algorithm is fused into the bytetrack algorithm as initialization data to reduce the initialization time and error of the bytetrack algorithm after switching. 2.The UAV target tracking method based on hybrid algorithm according to claim 1, characterized in that, The detection and identification of the unmanned aerial vehicle to be tracked based on the Yolov5 algorithm specifically comprises: generating detection parameters according to the current lighting condition and background feature complexity, and detecting and identifying the unmanned aerial vehicle to be tracked based on the generated detection parameters through the Yolov5 algorithm, wherein the detection parameters include an anchor box scale adjustment factor and a classification threshold. 3.The UAV target tracking method based on hybrid algorithm of claim 2, wherein, When the detection parameter is the anchor box scale adjustment factor, generating the detection parameter according to the current lighting condition and background feature complexity specifically comprises: ； wherein, is an anchor frame scale adjustment factor, represents a maximum value of image brightness, is a standard deviation of image brightness; is a background feature complexity index, is a first weight coefficient of image brightness, is a first weight coefficient of background feature complexity.

4. The unmanned aerial vehicle target tracking method based on hybrid algorithm according to claim 2, characterized in that, When the detection parameter is the classification threshold, generating the detection parameter according to the current lighting condition and background feature complexity specifically comprises: ； wherein, is a classification threshold, is an initial classification threshold, is an average luminance value of the image, is a second weight coefficient for image luminance, is a second weight coefficient for background feature complexity.

5. The unmanned aerial vehicle target tracking method based on hybrid algorithm according to claim 2, characterized in that, When the Yolov5 algorithm can identify the unmanned aerial vehicle to be tracked in the current frame of image, the corresponding lighting condition and the occlusion state of the unmanned aerial vehicle to be tracked both meet the preset identification condition; when the Yolov5 algorithm cannot identify the unmanned aerial vehicle to be tracked in the current frame of image, the corresponding lighting condition or the occlusion state of the unmanned aerial vehicle to be tracked does not meet the preset identification condition; wherein the preset identification condition includes a lighting identification condition and an occlusion state identification condition, wherein the lighting identification condition is that the brightness difference between the current frame of image and the previous frame of image is greater than a preset brightness threshold, or the color distribution difference between the current frame of image and the previous frame of image is greater than a preset color distribution difference threshold; the occlusion state identification condition is that the overlap rate of the detection box in the first preset number of continuous periods is less than a preset overlap rate threshold, and the number of detected feature points of the unmanned aerial vehicle target to be tracked is less than a preset feature point number threshold.

6. The unmanned aerial vehicle target tracking method based on a hybrid algorithm according to claim 5, characterized in that, The method for calculating the color distribution difference between the current frame of image and the previous frame of image is specifically as follows: ； wherein D is a color distribution difference between the current frame image and the previous frame image, denotes the i-th bin index of the color histogram of the current frame image, denotes the i-th bin index of the color histogram of the previous frame image, denotes the total number of bins of the color histogram, denotes the i-th bin index of the color histogram of the current frame image, denotes the i-th bin index of the color histogram of the previous frame image, each bin index corresponding to a frequency or normalized probability value of a color range.

7. The unmanned aerial vehicle target tracking method based on hybrid algorithm according to claim 5, characterized in that, The method for calculating the overlap rate of the detection box is specifically as follows: ； wherein, is an overlap ratio of a detection box including a UAV target to be tracked, and respectively represent a detection box of a UAV target to be tracked in two adjacent image frames, represents an intersection area of the detection boxes of the UAV target to be tracked in the two adjacent image frames, represents a union area of the detection boxes of the UAV target to be tracked in the two adjacent image frames.

8. The unmanned aerial vehicle target tracking method based on hybrid algorithm according to claim 5, characterized in that, When the kcf algorithm switches to the bytetrack algorithm, the corresponding lighting condition and the occlusion state of the unmanned aerial vehicle to be tracked both meet the preset switching condition; wherein the preset switching condition includes that the second preset number of continuous image frames is greater than the classification threshold, and the corresponding lighting condition and the occlusion state of the unmanned aerial vehicle to be tracked both meet the preset identification condition.

9. The unmanned aerial vehicle target tracking method based on a hybrid algorithm according to claim 8, characterized in that, The target motion trajectory information accumulated by the kcf algorithm is fused as initialization data into the Bytetrack algorithm, specifically: For the position coordinate data in the target motion trajectory information, coordinate conversion is performed according to the image coordinate system of the Bytetrack algorithm; For the speed information in the target motion trajectory information, the horizontal direction speed component and the vertical direction speed component are normalized respectively; For the motion direction information in the target motion trajectory information, the motion direction information is converted into an angle value, and the converted angle value is normalized; The coordinate-converted position coordinate data of the last third preset number of frames of images of the kcf algorithm, the normalized horizontal direction speed component, the vertical direction speed component and the motion direction angle value are fused as initialization data into the state prediction model of the Bytetrack algorithm.

10. A hybrid algorithm based unmanned aerial vehicle target tracking system, characterized in that, It includes: A detection and recognition module detects and recognizes each frame of image including a to-be-tracked unmanned aerial vehicle based on a Yolov5 algorithm; An acquisition module acquires the recognition result of each frame of image including the to-be-tracked unmanned aerial vehicle; A first output module, when the Yolov5 algorithm can recognize the to-be-tracked unmanned aerial vehicle in the current frame of image, performs ID number matching on the unmanned aerial vehicle target based on the Bytetrack algorithm of Yolov5, continuously tracks the to-be-tracked unmanned aerial vehicle target based on the unmanned aerial vehicle target ID, and outputs the position of the to-be-tracked unmanned aerial vehicle target; A second output module, when the Yolov5 algorithm cannot recognize the to-be-tracked unmanned aerial vehicle in the current frame of image, switches from the Bytetrack algorithm to the kcf algorithm, performs prediction tracking on the unmanned aerial vehicle target based on the kcf algorithm, and outputs the position of the to-be-tracked unmanned aerial vehicle target; A switching module, when the kcf algorithm switches to the Bytetrack algorithm, fuses the target motion trajectory information accumulated by the kcf algorithm as initialization data into the Bytetrack algorithm, so as to reduce the initialization time and error of the Bytetrack algorithm after switching.

Images