Unmanned aerial vehicle fast target detection and tracking method in complex environment

Abstract

The unmanned aerial vehicle rapid target detection and tracking method under complex environment comprises the following steps: collecting images; obtaining a multi-modal fusion image M through an enhanced double-scale image fusion algorithm; constructing a multi-modal fusion target detection model based on Yolov5 and pruning; constructing an airborne target tracking model based on SimaFc and pruning; inputting the image M into the multi-modal fusion target detection model based on Yolov5 for processing to obtain an initial target detection result Y1; according to Y1, a template image set Y2 is generated by cropping from the image M; according to Y1, a larger region near the target position determined from the image M is taken as a search area Y3 of the tracking target; Y2 and Y3 are input into two convolutional neural network branches of the airborne target tracking model based on SimaFc for feature extraction, and the most similar part in the search area to the template image is judged by using similarity and distance measurement, so that target tracking is realized. The target detection and tracking method provided in the scheme has high accuracy.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and particularly relates to a method for rapid target detection and tracking of unmanned aerial vehicles in complex environments. BACKGROUND

[0002] With the progress of science and technology, the application of unmanned aerial vehicles is becoming more and more extensive. By mounting a high-definition camera on an unmanned aerial vehicle, combined with a flight control system and related algorithms, the unmanned aerial vehicle can realize the identification, detection and tracking of the photographed object. The domestic unmanned aerial vehicle market has developed for nearly 30 years, and enterprises and institutions at home and abroad are actively researching and promoting this technology. The application of the unmanned aerial vehicle target detection and tracking system is also constantly expanding and developing. At present, domestic unmanned aerial vehicle manufacturers have increased investment in the target detection and tracking system technology, and have achieved certain research results. At the same time, some professional unmanned aerial vehicle technology enterprises have also made great contributions in this field, and various types of unmanned aerial vehicles are flourishing in the fields of security, disaster relief, agriculture and animal husbandry, power, aerial photography and surveying and mapping, etc.

[0003] Deeply combining unmanned aerial vehicle technology with artificial intelligence technology, using artificial intelligence technology, machine vision technology, etc. to endow unmanned aerial vehicle technology with wisdom core, the combat performance of unmanned aerial vehicle technology in the fields of civil and military use, etc. can be greatly improved, and human resources can be further liberated. By mounting visual perception equipment on the unmanned aerial vehicle platform, combined with the current intelligent perception technology and the light and flexible characteristics of unmanned aerial vehicles, the perception of the scene environment and the target in the environment can be effectively and comprehensively expanded.

[0004] In the field of unmanned aerial vehicle visual perception, the detection, identification, tracking and positioning of targets in the environment are the basis of many tasks. For example, the perception of targets in the battlefield environment is the basis for subsequent target attack and situation analysis tasks. Therefore, the research on unmanned aerial vehicle target detection has very important research significance.

[0005] In the field of unmanned aerial vehicle image target detection, the current methods for unmanned aerial vehicle vision target detection and recognition are mainly based on deep learning, which can be divided into Anchor-based and Anchor-free. The Anchor-based method is further divided into One-stage and Two-stage. In recent years, the Two-stage method includes Faster-RCNN, FPN, Light-RCNN and Cascade-RCNN, and the One-stage method includes YOLOv3, SSD, RetinaNet and RefineDet. For the Two-stage method, the target positioning module and the target classification module are implemented independently and then combined in sequence to achieve the target detection and recognition task. Therefore, the running process of this kind of algorithm needs to go through two stages: target positioning stage and target classification stage. Compared with the One-stage method, the accuracy is better, but the speed is slower. The Anchor-free method avoids the problem of manual design of target box in the Anchor-based method, which is sensitive to the size of the target in different data sets, and also reduces the large amount of calculation consumption caused by the regression calculation of the target box. However, this method still has problems in the application of unmanned aerial vehicle vision target detection, such as variable view angle, variable scale, small target or even extremely small target, motion blur, complex lighting conditions and other characteristics of unmanned aerial vehicle images, which cannot directly solve the inherent problems of unmanned aerial vehicle images.

[0006] In the field of unmanned aerial vehicle vision target tracking, the target tracking task of unmanned aerial vehicle vision can be roughly divided into image target detection tracking, video object detection tracking, single target tracking and multi-target tracking. Image target tracking, that is, given a set of predefined object classes, such as aircraft, vehicle and personnel, the algorithm will detect these classes of objects from a single image taken by the unmanned aerial vehicle, so that the image target detection tracking is also called image target detection. Video object detection tracking is similar to image target detection tracking, and the algorithm will detect the objects of the predefined object classes from the video taken by the unmanned aerial vehicle. Single target tracking is a continuous tracking of a single target in a video stream. There are three main methods for target tracking: one is the correlation filter-based method, such as ECO, C-COT and BACF; the second is the semantic network-based method, such as SiamRPN++, Siam R-CNN and SiamMask; and the third is the CNN-based method, such as MDNet, CFNet and ATOM. For the target tracking task of unmanned aerial vehicles, sudden motion, low resolution (small target), complex lighting and occlusion can easily cause the unmanned aerial vehicle to fail to accurately complete the target tracking, and this method cannot directly solve the inherent problems of unmanned aerial vehicle images.

[0007] In view of the above problems, based on the images collected by the unmanned aerial vehicle equipped with the photoelectric pod, it is of great significance and value to study the realization of target detection and tracking of the unmanned aerial vehicle. In the detection and tracking task of the unmanned aerial vehicle, the power supply of the unmanned aerial vehicle needs to support the power consumption of the unmanned aerial vehicle and the computing power in the field of machine vision technology, so the computing power of the unmanned aerial vehicle is difficult to bear the huge deep learning computing demand. Therefore, for the deep learning computing model of the unmanned aerial vehicle on-board, the neural network model needs to be lightened to reduce the computing load and improve the computing speed. SUMMARY

[0008] The object of the application is to provide a rapid target detection and tracking method for unmanned aerial vehicles in complex environments, based on images collected by unmanned aerial vehicles equipped with photoelectric pods, using a multi-modal fusion algorithm based on infrared + visible light to realize small target detection, and on this basis, realizing continuous tracking of target objects, and finally realizing lightweight deployment of the model in combination with the power consumption of the unmanned aerial vehicle and the computing power bearing capacity. In particular, the present application mainly solves the problems of target detection and tracking in the case of variable view angle, variable scale, small target or even extremely small target, motion blur, complex lighting, target occlusion and the like, and the model deployment problem suitable for the limited computing power resources and power bearing capacity of the unmanned aerial vehicle.

[0009] The technical solution is a rapid target detection and tracking method for unmanned aerial vehicles in complex environments, comprising the following steps:

[0010] Step S1, collect images including visible light images and infrared images by the unmanned aerial vehicle equipped with the photoelectric pod, then perform data augmentation processing to expand the original available data amount, and generate occluded target detection data types;

[0011] Step S2, convert the visible light image and the infrared image into a visible light video frame image M1 with a width of W and a height of H, and an infrared video frame M2 with a width of W and a height of H, and obtain a multi-modal fusion image M through an enhanced double-scale image fusion algorithm;

[0012] Step S3, construct a multi-modal fusion target detection model based on Yolov5, and perform channel pruning and layer pruning on the multi-modal fusion target detection model based on Yolov5 to ensure that the target detection model after pruning still maintains high detection accuracy while reducing the amount of calculation, and combine the actual situation of hardware resources to perform quantization operation on the model, further compress the model, and improve the model inference speed;

[0013] Step S4, a SimaFc-based airborne target tracking model is constructed, and path pruning and layer pruning are performed on the SimaFc-based airborne target tracking model to ensure that the target tracking model after pruning has reduced computational complexity without significantly affecting the target tracking effect, and the model is quantized according to the actual situation of the hardware resources to further compress the model and improve the model inference speed;

[0014] Step S5, inputting the multi-modal fusion image M obtained in step S2 into the multi-modal fusion target detection model based on Yolov5 constructed in step S3 for processing to obtain an initial target detection result Y1;

[0015] Step S6, according to the target detection result Y1 obtained in step S5, a template image set Y2 containing only targets is generated by cropping from the multi-modal fusion image M with a width of W and a height of H, and the Y2 set is a series of initial target template images [m1, m2,..., mn] contained in the image M';

[0016] Step S7, according to the target detection result Y1 obtained in step S5, a larger region near the target position determined from the multi-modal fusion image M with a width of W and a height of H is intercepted as a search area for tracking targets, and the search area is denoted as a set Y3, and the Y3 set is a series of search area images [n1, n2,..., nn] contained in the image M corresponding to the template image set Y2;

[0017] Step S8, inputting the template image set Y2 and the search area image set Y3 obtained in step S7 into the two convolutional neural network branches of the SimaFc-based airborne target tracking model constructed in step S4 for feature extraction, and using similarity and distance measurement to determine the most similar part in the search area to the template image, thereby realizing target tracking.

[0018] According to a further improvement of the application, the enhanced dual-scale image fusion algorithm in step S2 comprises the following steps:

[0019] Step S21, inputting two images, one being a visible light image and the other being a corresponding infrared image, and denoting them as I1 and I2 respectively, and decomposing the two images into a base layer and a detail layer using bilateral filtering; Step S22, fusing the decomposed base layer and detail layer respectively, wherein the base layer is fused by weighted averaging, and the detail layer is fused by weighted averaging using a fusion coefficient matrix constructed by a salient feature map;

[0020] Step S22, fusing the decomposed base layer and detail layer respectively, wherein the base layer is fused by weighted averaging, and the detail layer is fused by weighted averaging using a fusion coefficient matrix constructed by a salient feature map;

[0021] ​Step S23, after obtaining the fused base layer and detail layer, the base layer and the detail layer are fused by simple addition, that is, the channel elements corresponding to the visible light image and the infrared image are subjected to concat operation to obtain the final multi-modal fusion data M.

[0022] According to a further improvement of the application, the multi-modal fusion target detection model based on Yolov5 constructed in step S3 is an improved target detection model, comprising:

[0023] Step S31, constructing a backbone network Backbone: composed of Darknet combined with a CSP structure, extracting features at different levels through layer-by-layer convolution, activation, batch normalization and the like, and providing rich low-level and high-level features for subsequent detection modules;

[0024] Step S32, constructing a feature fusion layer Neck: adopting a path aggregation network structure, combining a feature pyramid network and a path aggregation network, and through multi-layer feature fusion, making the model more suitable for detecting targets of different sizes and enhancing the recognition ability of small targets and distant targets;

[0025] Step S33, constructing a detection head Head: generating different Anchor frames for each feature layer, and calculating the probability of target classes and the position of the bounding box for each Anchor, usually using three scales to predict large, medium and small targets, and the generated feature map specific bounding box information includes position, size, confidence and class, through a non-maximum suppression process, removing overlapping frames and retaining the prediction frame with the highest confidence to obtain the final detection result; wherein the C3-RS module is used to replace the C3 module in the original structure, R refers to the Bottleneck residual block in ResNet, and the SE module is combined to make the channel attention more concentrated and improve the recognition ability of small targets.

[0026] According to a further improvement of the application, the multi-modal fusion target detection model based on Yolov5 constructed in step S3 is an improved target detection model, comprising:

[0027] Step S41, constructing a backbone network Backbone: based on MobileNet, composed of two convolutional neural network branches sharing weights, and extracting features from the template image and the search area respectively;

[0028] Step S42, constructing a similarity measurement module: matching the template features and the search area features through the cross-correlation layer to measure the similarity of the two;

[0029] Step S43, a response map generation module is constructed: the result of the cross-correlation is used to generate a response map, i.e., a similarity distribution map, and the most likely position of the target is located by comparing the response values at different positions in the response map, and the peak value position in the response map, i.e., the position prediction of the target;

[0030] Step S44, a blocking detection module is constructed: by calculating the change mode in the response map, it is determined whether the target is blocked when the similarity suddenly decreases greatly or becomes blurred; when it is determined that blocking occurs, the blocking detection module triggers a repositioning strategy, and the target is repositioned through the response map of the last frame and the historical trajectory of the target;

[0031] Step S45, a target positioning and updating module is constructed, the position of the bounding box of the target is adjusted according to the peak value position of the response map, and the model is updated based on the confidence;

[0032] Step S46, a post-processing module is constructed: smoothing and screening operations are performed to improve the accuracy and stability of tracking in complex environments.

[0033] According to the further improvement of the application, the multi-modal fusion target detection model based on Yolov5 is pruned and quantized in step S3 by a model lightweight method based on joint channel pruning and layer pruning, which includes the following steps:

[0034] S3a, channel pruning: a pruning script is defined, and basic channel pruning is implemented in models / common.py in the Yolov5 code framework;

[0035] S3b, layer pruning: the contribution degree of a layer is evaluated by measuring the importance of the layer, and the layer with a low contribution degree is preferentially pruned; layer pruning is realized based on the sparsity of the feature map, and the proportion of non-zero values or the average absolute value in the activation output is counted to find the layer with high sparsity, i.e., the feature map output of the layer is very sparse or even close to zero, indicating that the role of the layer is small and the layer needs to be pruned;

[0036] S3c, model fine-tuning: after the channel pruning and layer pruning operations, the model is fine-tuned respectively to restore the accuracy of the model;

[0037] S3d, model quantization: the weights in the floating-point number FP32 format are quantized to the integer INT8 format, and a small part of real scene samples is used as calibration data, which is input into the model to collect the maximum and minimum value information of the activation layer, and then according to the range of the samples, the activation and weight of each layer in the model are quantized to INT8 representation.

[0038] According to a further improvement of the present application, the SimaFc-based on-board terminal target tracking model is pruned and quantized in step S4 through a model lightweight method based on joint channel pruning and layer pruning, comprising the following steps:

[0039] Step S4a, channel pruning: the importance of each channel is determined by calculating the batch normalization layer scaling factor gamma parameter of each channel, the scaling factor representing the activation amplitude of the channel; wherein the greater the gamma value, the more important the features of the channel, and the smaller the value, the smaller the influence of the channel on the features in the model, the channels with lower weights are selected as pruning candidates, a certain proportion of channels are retained based on the set pruning rate, and 80% of the total channel number is retained; after pruning, the number of channels of the model is adjusted to ensure that the input and output dimensions of the downstream convolutional layer are consistent;

[0040] Step S4b, layer pruning: the influence of deleting layers on model accuracy is evaluated layer by layer, sensitivity analysis is performed on each layer of the MobileNet convolutional layer, and the influence of each convolutional layer on model accuracy is recorded; the model accuracy is obtained by using the validation set for fast inference test, a layer with little influence on accuracy is marked as a pruning candidate; the influence of the accuracy is determined by setting an accuracy threshold, and the accuracy decreases by less than 1% after removing a layer, which is considered to be removed; according to the results of the sensitivity analysis, redundant layers are selected from the deep convolutional layers with the smallest influence for pruning;

[0041] Step S4c, model fine-tuning: after the channel pruning and layer pruning operations, the model is fine-tuned respectively to restore the accuracy of the model;

[0042] Step S4d, model quantization: the floating-point number FP32 format weight is quantized to integer INT8 format, and a small part of real scene samples is used as calibration data, which is input into the model, and the maximum and minimum value information of the activation layer is collected, and then according to the sampling range, the activation and weight of each layer in the model are quantized to INT8 representation.

[0043] Beneficial effects: the unmanned aerial vehicle rapid target detection and tracking method in complex environment provided by the application is a complete set of feasible scheme including target detection and tracking and model lightweight deployment; the multi-modal fusion target detection method based on Yolov5 provided by the application uses multi-modal fusion images as input, utilizes the feature complementarity in visible light and infrared images, and combines C3-Res-S module convolution design to enhance the detection ability of targets, especially small targets; the airborne end target tracking method based on SimaFc provided by the application uses a double-branch twin network structure, adopts a lightweight convolution network as a feature extractor to improve the calculation efficiency and ensure the real-time performance of target tracking, and combines a shielding detection module and multi-modal information to enhance the robustness; the model lightweight method based on joint channel pruning and layer pruning provided by the application is specially optimized for resource-limited airborne device deployment, and considers the computing power, storage space and real-time requirements of the device. Reduce model parameters and calculation amount, while avoiding excessive precision loss. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is the flow chart of the dual-scale image fusion algorithm of the application.

[0045] Figure 2 is the flow chart of the multi-channel data fusion method of the application.

[0046] Figure 3 is the multi-modal fusion target detection model structure diagram based on Yolov5 of the application.

[0047] Figure 4 is the airborne end target tracking model structure diagram based on SimaFc of the application.

[0048] Figure 5 is the comparison chart of the results before and after model compression and optimization of the application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the application clearer, the technical scheme in the embodiment of the application is described clearly and completely. Obviously, the described embodiment is a part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the application.

[0050] Embodiment 1

[0051] The unmanned aerial vehicle rapid target detection and tracking method in complex environment comprises the following steps:

[0052] Step S1, an unmanned aerial vehicle carrying a photoelectric pod is used to collect images, including visible light images and infrared images, then data augmentation processing is performed to expand the original available data amount, and an occluded target detection data type is generated;

[0053] Step S2, the visible light image and the infrared image are converted into a visible light video frame image M1 with a width of W and a height of H, and an infrared video frame M2 with a width of W and a height of H, and a multi-modal fusion image M is obtained through an enhanced double-scale image fusion algorithm;

[0054] Step S3, a multi-modal fusion target detection model based on Yolov5 is constructed, and channel pruning and layer pruning are performed on the multi-modal fusion target detection model based on Yolov5 to ensure that the pruned target detection model still maintains high detection accuracy while reducing the amount of calculation, and the model is quantized according to the actual situation of hardware resources to further compress the model and improve the model inference speed;

[0055] Step S4, a SimaFc-based airborne target tracking model is constructed, and channel pruning and layer pruning are performed on the SimaFc-based airborne target tracking model to ensure that the pruned target tracking model does not have a great impact on the target tracking effect while reducing the amount of calculation, and the model is quantized according to the actual situation of hardware resources to further compress the model and improve the model inference speed;

[0056] Step S5, the multi-modal fusion image M obtained in step S2 is input into the multi-modal fusion target detection model based on Yolov5 constructed in step S3 for processing to obtain an initial target detection result Y1;

[0057] Step S6, according to the target detection result Y1 obtained in step S5, a template image set Y2 containing only targets is generated by cropping from the multi-modal fusion image M with a width of W and a height of H, and the Y2 set is a series of initial target template images [m1, m2,..., mn] contained in the image M';

[0058] Step S7, according to the target detection result Y1 obtained in step S5, a larger region near the target position determined from the multi-modal fusion image M with a width of W and a height of H is intercepted as a search area for tracking targets, and the search area is denoted as a set Y3, and the Y3 set is a series of search area images [n1, n2,..., nn] contained in the image M, corresponding to the template image set Y2;

[0059] Step S8: Input the template image set Y2 and the search region image set Y3 obtained in step S7 into the two convolutional neural network branches of the SimaFc-based airborne target tracking model constructed in step S4 for feature extraction. Use similarity and distance metrics to determine the part in the search region that is most similar to the template image, thereby achieving target tracking.

[0060] According to a further improvement of the present invention, the enhanced dual-scale image fusion algorithm in step S2 includes the following steps:

[0061] Step S21: Input two images, one a visible light image and the other a corresponding infrared image, and use... and This indicates that bilateral filtering is used to decompose the two images into a base layer and a detail layer, respectively.

[0062] Step S22: The decomposed base layer and detail layer are fused separately. The base layer is fused using a weighted average method, while the detail layer is fused using a weighted average method based on the fusion coefficient matrix constructed from the salient feature map.

[0063] Step S23: After obtaining the fused base layer and detail layer, the base layer and detail layer are then fused by simple addition, that is, the corresponding channel elements of the visible light image and the infrared image are concatted to obtain the final multimodal fused data M.

[0064] According to a further improvement of the present invention, the Yolov5-based multimodal fusion target detection model constructed in step S3 is an improved target detection model, comprising:

[0065] Step S31: Construct the backbone network: It is composed of Darknet and CSP structure. It extracts features at different levels through layer-by-layer convolution, activation, batch normalization and other operations to provide rich low-level and high-level features for subsequent detection modules.

[0066] Step S32: Construct the feature fusion layer Neck: Adopt the path aggregation network structure, combine the feature pyramid network and the path aggregation network, and through multi-layer feature fusion, make the model more adaptable to the detection of targets of different sizes, and enhance the recognition ability of small targets and distant targets.

[0067] Step S33, constructing a detection head Head: different Anchor boxes are generated for each feature layer, and the probability of the target class and the boundary box position are calculated for each Anchor, usually three scales are used to predict large, medium and small targets respectively, the generated feature map specific boundary box information includes position, size, confidence and class, through the non-maximum suppression process, overlapping boxes are removed and the prediction box with the highest confidence is retained to obtain the final detection result; wherein the C3-RS module is used to replace the C3 module in the original structure, R refers to the Bottleneck residual block in ResNet, and the SE module is combined to make the channel attention more concentrated and improve the recognition ability of small targets.

[0068] According to a further improvement of the application, the SimaFc-based on-board target tracking model constructed in step S4 is an improved target tracking model, comprising:

[0069] Step S41, constructing a backbone network Backbone: based on MobileNet, the backbone network is composed of two convolutional neural network branches sharing weights, and features are extracted from the template image and the search area respectively;

[0070] Step S42, constructing a similarity measurement module: the template features and the search area features are matched through the cross-correlation layer to measure the similarity of the two;

[0071] Step S43, constructing a response map generation module: the cross-correlation result generates a response map, i.e. a similarity distribution map, by comparing the response values at different positions in the response map, the most likely position of the target is located, and the peak value position in the response map, also called the response maximum value, is the position prediction of the target;

[0072] Step S44, constructing an occlusion detection module: by calculating the change pattern in the response map, when a large drop or blurring in similarity is detected, it is determined whether the target is occluded; when it is determined that occlusion occurs, the occlusion detection module triggers a repositioning strategy to reposition the target through the response map of the last frame and the historical trajectory of the target;

[0073] Step S45, constructing a target positioning and updating module, adjusting the boundary box position of the target according to the peak value position of the response map, and updating the model based on the confidence;

[0074] Step S46, constructing a post-processing module: smoothing and screening operations are performed to improve the accuracy and stability of tracking in complex environments.

[0075] According to a further improvement of the application, the multi-modal fusion target detection model based on Yolov5 is pruned and quantized in step S3 through a model lightweight method based on joint channel pruning and layer pruning, comprising the following steps:

[0076] S3a, channel pruning: a custom pruning script is defined to implement basic channel pruning in models / common.py in the Yolov5 code framework;

[0077] S3b, layer pruning: the contribution of a certain layer is evaluated by the importance of the layer, and the layer with low contribution is preferentially pruned; layer pruning is implemented based on the sparsity of feature maps, and the proportion of non-zero values or the average absolute value in the activation output is counted to find the layer with high sparsity, that is, the feature map output of the layer is very sparse or even close to zero, indicating that the role of the layer is small and the layer needs to be pruned;

[0078] S3c, model fine-tuning: after channel pruning and layer pruning operations, the model is fine-tuned respectively to restore the accuracy of the model;

[0079] S3d, model quantization: the weight in floating point number FP32 format is quantized to integer INT8 format, and a small part of real scene samples is used as calibration data, which is input into the model, and the maximum and minimum value information of the activation layer is collected, and then according to the range of sampling, the activation and weight of each layer in the model are quantized to INT8 representation.

[0080] According to a further improvement of the application, the SimaFc-based airborne target tracking model is pruned and quantized in step S4 by a model lightening method based on joint channel pruning and layer pruning, comprising the following steps:

[0081] Step S4a, channel pruning: the batch normalization layer scaling factor γ parameter of each channel is calculated, and the scaling factor represents the activation amplitude of the channel. The importance of the channel is judged by analyzing these values; the greater the γ value, the more important the features of the channel, and the smaller the value, the smaller the influence of the channel on the features in the model. The channels with low weights are selected as pruning candidates, and a certain proportion of channels are reserved based on the set pruning rate. The total number of channels is 80%, and the number of channels of the model is adjusted after pruning to ensure that the input and output dimensions of the downstream convolutional layer are consistent;

[0082] Step S4b, layer pruning: the influence of deleting a layer on the model accuracy is evaluated layer by layer, and sensitivity analysis is performed on each convolutional layer of MobileNet to record the influence of each convolutional layer on the model accuracy; the validation set is used for fast inference test to obtain the model accuracy, and a layer with little influence on the accuracy is marked as a pruning candidate; the influence on the accuracy is determined by setting an accuracy threshold, and the layer is removed if the accuracy decreases by less than 1% after removing the layer; according to the results of the sensitivity analysis, redundant layers are selected from the deep convolutional layers with the smallest influence for pruning;

[0083] Step S4c, model fine-tuning: after the channel pruning and layer pruning operations are performed, the model is fine-tuned respectively to restore the accuracy of the model;

[0084] Step S4d, model quantization: the weights in the floating-point FP32 format are quantized into the integer INT8 format, and a small part of real scene samples are used as calibration data, input into the model, and the maximum and minimum value information of the activation layer is collected, and then according to the range of sampling, the activation and weight of each layer in the model are quantized to the INT8 representation.

[0085] Embodiment 2

[0086] The complex environment-based unmanned aerial vehicle rapid target detection and tracking method proposed by the application will be described in detail below, which is characterized by three parts: multi-modal fusion target detection based on Yolov5, airborne target tracking based on SimaFc, and model lightening based on joint channel pruning and layer pruning.

[0087] The multi-modal fusion target detection based on Yolov5 includes the following steps:

[0088] Step 1_1: take a visible light video frame image M1 with a width of W and a height of H, and an infrared video frame M2 with a width of W and a height of H, and obtain a multi-modal fusion image M by an enhanced double-scale image fusion algorithm;

[0089] Step 1_2: input the multi-modal fusion image M with a width of W and a height of H obtained in the above step into the multi-modal fusion target detection model based on Yolov5 to obtain the initial target detection result Y1, and the format of Y1 is [class_id, x_center, y_center, width, height, confidence], wherein class_id represents the category of the target, (x, y, width, height) represents the coordinates of the detection frame, (x, y) is the center coordinates of the frame, (width, height) is the width and height of the frame, and confidence represents the confidence of the predicted bounding box, which is a value between 0 and 1.

[0090] The airborne target tracking based on SimaFc includes the following steps:

[0091] Step 2_1: according to the target detection result Y1 obtained in the above step 1_2, the template image set Y2 containing only the target is generated by cropping from the multi-modal fusion image M with a width of W and a height of H, and the set Y2 contains a series of initial target template images [m1, m2,..., mn] contained in the image M'.

[0092] Step 2_2: According to the target detection result Y1 obtained in step 1_2 above, a larger region near the target position determined from the multi-modal fusion image M with a width of W and a height of H is taken as the search area of the tracking target, denoted as set Y3, which contains a series of search area images [n1, n2,..., nn] in image M corresponding to the template image set Y2.

[0093] Step 2_3: The template image set Y2 obtained in step 2_1 above and the search area image set Y3 obtained in step 2_2 above are respectively input into the two convolutional neural network branches of the SiamaFC (Siamese Fully-Convolutional Network) based on-board target tracking model for feature extraction. On this basis, the model uses similarity or distance measurement to determine the most similar part in the search area to the template image, thereby achieving target tracking.

[0094] Wherein, the model lightweighting based on joint channel pruning and layer pruning includes the following steps:

[0095] Step 3_1: Channel pruning and layer pruning are performed on the above-mentioned multi-modal fusion target detection model based on Yolov5 to ensure that the pruned target detection model still maintains high detection accuracy while reducing computational complexity. And combined with the actual situation of hardware resources, the model is quantized to further compress the model and improve the model inference speed.

[0096] Step 3_2: Channel pruning and layer pruning are performed on the above-mentioned SimaFc based on-board target tracking model to ensure that the pruned target tracking model does not have a significant impact on target tracking while reducing computational complexity. And combined with the actual situation of hardware resources, the model is quantized to further compress the model and improve the model inference speed.

[0097] The above steps are described in detail as follows:

[0098] In the above step 1_1, an enhanced dual-scale image fusion algorithm is used to fuse visible light images and infrared images, which includes the following steps:

[0099] Image decomposition:

[0100] Two images are input, one is a visible light image and the other is a corresponding infrared image, represented by and respectively. Bilateral filtering (BF) is used to decompose the two images into base layer and detail layer, which is calculated as follows:

[0101]

[0102] where i = 1, 2, is the input image; Ω is a neighborhood range, a set of neighborhood pixels centered at (x, y); is the intensity weight function, which controls the influence of intensity difference; f s (||(x, y)-(u, v)||) is the spatial weight function, which controls the influence of spatial distance; W x,y is the normalization coefficient, which is used to ensure the normal range of output brightness; denotes the base layer image.

[0103] After obtaining the shallow layer feature, the detail layer is obtained by difference operation between the original input image and the base layer image. The method for obtaining the detail layer is as follows:

[0104]

[0105] where i = 1, 2, denotes the original input image, denotes the gradient amplitude image obtained in the last step, i.e., the base layer image, is the output detail layer image.

[0106] For the weight calculation of the detail layer, the input image and are processed by the spectral residual method (SR) to generate two significant feature maps, and then inverse Fourier transform is performed on each image to obtain the significant feature maps ψ1(x, y) and ψ2(x, y) in the spatial domain, ψ i The calculation process of ψ1(x, y) and ψ2(x, y) is as follows:

[0107] ψ i (x, y) = F -1 {e R(u,v) ·e jP(u.v)}

[0108] where i = 1, 2, ψ i (x, y) is the significant map in the spatial domain, R(u, v) is the spectral residual calculated by the spectral residual method, j is the imaginary unit, P(u, v) is the phase spectrum of the input image, F-1 represents the inverse Fourier transform; eR(u, v) represents the value of adjusting the amplitude spectrum by the spectral residual R(u, v), which is converted into nonlinear scaling through the property of the exponential function to highlight the significant features.

[0109] Therefore, the fusion coefficient matrix of the detail layer is as follows:

[0110]

[0111] Image fusion:

[0112] For the base layer, the fusion strategy of weighted average is used:

[0113]

[0114] where B fused (x,y) represents the fused base layer, and respectively represent the base layers of visible light and infrared images, and the weight α ranges from 0 to 1, which can be adjusted according to the brightness characteristics of the image. Specifically, according to the different modes of day and night, the value of α is dynamically adjusted, for example, during the night, the quality of the infrared image is better than that of the visible light, and the value of α is 0.3-0.4, while if it is during the day, the quality of the visible light image is higher, and the value of α is 0.6-0.7.

[0115] For the detail layer, the fusion strategy of weighted average using the fusion coefficient matrix constructed by the salient feature map is used:

[0116]

[0117] where D fused (x,y) represents the fused detail layer, and ξ1(x,y) and ξ2(x,y) are the weighting coefficients obtained by the spectral residual method, and respectively represent the detail layers of visible light and infrared images.

[0118] Image reconstruction:

[0119] After obtaining the final base layer and detail layer, the base layer and the detail layer are fused by simple addition, that is, the corresponding channel elements of the visible light image and the infrared image are concatenated to obtain the final multi-modal fusion data γ(x,y). The specific multi-channel data fusion process is shown in the attached Figure 2 .

[0120] In the above step 1_2, the multi-modal fusion target detection model based on Yolov5 is obtained after training, and the training process includes the following steps:

[0121] Download the UAVDT-DatasetNinja dataset;

[0122] The training data and the test data are divided, 8000 pairs of visible light and infrared images are selected from the UAVDT-DatasetNinja dataset as the training data, and 2000 pairs of visible light and infrared images are selected as the test data. It is ensured that the visible light image and the infrared image are strictly corresponding, and the visible light image and the infrared image are fused by using the above-mentioned enhanced double-scale image fusion algorithm to obtain the fused training data and test data.

[0123] The above-mentioned prepared image data is used to pre-train the multi-modal fusion target detection model based on yolov5, and the model weight w1 obtained by pre-training is saved. In order to ensure the detection accuracy of the model in the real application scene, 1000 pairs of real scene data are used to fine-tune the model. Specifically, first, 1000 pairs of real scene data are divided into training data and test data according to the ratio of 8:2, the pre-weight w1 obtained by training is used to initialize the model, the hyperparameters related to model training are adjusted according to the training loss value, and the model weight w2 obtained by the second training is used as the final weight of the target detection model.

[0124] In the above step 2_1, the template image is obtained by cropping a fixed size of the target region in the initial frame, and the size of the cropped template image is 127x127 pixels;

[0125] In the above step 2_2, the search area is obtained by cropping a larger region in the next frame relative to the initial frame, which is used for detecting the target, and the image size of the search area is 255x255 pixels;

[0126] In the above step 2_3, the airborne target tracking model based on SimaFc is trained, and the training process includes the following steps:

[0127] According to the training set and the validation set divided according to the above-mentioned UAVDT-DatasetNinja dataset, the template image and the corresponding search area image are cropped by using the existing data label, which are used as the available data for model training.

[0128] The above-mentioned prepared image data is used to pre-train the airborne target tracking model based on SimaFc, and the model weight w3 obtained by pre-training is saved. In order to ensure the tracking accuracy of the model in the real application scene, 1000 pairs of real scene data are used to fine-tune the model. Specifically, first, 1000 pairs of real scene data are divided into training data and test data according to the ratio of 8:2, the pre-weight w3 obtained by training is used to initialize the model, the hyperparameters related to model training are adjusted according to the training loss value, and the model weight w4 obtained by the second training is used as the final weight of the target tracking model.

[0129] In step 3_1 above, the channel pruning, layer pruning and model quantization are performed on the multi-modal fusion target detection model based on Yolov5. The specific steps are as follows:

[0130] Channel pruning: a custom pruning script is implemented in models / common.py in the Yolov5 code framework to achieve basic channel pruning. Channel pruning is mainly implemented based on L1 norm, and the part with the smallest absolute value of the channel is considered to have the smallest impact on the model performance, i.e., the feature channel needs to be pruned.

[0131] Layer pruning: the contribution of a layer is evaluated by measuring the importance of the layer, and the layer with low contribution is pruned first. Layer pruning is mainly implemented based on the sparsity of feature maps. By counting the proportion of non-zero values or the average absolute value in the activation output, the layer with high sparsity is found, i.e., the feature map output of the layer is very sparse or even close to zero, indicating that the role of this layer is small and it is a layer that needs to be pruned.

[0132] Model fine-tuning: after channel pruning and layer pruning, the model needs to be fine-tuned respectively to restore the accuracy of the model.

[0133] Model quantization: model quantization aims to further reduce the size of the model, reduce memory consumption, and speed up the inference speed. Specifically, the floating-point number FP32 format weight is quantized to integer INT8 format, and a small part of real scene samples is used as calibration data, which is input into the model to collect the maximum and minimum value information of the activation layer. Then, according to the range of the sampled data, the activation and weight of each layer in the model are quantized to INT8 representation.

[0134] In step 3_2 above, the channel pruning and layer pruning are performed on the airborne target tracking model based on SimaFc. The specific steps are as follows:

[0135] Channel pruning: by calculating the batch normalization layer (Batch Normalization, BN) scaling factor γ parameter of each channel, the scaling factor represents the activation amplitude of the channel, and the importance of the channel is judged by analyzing these values. The larger the γ value, the more important the features of the channel, and the smaller the value, the less important the channel in the model. Select the channel with low weight as the pruning candidate, and retain the top proportion of channels based on the set pruning rate, retaining 80% of the total number of channels. After pruning, adjust the number of channels of the model to ensure that the input and output dimensions of the downstream convolutional layer are consistent.

[0136] Layer pruning: evaluate the impact of deleting layers on the model accuracy layer by layer, especially on the accuracy of the target response map generation and occlusion detection module. Sensitivity analysis is performed on each convolutional layer of MobileNet, and the impact of each convolutional layer on the model accuracy is recorded. The model accuracy is obtained by using the validation set for fast inference test. If a certain layer has little impact on the accuracy, it is marked as a pruning candidate. The impact on the accuracy is determined by setting an accuracy threshold. If the accuracy decreases by less than 1% after removing a certain layer, it is considered that the layer can be removed. According to the results of the sensitivity analysis, redundant layers are selected from the deep convolutional layers with the least impact for pruning.

[0137] Model fine-tuning: after the channel pruning and layer pruning operations, the model needs to be fine-tuned respectively to restore the accuracy of the model.

[0138] Model quantization: as described above, the weights in the floating-point FP32 format are quantized to the integer INT8 format, and a small part of real scene samples is used as calibration data to input into the model, collect the maximum and minimum value information of the activation layer, and then according to the range of the samples, the activation and weight of each layer in the model are quantized to the INT8 representation.

[0139] As described above, although the present application has been shown and described with reference to certain preferred embodiments, it is to be understood that such is by way of illustration and not of limitation. Various changes and modifications can be made therein without departing from the spirit and scope of the application as defined in the appended claims.

Claims

1. A method for rapid target detection and tracking of unmanned aerial vehicles in complex environments, characterized in that, The method comprises the following steps: Step S1, using an unmanned aerial vehicle carrying an optical pod to collect images, including visible light images and infrared images, then performing data augmentation processing, expanding the original available data amount, and generating occluded target detection data types; Step S2, converting the visible light image and the infrared image into a visible light video frame image M1 with a width of W and a height of H, and an infrared video frame M2 with a width of W and a height of H, obtaining a multi-modal fusion image through an enhanced two-scale image fusion algorithm ; Step S3, constructing a multi-modal fusion target detection model based on Yolov5, and performing channel pruning and layer pruning on the multi-modal fusion target detection model based on Yolov5, ensuring that the pruned target detection model still maintains high detection accuracy while reducing the amount of calculation, and combining the actual situation of hardware resources, performing quantization operation on the model, further compressing the model, and improving the model inference speed; Step S4, constructing a SimaFc-based airborne target tracking model, and performing channel pruning and layer pruning on the SimaFc-based airborne target tracking model, ensuring that the pruned target tracking model does not have a great impact on the target tracking effect while reducing the amount of calculation, and combining the actual situation of hardware resources, performing quantization operation on the model, further compressing the model, and improving the model inference speed; Step S5, obtaining the multi-modal fusion image obtained in step S2 The multi-modal fusion target detection model based on Yolov5 constructed in step S3 is input for processing, and an initial target detection result Y1 is obtained. Step S6, according to the target detection result Y1 obtained in step S5, from the multi-modal fusion image with a width of W and a height of H The template image set Y2 containing only the target is generated by cropping, and the Y2 set is a series of initial target template images [m1, m2,..., mn] contained in the image M. Step S7, according to the target detection result Y1 obtained in step S5, a larger region around the target position determined in the multi-modal fusion image with a width of W and a height of H is intercepted as a search region of the tracking target, and the search region is recorded as a set Y3, and the set Y3 contains a series of search region images [n1, n2,..., nn] corresponding to the template image set Y2. corresponding to the template image set Y2.​ Step S8, inputting the template image set Y2 obtained in step S7 and the search area image set Y3 into the two convolutional neural network branches of the SimaFc-based airborne target tracking model constructed in step S4 for feature extraction, using similarity and distance measurement to determine the part in the search area that is most similar to the template image, thereby realizing target tracking.

2. The method of claim 1, wherein, The enhanced dual-scale image fusion algorithm in step S2 comprises the following steps: Step S21, input two images, one is a visible light image, and the other is a corresponding infrared image, which are respectively represented as I and I and respectively, and use bilateral filtering to respectively decompose the two images into a base layer and a detail layer. Step S22, fusing the decomposed base layer and detail layer respectively, wherein the base layer is fused by using a weighted average method, and the detail layer is fused by using a fusion coefficient matrix constructed through a significant feature map for weighted average fusion; Step S23, after obtaining the fused base layer and detail layer, the base layer and the detail layer are fused by simple addition, that is, the channel elements corresponding to the visible light image and the infrared image are subjected to concat operation to obtain the final multi-modal fusion data M.

3. The method of claim 1, wherein, The multi-modal fusion target detection model based on Yolov5 constructed in step S3 is an improved target detection model, comprising: Step S31, constructing a backbone network: composed of Darknet combined with a CSP structure, extracting features at different levels through layer-by-layer convolution, activation, and batch normalization operations, and providing rich low-level and high-level features for subsequent detection modules; Step S32, constructing a feature fusion layer Neck: using a path aggregation network structure, combining a feature pyramid network and a path aggregation network, and through multi-layer feature fusion, the model is more suitable for detecting targets of different sizes, and the recognition ability of small targets and distant targets is enhanced; Step S33, constructing a detection head Head: different Anchor boxes are generated for each feature layer, and the probability of the target class and the boundary box position are calculated for each Anchor, three scales are used to predict large, medium and small targets respectively, the generated feature map specific boundary box information includes position, size, confidence and class, through the non-maximum suppression process, overlapping boxes are removed and the prediction box with the highest confidence is retained, and finally the detection result is obtained; wherein the C3-RS module is used to replace the C3 module in the original structure, R refers to the Bottleneck residual block in ResNet, and the SE module is combined to make the channel attention more concentrated and improve the recognition ability of small targets.

4. The method of claim 1, wherein, The airborne target tracking model based on SimaFc constructed in step S4 is an improved target tracking model, which comprises: Step S41, constructing a backbone network Backbone: based on MobileNet, composed of two convolutional neural network branches sharing weights, extracting features from the template image and the search area respectively; Step S42, constructing a similarity measurement module: matching the template features and the search area features through the cross-correlation layer to measure the similarity of the two; Step S43, constructing a response map generation module: generating a response map, i.e. a similarity distribution map, from the cross-correlation result, locating the most likely position of the target by comparing the response values at different positions in the response map, and the peak position in the response map, also called the response maximum, is the position prediction of the target; Step S44, constructing a occlusion detection module: by calculating the change pattern in the response map, when a large drop or blurring in similarity is detected, it is determined whether the target is occluded; when occlusion is determined to occur, the occlusion detection module triggers a repositioning strategy to reposition the target through the response map of the previous frame and the historical trajectory of the target; Step S45, constructing a target positioning and updating module, adjusting the boundary box position of the target according to the peak position of the response map, and updating the model based on the confidence; Step S46, constructing a post-processing module: smoothing and screening operations are performed to improve the accuracy and stability of tracking in complex environments.

5. The method of claim 1, wherein, In step S3, the multi-modal fusion target detection model based on Yolov5 is pruned and quantized through a model lightweight method based on joint channel pruning and layer pruning, comprising the following steps: S3a, channel pruning: a pruning script is defined, and basic channel pruning is implemented in models / common.py in the Yolov5 code framework; S3b, layer pruning: the contribution of a layer is evaluated by measuring the importance of the layer, and the layer with low contribution is pruned first; layer pruning is realized based on the sparsity of the feature map, by counting the proportion or average absolute value of non-zero values in the activation output, finding the layer with high sparsity, i.e. the feature map output of the layer is very sparse or close to zero, indicating that the role of this layer is small and it is a layer that needs to be pruned; S3c, model fine-tuning: after channel pruning and layer pruning operations, the model is fine-tuned respectively to restore the accuracy of the model; S3d, model quantization: quantize the weight in floating-point number FP32 format to integer INT8 format, and input a small part of real scene samples as calibration data into the model, collect the maximum and minimum value information of the activation layer, and then according to the sampling range, quantize the activation and weight of each layer in the model to INT8 representation.

6. The method of claim 1, wherein, In step S4, the SimaFc-based airborne terminal target tracking model is pruned and quantized by the model lightweight method based on joint channel pruning and layer pruning, including the following steps: Step S4a, channel pruning: by calculating the batch normalization layer scaling factor γ parameter of each channel, the scaling factor represents the activation amplitude of the channel, and the importance of the channel is judged by analyzing these values; wherein, the greater the γ value, the more important the features of the channel, and the smaller the channel, the smaller the influence of the channel on the features in the model, and the channel with lower weight is selected as the pruning candidate, and a certain proportion of channels are reserved based on the set pruning rate, and the total number of channels is reserved 80%, after pruning, the number of channels of the model is adjusted to ensure that the input dimension and output dimension of the downstream convolution layer are consistent; Step S4b, layer pruning: evaluate the influence of deleting layers on model accuracy layer by layer, respectively perform sensitivity analysis on each convolution layer of MobileNet, and record the influence of each convolution layer on model accuracy; use the validation set to perform fast inference test, get the model accuracy, and if the influence of a certain layer on the accuracy is small, mark it as a pruning candidate; the influence on the accuracy is determined by setting the accuracy threshold, and if the accuracy decreases by less than 1% after removing a certain layer, it is considered that the layer can be removed; according to the results of sensitivity analysis, redundant layers are selected from the deep convolution layer with the smallest influence for pruning; Step S4c, model fine-tuning: after the channel pruning and layer pruning operations, the model is fine-tuned respectively to restore the accuracy of the model; Step S4d, model quantization: quantize the weight in floating-point number FP32 format to integer INT8 format, and input a small part of real scene samples as calibration data into the model, collect the maximum and minimum value information of the activation layer, and then according to the sampling range, quantize the activation and weight of each layer in the model to INT8 representation.

Images