An infrared small target detection method for air-to-ground application

Abstract

The application discloses an infrared small target detection method for air-ground application, and belongs to the field of infrared small target detection, comprising: acquiring a data set, and updating anchor boxes of infrared small targets based on the data set; constructing a GE module based on a GE attention mechanism, embedding the GE module into a backbone feature extraction network of a YOLOv7 model; constructing an NWD module based on NWD metrics, integrating the NWD module into a loss function of the YOLOv7 model to obtain an ISTD-YOLOv7 model; training and verifying the ISTD-YOLOv7 model based on the data set; and detecting infrared small targets through the trained ISTD-YOLOv7 model based on the updated anchor boxes. The infrared small target detection method provided by the application has superiority in infrared small target detection, and can be applied to complex time and weather conditions and terrain scenes to accurately and quickly realize infrared small target detection.

Description

Technical Field

[0001] This application relates to an infrared small target detection method for air-to-ground applications, belonging to the field of infrared small target detection. Background Technology

[0002] Infrared detection technology is one of the main means of acquiring modern information. Compared with visible light detection systems, infrared detection systems have advantages such as strong penetration, long detection range, and all-weather visibility. Therefore, infrared detection technology is receiving increasing attention from researchers and is widely used in military, medical, meteorological, and other fields. With the gradual opening of low-altitude airspace, unmanned aerial vehicles (UAVs) can carry infrared equipment to collect and track ground targets. How to effectively detect small targets from an aerial perspective not only has significant theoretical and engineering requirements but also has important social and economic implications.

[0003] In recent years, with the rapid development of deep learning technology, object detection methods have shifted from traditional methods based on manually designed features to deep neural network methods based on automatically learned features. Deep learning-based object detection methods are generally divided into two-stage and single-stage methods. Two-stage methods generate proposal regions and classify them. Classic models are the R-CNN series, including Fast R-CNN, Faster R-CNN, Mask R-CNN, etc. They have high detection accuracy but slow detection speed, making them difficult to apply to real-time detection scenarios. Single-stage methods do not have a proposal region generation stage. They directly produce the final detection result in a single stage, thus having a faster detection speed. Classic models are the YOLO series, including YOLOv3, YOLOv5, YOLOX, etc.

[0004] YOLOv7 is a new model in the YOLO series, surpassing most known object detectors in both accuracy and speed. Since 2022, YOLOv7 has been applied to several real-world detection tasks. Soeb et al. created a dataset of leaf images from Bangladesh and used YOLOv7 for disease diagnosis. This research provides a solution for precision agriculture applications. Li et al. improved YOLOv7 by embedding Gamma correction, an improved attention mechanism module, and Alpha GIOU. The improved model was used for damage detection on aero-engine blades. Abnormal driver behavior seriously threatens public safety. Liu et al. proposed a CEAMYOLOv7 model for distracted behavior recognition. A global attention mechanism (GAM) was introduced into YOLOv7 to enhance the network's ability to extract key features.

[0005] Despite the excellent performance of these models in the aforementioned related works, infrared small target detection remains a challenge. On the one hand, infrared small targets are characterized by their small size and weak signal; due to the long observation distance, their shape and texture information is scarce. On the other hand, due to complex backgrounds, infrared small targets may overlap or be occluded. Considering these issues, some researchers have made progress. Liu et al. proposed a lightweight model for ship detection in SAR images. They added a Coordinated Attention (CA) mechanism to the backbone network of YOLOv7-tiny and improved the SPP module and loss function. Compared to the original model, the accuracy was improved by 4.6%. However, this work has not yet been deployed on edge devices. Similarly, Guo et al. proposed a lightweight SAR ship target detection method based on YOLO, called LMSD-YOLO. This model has good multi-scale adaptability and has been successfully deployed on mobile platforms. However, LMSD-YOLO still struggles to directly achieve target detection from large scene images. Zhou et al. improved YOLOv5, enabling the model to perform small target detection tasks. Notably, the authors used a super-resolution generative adversarial network (SRGAN) to generate super-resolution images and fed them into an improved detection model. Experiments demonstrate that super-resolution reconstruction of the input image can improve the detection accuracy of small targets. The drawback is that the super-resolution reconstruction process is very time-consuming.

[0006] In summary, infrared small targets are extremely faint and have a very low signal-to-noise ratio. In complex weather and terrain conditions, infrared small targets are easily overlooked, and most current models are unable to effectively detect them. Summary of the Invention

[0007] The purpose of this application is to provide an infrared small target detection method for air-to-ground applications. It proposes an infrared small target detection model ISTD-YOLOv7 with good adaptability, which can accurately and quickly achieve target detection in complex weather and terrain scenarios.

[0008] To achieve the above objectives, the first aspect of this application provides an infrared small target detection method for air-to-ground applications, comprising:

[0009] Obtain the dataset and update the anchor boxes of small infrared targets based on the dataset;

[0010] A GE module is constructed based on the GE attention mechanism, and the GE module is embedded into the backbone feature extraction network of the YOLOv7 model.

[0011] An NWD module is constructed based on the NWD metric, and the NWD module is integrated into the loss function of the YOLOv7 model to obtain the ISTD-YOLOv7 model.

[0012] The ISTD-YOLOv7 model was trained and validated based on the dataset.

[0013] Based on the updated anchor boxes, infrared small targets are detected using the trained ISTD-YOLOv7 model.

[0014] In one implementation, after acquiring the dataset, the process further includes:

[0015] The dataset is randomly augmented using data augmentation techniques.

[0016] In one implementation, updating the anchor boxes of small infrared targets based on the dataset includes:

[0017] The size of small infrared targets is re-clustered on the dataset to update the anchor boxes.

[0018] In one implementation, the GE module includes:

[0019] ξ is used to extract features from the local spatial location of the input data. G Operators and ξ used to map the extracted features to the original scale of the input data E Operators.

[0020] In one implementation, embedding the GE module into the backbone feature extraction network of the YOLOv7 model includes:

[0021] Add three GE modules to the three output branches of the backbone feature extraction network in the YOLOv7 model.

[0022] In one implementation, integrating the NWD module into the loss function of the YOLOv7 model includes:

[0023] The IoU metric in the original loss function of the YOLOv7 model is replaced by the NWD module.

[0024] In one implementation, the ISTD-YOLOv7 model includes: a backbone feature extraction network, a neck network, and a head network;

[0025] The infrared small target detection based on the updated anchor boxes using the trained ISTD-YOLOv7 model includes:

[0026] The system extracts features from the input data using a backbone feature extraction network embedded with GE modules, and mines the contextual and spatial information of the extracted features.

[0027] The extracted features are fused using the neck network to obtain spatial and semantic information;

[0028] Based on the spatial and semantic information, a head network is used to make predictions and obtain prediction results.

[0029] The prediction results are iteratively optimized using a loss function that integrates an NWD module;

[0030] The bounding box is adjusted based on the updated anchor box, and the final detection result is obtained through nonmaximum suppression.

[0031] A second aspect of this application provides an infrared small target detection device for air-to-ground applications, comprising:

[0032] An acquisition unit is used to acquire a dataset and update the anchor frame of the infrared small target based on the dataset;

[0033] An improved unit is used to construct a GE module based on the GE attention mechanism and embed the GE module into the backbone feature extraction network of the YOLOv7 model.

[0034] The improved unit is also used to construct an NWD module based on the NWD metric, and integrate the NWD module into the loss function of the YOLOv7 model to obtain the ISTD-YOLOv7 model.

[0035] The training unit is used to train and validate the ISTD-YOLOv7 model based on the dataset.

[0036] The detection unit is used to detect small infrared targets based on the updated anchor boxes using the trained ISTD-YOLOv7 model.

[0037] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the first aspect or any embodiment of the first aspect.

[0038] The fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the first aspect or any embodiment of the first aspect.

[0039] As shown above, this application provides an infrared small target detection method for air-to-ground applications, proposing a network model ISTD-YOLOv7 with high infrared small target detection performance. Firstly, anchor box updating in YOLOv7 improves network model convergence speed and efficiency. Then, the GE attention mechanism is added to YOLOv7, enabling the network model to more effectively mine contextual and spatial information of features. Furthermore, the original IoU metric in the YOLOv7 loss function is replaced with the NWD metric to overcome YOLOv7's sensitivity to small target positional biases. ISTD-YOLOv7 demonstrates superior infrared small target detection performance compared to existing models such as YOLOv3, YOLOv5s, SSD, CenterNet, FCOS, and YOLOv7. It can be applied to infrared small target detection in complex weather and terrain scenarios. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A schematic diagram of the structure of an ISTD-YOLOv7 model provided in an embodiment of this application;

[0042] Figure 2 This is a schematic diagram illustrating the result of dataset clustering provided in an embodiment of this application;

[0043] Figure 3 A schematic diagram of the structure of a GE module provided in an embodiment of this application;

[0044] Figure 4 A schematic diagram illustrating the location of a GE module in a YOLOv7 model, provided as an embodiment of this application;

[0045] Figure 5 A sensitivity analysis chart of IoU provided for embodiments of this application;

[0046] Figure 6 This application provides illustrations of different scenarios for its embodiments.

[0047] Figure 7 A data-enhanced result diagram provided in an embodiment of this application;

[0048] Figure 8 A schematic diagram of test set information provided in an embodiment of this application;

[0049] Figure 9 A schematic diagram of AP values ​​for each category of different models provided in this application embodiment;

[0050] Figure 10 This is a schematic diagram of a detection result provided in an embodiment of this application. Detailed Implementation

[0051] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0052] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0053] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0055] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0056] YOLOv7, one of the latest representative models in the YOLO series, was proposed by Wang et al. in 2022. The YOLOv7 model first resizes the input image to 640×640 pixels. Then, the image is fed into the backbone network for feature extraction. The YOLOv7 backbone network consists of multiple CBS modules, E-ELAN modules, MP modules, and one SPPCSPC module. The obtained features are fused using PAFPN. The neck network aims to fuse feature information at different scales. The prediction network adjusts the number of channels based on the REP and CBM modules. Finally, it outputs bounding box information, confidence scores, and class probabilities.

[0057] Example 1

[0058] This application provides an infrared small target detection method for air-to-ground applications, such as... Figure 1 As shown, the method includes:

[0059] S100 acquires the dataset and updates the anchor frame of the infrared small target based on the dataset;

[0060] Optionally, the dataset includes infrared images under several different complex environments. Each infrared image can be obtained by taking pictures with a drone equipped with an infrared camera, or it can be obtained directly based on existing publicly available data. No limitation is made here.

[0061] Optionally, after obtaining the dataset, the method further includes: randomly augmenting the dataset using data augmentation techniques.

[0062] In one implementation, random augmentation is performed using data augmentation methods such as Mixup and Mosaic to greatly enrich the training data, improve the model's generalization ability, and enhance the network's robustness.

[0063] Optionally, updating the anchor boxes of small infrared targets based on the dataset includes: re-clustering the sizes of the small infrared targets on the dataset to update the anchor boxes.

[0064] In one implementation, the anchor box size is obtained by clustering the width and height of the ground truth boxes of the training samples. The appropriateness of the anchor boxes significantly impacts the model's detection performance. Generally, YOLOv7 anchor boxes are obtained during training based on clustering of the COCO or VOC datasets. The target sizes in these datasets differ significantly from those in the infrared small target dataset. In this embodiment, to improve and accelerate YOLOv7 convergence, the target sizes are re-clustered on the selected dataset. Figure 2The clustering results for the VOC dataset and the selected dataset are shown. It can be seen that the distribution of cluster centers differs significantly. The target size in the VOC dataset can reach several hundred pixels, while the target size in the selected dataset is clearly much smaller. Updating the anchor boxes can provide a more reasonable prior for the detection model.

[0065] S200 constructs a GE module based on the Gather-Excite (GE) attention mechanism and embeds the GE module into the backbone feature extraction network of the YOLOv7 model;

[0066] Optionally, the GE attention mechanism fully utilizes the contextual and spatial information of the feature space and improves the network's representational ability. The GE module includes: ξ for extracting features from the local spatial location of the input data. G Operators and ξ used to map the extracted features to the original scale of the input data E Operators.

[0067] In one implementation, Figure 3 The operation flow of two operators in the GE module is shown. ξ G Features are extracted from local spatial locations, as defined below:

[0068] ξ G :R H×W×C →R H’×W’×C

[0069] Where H, W, and C represent the height, width, and number of channels of any input x, respectively, and e represents the range ratio, H' = H / e, W' = W / e.

[0070] ξ E The extracted features are mapped back to the original scale, as defined below:

[0071]

[0072] f:R H’×W’×C →[0,1] H×W×C

[0073] in, Indicates passing through ξ G The processed output shows that ⊙ represents the Hadamard product and f represents the mapping relationship.

[0074] Optionally, embedding the GE module into the backbone feature extraction network of the YOLOv7 model includes adding three GE modules to each of the three output branches of the backbone feature extraction network of the YOLOv7 model. The specific addition locations are as follows: Figure 4As shown, small infrared targets are characterized by their small size and weak signal. Therefore, location information is crucial for their detection. By adding a GE module to the backbone feature extraction network of YOLOv7, the model can more effectively extract contextual and spatial location information of infrared small target features.

[0075] S300 constructs an NWD module based on the Normalized Wasserstein Distance (NWD) metric, and integrates the NWD module into the loss function of the YOLOv7 model to obtain the ISTD-YOLOv7 model;

[0076] Optionally, because the IoU metric is highly sensitive to targets of different scales, for smaller targets, slight changes in position can lead to significant changes in IoU. However, for normal-sized targets, the change in IoU is very small for the same positional deviation. Figure 5 A specific analysis is provided. For small targets, positional deviation causes the IoU to decrease from 0.47 to 0.02. However, for normal targets, the same positional deviation only causes the IoU to decrease from 0.83 to 0.49. Therefore, this application adopts a metric method based on Wasserstein distance. Specifically, the bounding boxes are modeled as two-dimensional Gaussian distributions, and then the proposed metric, namely the NWD metric, is used to calculate the similarity between the corresponding Gaussian distributions. The advantage of NWD is that it is more suitable for measuring the similarity between small targets. In addition, distribution similarity can measure whether there is overlap between small targets.

[0077] Specifically, for a bounding box (cx, cy, w, h), its inscribed ellipse can be represented as:

[0078]

[0079] Where (cx, cy), w, and h represent the center coordinates, width, and height of the bounding box, respectively. (μ x ,μ y ), σ x and σ y Let μ represent the coordinates of the ellipse's center, the length of the X-axis, and the length of the Y-axis, respectively. Therefore, μ x =cx,μ y =cy,σ x =w / 2, σ y = h / 2. The probability density function of a two-dimensional Gaussian distribution is:

[0080]

[0081] Where x, μ, and Σ represent the coordinates (x, y), the mean, and the covariance. (x-μ) T Σ-1 When (x-μ)=1, it means that the bounding box can be modeled as a 2D Gaussian distribution N(μ,Σ):

[0082]

[0083] Where N a and N b Corresponding to the bounding box (cx) a ,cy a ,w a ,h a ) and (cx b ,cy b ,w b ,h b The Gaussian distribution modeled is shown below, with the Wasserstein distance as follows:

[0084]

[0085] The final form of the NWD metric after normalization is shown below:

[0086]

[0087] Optionally, integrating the NWD module into the loss function of the YOLOv7 model includes replacing the IoU metric in the original loss function of the YOLOv7 model with the NWD module. Using an NWD-based regression loss function not only addresses the sensitivity of YOLOv7 to small target location biases but can also provide gradients for network optimization in certain situations.

[0088] The improved loss function of the ISTD-YOLOv7 model is as follows:

[0089] L ISTD-YOLOv7 =1-NWD(N p N g )

[0090] Where NWD represents the NWD metric, N p and N g Let p and g represent the Gaussian distribution models of the predicted box and the ground truth box, respectively.

[0091] S400 trains and validates the ISTD-YOLOv7 model based on the dataset;

[0092] Optionally, the dataset can be randomly divided into training, validation, and test sets in an 8:1:1 ratio. The ISTD-YOLOv7 model can be trained and validated using the training and validation sets, and its performance can be evaluated using the test set.

[0093] The S500 detects small infrared targets based on updated anchor boxes using the trained ISTD-YOLOv7 model.

[0094] Optional, such as Figure 1 As shown, the ISTD-YOLOv7 model includes: a backbone feature extraction network, a neck network, and a head network;

[0095] The infrared small target detection based on the updated anchor box and the trained ISTD-YOLOv7 model includes: input data (infrared image) enters the backbone feature extraction network composed of convolutional modules, the backbone feature extraction network with embedded GE module extracts features from the input data, and the extracted features enter the GE module to mine context information and spatial location information.

[0096] The extracted features are fused using a neck network with PAFPN to generate better spatial and semantic information. After adjusting the number of channels in the network using RepConv, the fused features are fed into the head network for prediction, yielding the prediction results.

[0097] Through the loss function L integrated with the NWD module ISTD-YOLOv7 The prediction results are iteratively optimized;

[0098] The bounding box is adjusted based on the updated anchor box, and the final detection result Ouput is obtained through nonmaximum suppression (NMS).

[0099] As can be seen from the above, this application provides an infrared small target detection method for air-to-ground applications, and proposes a network model ISTD-YOLOv7 with high infrared small target detection performance. Firstly, anchor box updating of YOLOv7 improves the network model's convergence speed and efficiency. Then, the addition of a GE attention mechanism to YOLOv7 allows the network model to more effectively mine the contextual and spatial information of features. Furthermore, the original IoU metric in the YOLOv7 loss function is replaced with an NWD metric to overcome YOLOv7's sensitivity to small target positional biases. ISTD-YOLOv7 demonstrates superior infrared small target detection performance compared to existing models such as YOLOv3, YOLOv5s, SSD, CenterNet, FCOS, and YOLOv7, showcasing its advantages in infrared small target detection.

[0100] Example 2

[0101] This application demonstrates the effectiveness of the infrared small target detection method provided in Embodiment 1 through experiments. The specific experimental process is as follows:

[0102] Experimental platform:

[0103] All experiments were run on a computer equipped with an Intel(R) Core(TM) i9-12900KF (64GB DDR5) CPU, an NVIDIA GeForce RTX 3090Ti (24GB) GPU, and Microsoft Windows 10. The deep learning framework used was PyTorch 1.7.1. A stochastic gradient descent optimizer (SGD) with an initial learning rate of 0.01, weight decay of 0.0005, and momentum of 0.937 was chosen to reduce the loss function. The batch size was 32, and the epoch size was 300.

[0104] Dataset introduction:

[0105] All images in the dataset of this application embodiment were taken by drones equipped with infrared cameras. The dataset includes 21,750 images, 8 categories, and 89,174 targets, where the small infrared targets are vehicles against a ground background. Table 1 provides further details of the dataset. Examples of this dataset are shown below. Figure 6 As shown, the training, validation, and test sets were randomly divided in an 8:1:1 ratio. The main challenges of this dataset lie in complex environmental interference and complex imaging conditions. This dataset provides a material basis for the study of infrared image characteristics and the detection and tracking of small infrared targets.

[0106] Table 1. Dataset Details

[0107] Resolution Depth Format Memory (640×480) pixels 8bit .bmp ≈300k

[0108] Evaluation Indicators Introduction:

[0109] This application selects five evaluation metrics to evaluate the detection performance of the model: Precision, Recall, F1 score, Average Precision, and Mean Average Precision. The equations for each evaluation metric are as follows:

[0110]

[0111]

[0112]

[0113]

[0114]

[0115] All evaluation metrics are within the range of [0,1], with higher values ​​indicating better results. TP represents true positives, FP represents false positives, and FN represents false negatives. The confusion matrix is ​​shown in Table 2. C represents the number of classes. P represents Precision, R represents Recall, F1 represents the F1 score, AP represents Average Precision, and mAP represents Average Precision. mAP is the average AP across all classes, used to evaluate the overall detection accuracy of the model.

[0116] Table 2. Confusion Matrix

[0117] Predicted result = Positive Predicted result = Negative Actual result = True TP (True Positive) FN(False Negative) Actual result = False FP (False Positive) TN (True Negative)

[0118] Analysis of the comparison results with the YOLOv7 model:

[0119] This application compares the performance of ISTD-YOLOv7 and YOLOv7. Before training both models, data augmentation techniques are used to randomly augment the data. Mixup and Mosaic data augmentation methods are used as examples. Figure 7 These are the infrared image results processed by the two methods. Mixup performs simple linear interpolation on two randomly selected infrared images to construct new training samples, such as... Figure 7 As shown in (a)-(d), Mosaic randomly selects four infrared images and merges them into a single infrared image as new training data, as follows. Figure 7 As shown in (e)-(h).

[0120] The performance of the two models was evaluated on a test set. The test set contained 2,175 infrared small target images. The number of targets in each category was as follows: Figure 8 As shown in Table 3, the evaluation results of the two models on the test set are compared. The best results in the embodiments of this application are marked in bold. As can be seen from Table 3, ISTD-YOLOv7 has improved over YOLOv7 in terms of accuracy (from 97.52% to 98.80%), recall (from 96.23% to 96.87%), F1 (from 96.87% to 97.83%), and mAP (from 97.44% to 98.43%). The application of the improved strategy enhances the network's ability to extract features from limited information, reduces the false negative rate of small infrared targets, improves the model's recall, and makes the detection of ISTD-YOLOv7 more accurate.

[0121] Table 3. Evaluation results of YOLOv7 and ISTD-YOLOv7

[0122] Model P(％) R(％) F1(％) mAP(%) YOLOv7 97.52 96.23 96.87 97.44 ISTD-YOLOv7 98.80 96.87 97.83 98.43

[0123] Analysis of comparison results with other models:

[0124] This application also compares ISTD-YOLOv7 with other advanced detection models. YOLOv3 and YOLOv5s also belong to the YOLO family, but have not yet been tested on datasets. SSD is an anchor-box based model. CenterNet and FCOS are anchorless models.

[0125] Figure 9 The AP values ​​for each class of different models are shown. The AP metric comprehensively considers the balance between precision and recall at different confidence levels. ISTD-YOLOv7 is the only model with an AP value exceeding 96% for all classes. The results indicate that this model has good overall detection performance on the given dataset. Furthermore, it is easy to see that, except for FCOS, the AP values ​​for class 8 of other models are the lowest. This is because there are fewer targets in class 8 in the training set, and the model cannot learn the feature information of this class more fully. However, the ISTD-YOLOv7 model in this embodiment of the application has an AP value exceeding 96% for class 8, while the SSD model only exceeds 75% for class 8.

[0126] Table 4 provides further quantitative results. In terms of accuracy, ISTD-YOLOv7 achieved the best result of 98.80%. YOLOv3 had the highest recall rate at 97.45%, followed by ISTD-YOLOv7. F1 and mAP are two comprehensive metrics, and the ISTD-YOLOv7 model significantly outperformed the comparison models. On the given dataset, SSD's performance was unsatisfactory. These results indicate that ISTD-YOLOv7 generally outperforms the comparison models in its ability to detect small infrared targets. When dealing with small infrared targets in complex scenes, the anchor box update, GE attention mechanism, and NWD-based loss function in ISTD-YOLOv7 significantly improve the network's convergence performance and feature extraction ability, alleviate the model's sensitivity to small target positional biases, and reduce the false negative rate.

[0127] Table 4. Evaluation results of different models

[0128]

[0129]

[0130] The qualitative results of ISTD-YOLOv7 were compared with those of other models, such as... Figure 10 As shown. In Figure 10Among the models tested, only CenterNet and ISTD-YOLOv7 were able to detect all targets, while other models exhibited varying degrees of missed detections. Further analysis revealed that "Target 7" was extremely faint, almost completely obscured by the background, making it even more difficult to detect. Even in this case, ISTD-YOLOv7 could still detect it with a confidence score of 0.78. ISTD-YOLOv7 had the most severe missed detection issue, only detecting "Target 1".

[0131] As can be seen from the above, the experiments show that the model proposed in this application has stronger detection performance than YOLOv3, YOLOv5s, SSD, CenterNet, FCOS and the baseline model. Compared with the baseline model, the mAP of ISTD-YOLOv7 is improved from 97.44% to 98.43%.

[0132] Example 3

[0133] This application provides an infrared small target detection device for air-to-ground applications, comprising: an acquisition unit, an improvement unit, a training unit, and a detection unit.

[0134] The acquisition unit is used to: acquire a dataset and update the anchor frame of the infrared small target based on the dataset;

[0135] The improved unit is used to: construct a GE module based on the GE attention mechanism, and embed the GE module into the backbone feature extraction network of the YOLOv7 model;

[0136] The improved unit is also used to: construct an NWD module based on the NWD metric, and integrate the NWD module into the loss function of the YOLOv7 model to obtain the ISTD-YOLOv7 model;

[0137] The training unit is used to: train and validate the ISTD-YOLOv7 model based on the dataset;

[0138] The detection unit is used to detect small infrared targets based on the updated anchor boxes and the trained ISTD-YOLOv7 model.

[0139] As can be seen from the above, the infrared small target detection device provided in this application update the anchor boxes of YOLOv7 through the acquisition unit to make the network model converge better and faster; then, the GE attention mechanism is added to YOLOv7 through the improvement unit to enable the network model to more effectively mine the contextual information and spatial location information of the features; in addition, the original IoU metric in the YOLOv7 loss function is replaced with the NWD metric through the improvement unit to overcome the problem that YOLOv7 is sensitive to the positional deviation of small targets.

[0140] Example 4

[0141] This application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory stores software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory and processor are connected via a bus. Specifically, the processor implements any of the steps in Embodiment 1 by running the computer program stored in the memory.

[0142] It should be understood that, in the embodiments of this application, the processor may be a Central Processing Unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0143] Memory may include read-only memory, flash memory, and random access memory, and provides instructions and data to the processor. Some or all of the memory may also include non-volatile random access memory.

[0144] As can be seen from the above, the electronic device provided in this application implements the infrared small target detection method as described in Embodiment 1 by running a computer program. Updating the anchor boxes in YOLOv7 allows the network model to converge better and faster. Then, adding the GE attention mechanism to YOLOv7 enables the network model to more effectively mine the contextual and spatial information of features. Furthermore, replacing the original IoU metric in the YOLOv7 loss function with the NWD metric overcomes the sensitivity of YOLOv7 to small target positional bias. The ISTD-YOLOv7 exhibits stronger detection performance on infrared small targets compared to other existing models such as YOLOv3, YOLOv5s, SSD, CenterNet, FCOS, and YOLOv7, demonstrating superiority in infrared small target detection.

[0145] It should be understood that if the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0146] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0147] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0148] It should be noted that the methods and detailed examples provided in the above embodiments can be incorporated into the apparatus and devices provided in the embodiments, and can be referred to each other, without further elaboration.

[0149] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0150] In the embodiments provided in this application, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of the modules or units described above is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0151] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for detecting small infrared targets for air-to-ground applications, characterized in that, include: Obtain the dataset and update the anchor boxes of small infrared targets based on the dataset; A GE module is constructed based on the GE attention mechanism, and the GE module is embedded into the backbone feature extraction network of the YOLOv7 model. An NWD module is constructed based on the NWD metric, and the NWD module is integrated into the loss function of the YOLOv7 model to obtain the ISTD-YOLOv7 model. The ISTD-YOLOv7 model was trained and validated based on the dataset. Based on the updated anchor boxes, infrared small targets are detected using the trained ISTD-YOLOv7 model; The step of updating the anchor boxes of small infrared targets based on the dataset includes: The size of small infrared targets is re-clustered on the dataset to update the anchor boxes; The step of embedding the GE module into the backbone feature extraction network of the YOLOv7 model includes adding three GE modules to the three output branches of the backbone feature extraction network of the YOLOv7 model, respectively. The size of the anchor box is obtained by clustering the width and height of the ground truth boxes of the training samples of the ISTD-YOLOv7 model; The updated anchor boxes provide a more reasonable prior for the trained ISTD-YOLOv7 model, enabling it to converge quickly.

2. The infrared small target detection method as described in claim 1, characterized in that, After obtaining the dataset, the process further includes: randomly augmenting the dataset using data augmentation techniques.

3. The infrared small target detection method as described in claim 1, characterized in that, The GE module includes: features for extracting features from the local spatial location of the input data. Operators and methods for mapping extracted features to the original scale of the input data Operators.

4. The infrared small target detection method according to any one of claims 1-3, characterized in that, The integration of the NWD module into the loss function of the YOLOv7 model includes replacing the IoU metric in the original loss function of the YOLOv7 model with the NWD module.

5. The infrared small target detection method according to any one of claims 1-3, characterized in that, The ISTD-YOLOv7 model includes: a backbone feature extraction network, a neck network, and a head network; The infrared small target detection based on the updated anchor boxes using the trained ISTD-YOLOv7 model includes: The system extracts features from the input data using a backbone feature extraction network embedded with GE modules, and mines the contextual and spatial information of the extracted features. The extracted features are fused using the neck network to obtain spatial and semantic information; Based on the spatial and semantic information, a head network is used to make predictions and obtain prediction results. The prediction results are iteratively optimized using a loss function integrated with an NWD module; the bounding boxes are adjusted based on the updated anchor boxes, and the final detection results are obtained through nonmaximum suppression.

6. An infrared small target detection device for air-to-ground applications, characterized in that, include: An acquisition unit is used to acquire a dataset and update the anchor frame of the infrared small target based on the dataset; An improved unit is used to construct a GE module based on the GE attention mechanism and embed the GE module into the backbone feature extraction network of the YOLOv7 model. The improved unit is also used to construct an NWD module based on the NWD metric, and integrate the NWD module into the loss function of the YOLOv7 model to obtain the ISTD-YOLOv7 model. The training unit is used to train and validate the ISTD-YOLOv7 model based on the dataset. A detection unit is used to detect small infrared targets based on the updated anchor boxes using the trained ISTD-YOLOv7 model. The step of updating the anchor boxes of small infrared targets based on the dataset includes: The size of small infrared targets is re-clustered on the dataset to update the anchor boxes; The step of embedding the GE module into the backbone feature extraction network of the YOLOv7 model includes adding three GE modules to the three output branches of the backbone feature extraction network of the YOLOv7 model, respectively. The size of the anchor box is obtained by clustering the width and height of the ground truth boxes of the training samples of the ISTD-YOLOv7 model; The updated anchor boxes provide a more reasonable prior for the trained ISTD-YOLOv7 model, enabling it to converge quickly.

7. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the steps of the method as claimed in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images