Air infrared weak and small target detection method based on space-frequency domain feature fusion

Abstract

The invention discloses an air infrared weak and small target detection method based on space-frequency domain feature fusion. The method comprises the following steps: acquiring a plurality of infrared weak and small target images under a complex background; dividing the infrared weak and small target images under the plurality of complex backgrounds according to a preset proportion to obtain a training set and a test set of the infrared weak and small target images under the plurality of complex backgrounds; and inputting the training set of the infrared weak and small target image under the complex background into the pre-constructed target detection network with the space-frequency domain feature fusion, and training the pre-constructed target detection network with the space-frequency domain feature fusion to obtain the successfully trained target detection network with the space-frequency domain feature fusion. According to the method, the technical problems of low detection precision and relatively high false alarm rate and omission ratio when an existing convolutional neural network is used for detecting an infrared weak and small target under a complex background are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a method for detecting small infrared targets in the air based on spatial-frequency domain feature fusion. Background Art

[0002] In order to solve the problem of infrared dim small target detection, a variety of traditional methods have been proposed, which are mainly divided into filtering methods, local contrast methods, and low-rank methods; however, these methods generally rely on prior knowledge and are sensitive to hyperparameters, which requires them to be closely matched with specific data sets in practical applications, limiting their versatility and flexibility; in addition, traditional mathematical modeling algorithms are complex in design and often cannot effectively cope with diverse practical scenarios, and their generalization capabilities are weak; these factors make traditional methods perform poorly in infrared dim small target detection tasks under complex backgrounds.

[0003] In recent years, with the development of deep learning technology, Convolutional Neural Networks (CNN) has emerged and gradually dominated the field of infrared weak target detection with its excellent feature extraction and model fitting capabilities. Through the training of a large amount of sample data, CNN can automatically learn and extract complex hierarchical features in the data, significantly improving the ability to distinguish between targets and backgrounds and the detection accuracy of weak targets. Although the detection algorithm based on deep learning has better adaptability, due to the small imaging area of ​​infrared weak targets, serious noise interference, motion blur and deformation, and the low sensitivity and disappearance of feature information when extracting features from images with low local signal-to-noise ratio, the detection of infrared weak targets under complex backgrounds still has difficulties such as low detection accuracy and high false alarm and missed detection rates. Therefore, it is urgent to conduct in-depth research on infrared weak target detection methods based on deep learning to improve their detection performance. Summary of the invention

[0004] The embodiment of the present invention provides an aerial infrared dim small target detection method based on space-frequency domain feature fusion, so as to at least solve the technical problems that the existing convolutional neural network detects infrared dim small targets in complex backgrounds, has low detection accuracy, and high false alarm and missed detection rates.

[0005] According to one aspect of an embodiment of the present invention, a method for detecting infrared small and weak targets in the air based on space-frequency domain feature fusion is provided. The method may include: obtaining a number of infrared small and weak target images under complex backgrounds; dividing the infrared small and weak target images under complex backgrounds according to a preset ratio to obtain a training set and a test set of infrared small and weak target images under complex backgrounds; inputting the training set of infrared small and weak target images under complex backgrounds into a pre-constructed target detection network with space-frequency domain feature fusion, training the pre-constructed target detection network with space-frequency domain feature fusion, and obtaining a successfully trained target detection network with space-frequency domain feature fusion, wherein the pre-constructed target detection network with space-frequency domain feature fusion includes an encoder module, a space-frequency dual-branch feature extraction module, an asymmetric space-frequency feature fusion decoder module, and a segmentation head module; inputting a test set of several infrared images in the air into the successfully trained target detection network with space-frequency domain feature fusion to obtain the segmentation result of the test set image.

[0006] Optionally, the encoder module includes four convolutional layers of different scales, and multi-scale extraction is performed on the training set images in sequence through the four convolutional layers of different scales to obtain four feature maps of different scales, wherein the four feature maps of different scales include a feature map of a first scale, a feature map of a second scale, a feature map of a third scale, and a feature map of a fourth scale.

[0007] Optionally, the space-frequency dual-branch feature extraction module includes a neck block, a spatial domain branch, a frequency domain branch, a connection module and a 1×1 convolution layer, the spatial domain branch includes a convolution module, a CBR module, a codebook module, a fusion module, a full connection and a 1×1 convolution layer, a channel-level multiplication module, and a channel-level addition module; the frequency domain branch includes a first normalization layer, a second normalization layer, a first self-attention mechanism, a second self-attention mechanism, a Fourier transform layer, a channel-level multiplication module, and a channel-level addition module; the feature map of the fourth scale is input into the neck block to obtain the initial feature; the initial feature is input into the spatial domain branch to obtain the spatial domain feature of the feature map of the fourth scale; the initial feature is input into the frequency domain branch to obtain the frequency domain feature of the feature map of the fourth scale; the spatial domain feature and the frequency domain feature are sequentially input into the connection module and the 1×1 convolution layer to obtain the target feature of the feature map of the fourth scale.

[0008] Optionally, the inputting the initial features into the spatial domain branch to obtain the spatial domain features of the feature map of the fourth scale includes: inputting the initial features into the convolution module and the CBR module in sequence to obtain the target feature map; inputting the target feature map into the codebook module to obtain the information of any codeword corresponding to the target feature map; inputting the information of all codewords into the fusion module to obtain the overall information of the target feature map for all codewords; inputting the overall information of the target feature map for all codewords into the fully connected layer and the 1×1 convolution layer in sequence to obtain the significant features of the target in the feature map of the fourth scale; after calculating the significant features of the target in the feature map of the fourth scale and the scale factor coefficient, they are input into the channel-level multiplication module together with the target feature map to obtain the local area features corresponding to the feature map of the fourth scale; the initial features and the local area features are input into the channel-level addition module to obtain the spatial domain features of the feature map of the fourth scale.

[0009] Optionally, the decoder module of the asymmetric space-frequency feature fusion includes four asymmetric space-frequency feature fusion modules of different scales and one U-shaped dilated convolution, wherein the four asymmetric space-frequency feature fusion modules of different scales include an asymmetric space-frequency feature fusion module of a first scale, an asymmetric space-frequency feature fusion module of a second scale, an asymmetric space-frequency feature fusion module of a third scale and an asymmetric space-frequency feature fusion module of a fourth scale; the target features and feature maps of the feature map of the fourth scale are input into the asymmetric space-frequency feature fusion module of the first scale to obtain a fused feature map of the first scale; the fused feature map of the first scale is input into the U-shaped dilated convolution module to obtain a target fine feature map; the target fine feature map and the feature map of the third scale are input into the asymmetric space-frequency feature fusion module of the second scale to obtain a fine fused feature map of the second scale; the fine fused feature map of the second scale and the feature map of the second scale are input into the asymmetric space-frequency feature fusion module of the third scale to obtain a fine fused feature map of the third scale; the fine fused feature map of the second scale and the feature map of the first scale are input into the asymmetric space-frequency feature fusion module of the fourth scale to obtain a fine fused feature map of the fourth scale.

[0010] Optionally, the fourth-scale fine fusion feature map is input into a segmentation head module to obtain a small target of an infrared small target image under a complex background.

[0011] Beneficial effects of the present invention: (1) The present invention introduces a U-shaped network structure, which achieves an efficient balance between feature extraction and feature fusion through the symmetrical design of the encoder and decoder. In the feature extraction stage, the encoder gradually extracts the deep semantic information in the image through layer-by-layer convolution and pooling operations; in the feature fusion stage, the decoder combines the deep semantic information with the shallow texture information through layer-by-layer upsampling and convolution operations to achieve comprehensive information fusion; this symmetrical structure not only helps to maintain the consistency of the spatial information and semantic information of the image, but also significantly improves the accuracy of the network; through the U-shaped network structure, the key information in the image can be more effectively extracted and utilized, providing a solid foundation for subsequent image processing tasks.

[0012] (2) The present invention introduces a frequency domain processing method of fast Fourier transform in a convolutional neural network. The image is converted from the spatial domain to the frequency domain through FFT, and the frequency information in the frequency domain is used to effectively filter the image. This method can more accurately capture the detailed information in the image, especially the feature information of small targets. At the same time, the frequency domain processing method can also effectively suppress noise and interference and improve the contrast of the image. Therefore, the combination of spatial domain feature and frequency domain feature processing can significantly improve the network's detection capability for small targets, providing new ideas and methods for applications in the fields of image processing and computer vision.

[0013] (3) The present invention adopts an asymmetric fusion strategy, which achieves information complementarity and enhancement by effectively fusing shallow texture information and deep semantic information. In the feature fusion process, the asymmetric fusion strategy can flexibly adjust the fusion method and weight according to the characteristics of features at different levels, so as to obtain a more accurate and comprehensive feature representation. This strategy not only improves the detection performance of the network, but also enhances the robustness and generalization ability of the network. Through the asymmetric fusion strategy, the target object in the image can be more accurately identified, providing strong support for tasks such as image classification and target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 is a flow chart of a method for detecting small infrared targets in the air based on spatial-frequency domain feature fusion according to an embodiment of the present invention; Figure 2 is a schematic diagram of a target detection network with pre-constructed spatial-frequency domain feature fusion according to an embodiment of the present invention; Figure 3 is a schematic diagram of a space-frequency dual-branch feature extraction module according to an embodiment of the present invention; Figure 4is a schematic diagram of an asymmetric space-frequency feature fusion module according to an embodiment of the present invention; Figure 5 is a schematic diagram of ROC curves of SFNet and other methods on the NUAA dataset according to an embodiment of the present invention; Figure 6 It is a visualization diagram of the detection result according to the embodiment of the present invention and the detection result of the comparison network. DETAILED DESCRIPTION

[0015] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0017] Example 1 According to an embodiment of the present invention, a method for detecting weak and small infrared targets in the air based on spatial-frequency domain feature fusion is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system comprising at least one set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0018] Figure 1 is a flow chart of a method for detecting small infrared targets in the air based on spatial-frequency domain feature fusion according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps: Step S101, acquiring a plurality of infrared small target images under complex backgrounds.

[0019] In the technical solution provided in the above step S101 of the present invention, three public datasets, NUDT, NUAA, and IRSTD-1K, are used, and the training set images thereof are resized to 256×256 respectively.

[0020] Step S102 , dividing a plurality of infrared small target images under complex backgrounds according to a preset ratio to obtain a training set and a test set of the plurality of infrared small target images under complex backgrounds.

[0021] In the technical solution provided in the above step S102 of the present invention, the NUDT, NUAA, and IRSTD-1K public data sets are divided according to a preset ratio to obtain training sets and test sets of infrared weak target images under several complex backgrounds, and the images of the training sets and the test sets are resized to 256×256.

[0022] Step S103, input the training set of infrared dim target images under complex background into the pre-constructed target detection network of space-frequency domain feature fusion, train the pre-constructed target detection network of space-frequency domain feature fusion, and obtain a successfully trained target detection network of space-frequency domain feature fusion, wherein the pre-constructed target detection network of space-frequency domain feature fusion includes an encoder module, a space-frequency dual-branch feature extraction module, an asymmetric space-frequency feature fusion decoder module and a segmentation head module.

[0023] In the technical solution provided in step S103 of the present invention, Figure 2 is a schematic diagram of a target detection network based on pre-constructed spatial-frequency domain feature fusion according to an embodiment of the present invention, such as Figure 2 As shown, a training set of infrared dim target images under complex background is input into a pre-constructed target detection network with space-frequency domain feature fusion, and the pre-constructed target detection network with space-frequency domain feature fusion is trained to obtain a successfully trained target detection network with space-frequency domain feature fusion, wherein the pre-constructed target detection network with space-frequency domain feature fusion includes an encoder module, a space-frequency dual-branch feature extraction module, an asymmetric space-frequency feature fusion decoder module and a segmentation head module.

[0024] Step S104, inputting a test set of several aerial infrared images into the successfully trained target detection network with spatial-frequency domain feature fusion to obtain the segmentation results of the test set images.

[0025] In the technical solution provided in the above step S104 of the present invention, a test set of several aerial infrared images is input into a successfully trained target detection network with spatial-frequency domain feature fusion to obtain a segmentation result of the test set image.

[0026] The above method of this embodiment is further introduced below.

[0027] As an optional implementation method, in step S103, the encoder module includes four convolutional layers of different scales, and multi-scale extraction is performed on the training set image in turn through the four convolutional layers of different scales to obtain four feature maps of different scales, wherein the four feature maps of different scales include a feature map of a first scale, a feature map of a second scale, a feature map of a third scale, and a feature map of a fourth scale.

[0028] In this embodiment, if Figure 2 The four convolutional layers of different scales in the encoder in the training set extract the training set images in sequence to obtain the feature map of the first scale, the feature map of the second scale, the feature map of the third scale and the feature map of the fourth scale.

[0029] As an optional implementation method, step S103, the space-frequency dual-branch feature extraction module includes a neck block and a spatial domain branch and a frequency domain branch, a connection module and a 1×1 convolution layer, the spatial domain branch includes a convolution module, a CBR module, a codebook module, a fusion module, a full connection and a 1×1 convolution layer, a channel-level multiplication module, and a channel-level addition module; the frequency domain branch includes a first normalization layer, a second normalization layer, a first self-attention mechanism, a second self-attention mechanism, a Fourier transform layer, a channel-level multiplication module, and a channel-level addition module; the feature map of the fourth scale is input to the neck block to obtain an initial feature; the initial feature is input to the spatial domain branch to obtain the spatial domain feature of the feature map of the fourth scale; the initial feature is input to the frequency domain branch to obtain the frequency domain feature of the feature map of the fourth scale; the spatial domain feature and the frequency domain feature are sequentially input to the connection module and the 1×1 convolution layer to obtain the target feature of the feature map of the fourth scale.

[0030] In this embodiment, Figure 3 is a schematic diagram of a space-frequency dual-branch feature extraction module according to an embodiment of the present invention, such as Figure 3 As shown in , the space-frequency dual-branch feature extraction module includes a neck block and a space domain branch and a frequency domain branch, a connection module and a 1×1 convolution layer. The space domain branch includes a convolution module, a CBR module, a codebook module, a fusion module, a full connection and a 1×1 convolution layer, a channel-level multiplication module, and a channel-level addition module; the frequency domain branch includes a first normalization layer, a second normalization layer, a first self-attention mechanism, a second self-attention mechanism, a Fourier transform layer, a channel-level multiplication module, and a channel-level addition module. Figure 2 and Figure 3 As shown, the feature map of the fourth scale is input into the neck block to obtain the initial feature; the initial feature is input into the spatial domain branch to obtain the spatial domain feature of the feature map of the fourth scale; the initial feature is input into the frequency domain branch to obtain the frequency domain feature of the feature map of the fourth scale, and the spatial domain feature and the frequency domain feature are sequentially input into the connection module and the 1×1 convolution layer to obtain the target feature of the feature map of the fourth scale. Figure 3The intrinsic codebook encoder in is the spatial domain branch, and the space-frequency query attention is the frequency domain branch.

[0031] As an optional implementation method, the initial feature is input into the spatial domain branch to obtain the spatial domain feature of the feature map of the fourth scale, including: inputting the initial feature into the convolution module and the CBR module in sequence to obtain the target feature map; inputting the target feature map into the codebook module to obtain the information of any codeword corresponding to the target feature map; inputting the information of all codewords into the fusion module to obtain the overall information of the target feature map for all codewords; inputting the overall information of the target feature map for all codewords into the fully connected layer and the 1×1 convolution layer in sequence to obtain the significant features of the target in the feature map of the fourth scale; after calculating the significant features of the target in the feature map of the fourth scale and the scale factor coefficient, they are input into the channel-level multiplication module with the target feature map to obtain the local area features corresponding to the feature map of the fourth scale; the initial features and the local area features are input into the channel-level addition module to obtain the spatial domain features of the feature map of the fourth scale.

[0032] In this embodiment, if Figure 3 As shown in Figure 2, the spatial domain branches mainly include: (1) Unique code book : A set of learnable visual codewords, represented by , where K represents the total number of visual centers; (2) Scaling factor set :With codebook The corresponding set of factors is expressed as , which is used to adjust the mapping relationship between input features and codebook.

[0033] Target feature map After encoding, the features are obtained , the image information corresponding to the kth codeword is calculated by the following formula:

[0034] in, is the i-th pixel in the target feature map; is the kth learnable visual codeword; is the kth scaling factor; Relative to the codeword The position information of each pixel point will be The information fusion of the target feature map is expressed as follows:

[0035] in, Contains BN layer, ReLU layer and average layer.

[0036] After obtaining the output of the codebook, Input to a fully connected layer and a 1×1 convolutional layer, the salient features of the target in the feature map of the fourth scale. Then, the target feature map and the scale factor The above process can be expressed as:

[0037] in, Represents 1×1 convolution; is the sigmoid activation function; represents channel-level multiplication, It is the local area feature corresponding to the feature map of the fourth scale.

[0038] Finally, the expression for the channel-level addition operation of the local area feature Z corresponding to the initial feature and the feature map of the four scales is:

[0039] in, is channel-level addition, It is the spatial domain feature of the feature map of the fourth scale.

[0040] As an optional implementation method, the initial feature is input into the frequency domain branch to obtain the frequency domain feature of the feature map of the fourth scale, including: inputting the initial feature into the first normalization layer to obtain the normalized initial feature, inputting the normalized initial feature into the first self-attention mechanism to obtain the first target feature; inputting the normalized initial feature into the frequency domain feature processor to obtain the frequency domain feature of the normalized initial feature; inputting the frequency domain feature of the normalized initial feature and the first target feature into the second self-attention mechanism to obtain the second target feature; after inputting the second target feature and the initial feature into the channel-level addition module, they are successively input into the second normalization layer and the Fourier transform layer to obtain the frequency domain map of the feature map of the fourth scale; after inputting the second target feature and the initial feature into the channel-level addition module, they are input into the channel-level addition module together with the frequency domain map of the feature map of the fourth scale to obtain the frequency domain feature of the feature map of the fourth scale.

[0041] In this embodiment, if Figure 3 As shown in Figure 1, Spatial-Frequency QueryAttention (SFQA) consists of two layers of self-attention mechanisms.

[0042] Among them, X and Y represent the frequency domain features of the input initial features and the output fourth-scale feature map of the SFQA module, respectively; is the first target feature, A depthwise convolution operator representing the channel space context with a kernel size of m×m; Represents the convolution operator with a kernel size of m×m, which is used to capture cross-channel pixel-level contextual features. m can be taken from the set {1,2,3}. Before applying the Softmax function, the learning parameter α is adjusted to modulate the sum dot product. This scaling factor can control the size of the dot product, thereby affecting the intensity of attention.

[0043] As an optional implementation method, step S103, the decoder module of the asymmetric space-frequency feature fusion includes four asymmetric space-frequency feature fusion modules of different scales and a U-shaped dilated convolution, wherein the four asymmetric space-frequency feature fusion modules of different scales include an asymmetric space-frequency feature fusion module of a first size, an asymmetric space-frequency feature fusion module of a second size, an asymmetric space-frequency feature fusion module of a third size, and an asymmetric space-frequency feature fusion module of a fourth size; the target features and feature maps of the feature map of the fourth scale are input into the asymmetric space-frequency feature fusion module of the first scale to obtain a fused feature map of the first scale; the fused feature map of the first scale is input into the U-shaped dilated convolution module to obtain a target fine feature map; the target fine feature map and the feature map of the third scale are input into the asymmetric space-frequency feature fusion module of the second scale to obtain a fine fused feature map of the second scale; the fine fused feature map of the second scale and the feature map of the second scale are input into the asymmetric space-frequency feature fusion module of the third scale to obtain a fine fused feature map of the third scale; the fine fused feature map of the second scale and the feature map of the first scale are input into the asymmetric space-frequency feature fusion module of the fourth scale to obtain a fine fused feature map of the fourth scale.

[0044] In this embodiment, the decoder module of the asymmetric space-frequency feature fusion is as follows Figure 2 As shown in the decoder, Figure 4 is a schematic diagram of an asymmetric space-frequency feature fusion module according to an embodiment of the present invention. Figure 4As shown in Figure 1, the processing process of the asymmetric space-frequency feature fusion module is as follows: shallow features are processed by the space-frequency pixel attention mechanism to retain the texture and position information of the target; this mechanism uses the pixel-level attention weights in the spatial and frequency domains to weight shallow features, highlight the target area, and suppress background noise; deep features are processed by the channel attention mechanism to strengthen semantic information; the channel attention mechanism weights deep features according to the feature importance of each channel, highlights channels with significant semantic information, and improves the semantic understanding ability of the network; the two are fused in an asymmetric way to ensure that more important information of weak targets is retained during the fusion process; the asymmetric fusion strategy flexibly adjusts the fusion method and weight according to the different characteristics of shallow features and deep features to achieve information complementarity and enhancement; the structure of the U-shaped dilated convolution module is shown in Figure 1. Figure 2 shown.

[0045] As an optional implementation manner, the fourth-scale fine fusion feature map is input into the segmentation head module to obtain a weak target of an infrared weak target image under a complex background.

[0046] In this embodiment, the input segmentation head calculates the segmentation loss function SoftIoULoss to guide the network to perform optimization training. The specific formula is as follows:

[0047]

[0048]

[0049] in, is the intersection area between the predicted result and the true value; is the union area between the predicted result and the true value, is the first in the input vector corresponding to the fourth scale fine fusion feature map i elements, K is the total number of elements in the input vector, i.e. the total number of categories.

[0050] Experimental part: The training sets of the embodiments of the present invention include training sets of three public data sets, namely NUDT, NUAA, and IRSTD-1K, which include 1,000 synthetic images, 223 real images, and 800 real images, respectively. The test sets of the embodiments of the present invention are test sets of three public data sets, namely NUDT, NUAA, and IRSTD-1K, which include 327 synthetic images, 224 real images, and 200 real images, respectively.

[0051] The embodiment of the present invention takes a synthetic infrared image, a real infrared image and its corresponding transmission map as input, and uses a reference image to calculate the loss function. All images are resized to 256×256. The embodiment of the present invention uses SoftIoULoss as the loss function, adopts the AdamW optimizer, and uses a multi-learning rate decay strategy and a deep supervision training strategy. The experimental batch size is set to 8, the initial learning rate is 0.0003, and the input dimension is set to 64 by default. The embodiment of the present invention uses an NVIDIA GeForce RTX 3090 graphics card to train the entire network, and during the training process, the Adam optimizer is used for 1500 training iterations.

[0052] Based on the above description, Table 1 shows the test results of the method of the embodiment of the present invention and some related methods on the NUDT, NUAA, and IRSTD-1K datasets, and uses the target detection evaluation indicators intersection over Union (IoU), detection probability (Pd), false alarm rate (Fa), and ROC curve (Receiver Operating Characteristic) to describe the accuracy of the test results. It can be found that the method of the embodiment of the present invention basically obtains the best results in all indicators.

[0053] 1) Intersection over Union (IoU): A pixel-level evaluation indicator used to evaluate the segmentation performance of the algorithm. IoU is defined as the ratio of the intersection area of ​​the predicted result and the true label to the union area. The formula is as follows:

[0054] in, is the intersection area; is the union area.

[0055] 2) Detection Probability (Pd): An object-level evaluation indicator used to evaluate the accuracy of object detection. Pd is defined as the number of correctly predicted objects. Total number of targets The ratio is as follows:

[0056] When the centroid deviation of the predicted target is less than a predefined threshold (set to 3 pixels in the embodiment of the present invention), the prediction is considered to be a correct detection.

[0057] 3) False Alarm (Fa): Another target-level evaluation metric used to evaluate mispredictions. Fa is defined as the number of pixels that are mispredicted. Total number of pixels The ratio is as follows:

[0058] If the centroid deviation of the predicted target is greater than a predefined threshold (set to 3 pixels in the embodiment of the present invention), the prediction is considered to be a false positive.

[0059] 4) Receiver Operating Characteristic (ROC) curve: By plotting the relationship between the true positive rate (TruePositive Ratio) and the false positive rate (FalsePositive Ratio), the detection performance of the network at different thresholds is evaluated. The closer the ROC curve is to the upper left corner, the higher the detection accuracy. At the same time, the detection performance can also be reflected by calculating the area under the ROC curve (Area Under Curve, referred to as AUC).

[0060] Table 1 Comparison of the method of the embodiment of the present invention with other related methods on the NUDT, NUAA, and IRSTD-1K datasets

[0061] besides, Figure 5 : is a schematic diagram of ROC curves of SFNet and other methods on the NUAA dataset according to an embodiment of the present invention, as shown in Figure 5 As shown, the area enclosed by the curve of the embodiment of the present invention and the x-axis is the largest, and the inflection point of the curve is closest to the upper left corner, indicating that the embodiment of the present invention can achieve a higher detection rate while maintaining a lower false alarm rate and has the best performance.

[0062] Figure 6 is a visualization diagram of the detection results according to an embodiment of the present invention and the detection results of a comparison network, Figure 6 The detection results of the comparison networks DNANet, ABCNet and the embodiment of the present invention are visualized. The red box represents the correctly detected target and the yellow box represents the false alarm. It can be seen that although the three networks all have the ability to achieve target segmentation, compared with DNANet and ABCNet, the number of false alarms in the embodiment of the present invention is significantly reduced, and the description of the target edge contour is more accurate.

[0063] In the embodiment of the present invention, a plurality of infrared small target images under complex backgrounds are acquired; the plurality of infrared small target images under complex backgrounds are divided according to a preset ratio to obtain a training set and a test set of the infrared small target images under complex backgrounds; the training set of the infrared small target images under complex backgrounds is input into a pre-constructed target detection network of space-frequency domain feature fusion, and the pre-constructed target detection network of space-frequency domain feature fusion is trained to obtain a successfully trained target detection network of space-frequency domain feature fusion, wherein the pre-constructed target detection network of space-frequency domain feature fusion comprises an encoder module, a space-frequency dual-branch feature extraction module, an asymmetric space-frequency feature fusion decoder module and a segmentation head module; the test set of the plurality of aerial infrared images is input into the successfully trained target detection network of space-frequency domain feature fusion to obtain a segmentation result of the test set image, thereby solving the technical problem that the existing convolutional neural network detects infrared small target under complex backgrounds with low detection accuracy and high false alarm and missed detection rates, and achieving the technical effect that the designed network framework is used to detect infrared small target under complex backgrounds, improves detection accuracy, and reduces false alarm and missed detection rates.

[0064] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0065] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0066] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0067] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0068] In addition, each functional unit in each embodiment of the present invention may be integrated into a first processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0069] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for detecting small infrared targets in the air based on spatial-frequency domain feature fusion, characterized in that: include: Acquire several infrared weak target images under complex backgrounds; Dividing a number of infrared dim target images under complex backgrounds according to a preset ratio to obtain a training set and a test set of the infrared dim target images under complex backgrounds; Inputting a training set of infrared dim target images under complex background into a pre-constructed target detection network with space-frequency domain feature fusion, training the pre-constructed target detection network with space-frequency domain feature fusion, and obtaining a successfully trained target detection network with space-frequency domain feature fusion, wherein the pre-constructed target detection network with space-frequency domain feature fusion includes an encoder module, a space-frequency dual-branch feature extraction module, an asymmetric space-frequency feature fusion decoder module, and a segmentation head module; A test set of several aerial infrared images is input into the successfully trained target detection network with spatial-frequency domain feature fusion to obtain the segmentation results of the test set images.

2. The method according to claim 1, characterized in that The encoder module includes four convolutional layers of different scales, and multi-scale extraction is performed on the training set images in sequence through the four convolutional layers of different scales to obtain feature maps of four different scales, wherein the four feature maps of different scales include a feature map of a first scale, a feature map of a second scale, a feature map of a third scale, and a feature map of a fourth scale.

3. The method according to claim 2, characterized in that The space-frequency dual-branch feature extraction module includes a neck block, a space domain branch, a frequency domain branch, a connection module, and a 1×1 convolution layer; the space domain branch includes a convolution module, a CBR module, a codebook module, a fusion module, a full connection and a 1×1 convolution layer, a channel-level multiplication module, and a channel-level addition module; the frequency domain branch includes a first normalization layer, a second normalization layer, a first self-attention mechanism, a second self-attention mechanism, a Fourier transform layer, a channel-level multiplication module, and a channel-level addition module; Input the feature map of the fourth scale into the neck block to obtain the initial feature; Input the initial features into the spatial domain branch to obtain the spatial domain features of the feature map of the fourth scale; Input the initial features into the frequency domain branch to obtain the frequency domain features of the feature map of the fourth scale; The spatial domain features and frequency domain features are input into the connection module and the 1×1 convolution layer in sequence to obtain the target features of the feature map of the fourth scale.

4. The method according to claim 3, characterized in that The step of inputting the initial features into the spatial domain branch to obtain the spatial domain features of the feature map of the fourth scale includes: The initial features are sequentially input into the convolution module and the CBR module to obtain the target feature map; Input the target feature map into the codebook module to obtain information of any codeword corresponding to the target feature map; Input the information of all codewords into the fusion module to obtain the overall information of the target feature map for all codewords; The overall information of the target feature map for all codewords is sequentially input into the fully connected layer and the 1×1 convolutional layer to obtain the salient features of the target in the feature map of the fourth scale; After calculating the significant features of the target in the feature map of the fourth scale and the scale factor coefficient, the feature map of the target is input into the channel-level multiplication module to obtain the local area features corresponding to the feature map of the fourth scale; The initial features and local area features are input into the channel-level addition module to obtain the spatial domain features of the feature map of the fourth scale.

5. The method according to claim 3, characterized in that: The inputting the initial features into the frequency domain branch to obtain the frequency domain features of the feature map of the fourth scale includes: Input the initial features into the first normalization layer to obtain the normalized initial features, and input the normalized initial features into the first self-attention mechanism to obtain the first target features; Inputting the normalized initial features into a frequency domain feature processor to obtain frequency domain features of the normalized initial features; The normalized frequency domain features of the initial features and the first target features are input into the second self-attention mechanism to obtain the second target features; After the second target feature and the initial feature are input into the channel-level addition module, they are sequentially input into the second normalization layer and the Fourier transform layer to obtain a frequency domain map of the feature map of the fourth scale; After the second target feature and the initial feature are input into the channel-level addition module, the frequency domain map of the feature map of the fourth scale is input into the channel-level addition module to obtain the frequency domain features of the feature map of the fourth scale.

6. The method according to claim 3, characterized in that The decoder module of the asymmetric space-frequency feature fusion includes four asymmetric space-frequency feature fusion modules of different scales and a U-shaped dilated convolution, wherein the four asymmetric space-frequency feature fusion modules of different scales include an asymmetric space-frequency feature fusion module of a first size, an asymmetric space-frequency feature fusion module of a second size, an asymmetric space-frequency feature fusion module of a third size, and an asymmetric space-frequency feature fusion module of a fourth size; Inputting the target features and feature map of the feature map of the fourth scale into the asymmetric space-frequency feature fusion module of the first scale to obtain a fused feature map of the first scale; The first scale fusion feature map is input into the U-shaped dilated convolution module to obtain the target fine feature map; Inputting the target fine feature map and the feature map of the third scale into the asymmetric space-frequency feature fusion module of the second scale to obtain a fine fused feature map of the second scale; Inputting the second-scale fine fusion feature map and the second-scale feature map into the asymmetric space-frequency feature fusion module of the third scale to obtain the third-scale fine fusion feature map; The second-scale fine fusion feature map and the first-scale feature map are input into the asymmetric space-frequency feature fusion module of the fourth scale to obtain a fourth-scale fine fusion feature map.

7. The method according to claim 6, characterized in that The fourth scale fine fusion feature map is input into the segmentation head module to obtain the weak target of the infrared weak target image under the complex background.

8. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.

9. A computer-readable storage medium, characterized in that Computer executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.

10. A computer program product, characterized in that The invention comprises computer executable instructions, which are used to implement the method of claim 1 when being executed.

Images