Infrared Small Target Detection Method for Sea and Sky Scenes Based on Multidimensional Feature Fusion

Abstract

This invention discloses a method for detecting small infrared targets in seascape scenes based on multi-dimensional feature fusion in the field of infrared target detection technology. This method includes the following steps: extracting information from the input infrared image using semantic information, edge information, and detail information modules; optimizing and fusing the multi-dimensional information using a dynamic weight allocation mechanism dominated by semantic information; and generating a mask image using the output detection image. This method effectively improves the detection accuracy of small infrared targets in seascape scenes.

Description

Technical Field

[0001] This invention relates to the field of infrared target detection technology, and more specifically, to a method for detecting small infrared targets in sea and sky scenes based on multi-dimensional feature fusion. Background Technology

[0002] Infrared small target detection technology has wide applications in fields such as maritime defense and maritime surveillance. However, under long-distance observation, because the target is far away from the observation equipment, infrared small targets occupy very few pixels in the image, appearing as spots or even dots, lacking effective shape features, and also lacking the texture, color, and shape features of common objects. Coupled with the interference of complex backgrounds, the detection of weak small targets in sea and sky scenes faces great difficulties.

[0003] Infrared small target detection technology is mainly divided into traditional algorithms and deep learning algorithms. Due to the complexity and variability of sea surface targets and scenes, traditional algorithms have poor adaptability. Unlike traditional algorithms that rely on manually designed features, deep learning relies on massive amounts of data to learn target features, achieving remarkable results in both speed and accuracy of target detection on general datasets. However, most current deep learning-based small target detection algorithms are trained on datasets from non-sea surface scenes. Due to the special characteristics of sea surface targets and backgrounds, directly applying models trained on non-sea surface small target detection datasets to sea surface small target detection yields poor results. Therefore, there is an urgent need to design an infrared small target detection method for sea and sky scenes based on multi-dimensional feature fusion to improve the detection accuracy of infrared small targets in sea surface scenes. Summary of the Invention

[0004] To address the aforementioned issues, this invention aims to provide an infrared small target detection method for sea and sky scenes based on multi-dimensional feature fusion. This method solves the problems of existing technologies, such as the complexity and variability of sea surface targets and scenes, the poor adaptability of traditional algorithms, and the poor detection performance of most deep learning-based small target detection algorithms trained on datasets from non-sea surface scenes. The method extracts semantic information from the image using an improved FPN network, extracts edge information using a gradient information extraction operator, and extracts detail information using an improved convolution module. Finally, the three types of information are fused by a multi-dimensional information extraction module and then processed by a segmentation module to obtain a binary mask image containing target information, effectively improving the detection accuracy of infrared small targets in sea and sky scenes.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] A method for detecting small infrared targets in sea and sky scenes based on multi-dimensional feature fusion includes the following steps:

[0007] S1. Extract information from the input infrared image using semantic information, edge information, and detail information modules;

[0008] S2. Multi-dimensional information is optimized and fused using a dynamic weight allocation mechanism dominated by semantic information;

[0009] S3. Generate a mask image using the output detection image.

[0010] Furthermore, S1 includes the following steps:

[0011] S11. Input infrared image;

[0012] S12. Using the FPN architecture with ResNet-20 as the backbone network, semantic information features are extracted from the input infrared image, and the features x2, x3, and x4 processed by the first, second, and third modules are obtained in sequence.

[0013] Furthermore, S2 includes the following steps:

[0014] S21. The obtained feature x4 and feature x3 are fused using the multi-scale feature fusion module MSFM to obtain feature x5;

[0015] S22. The obtained feature x5 and feature x2 are fused using MSFM to obtain feature x6;

[0016] S23. Obtain edge information feature x7 through the edge information processing module;

[0017] S24. Obtain detailed information feature x8 through the detailed information processing module;

[0018] S25. Semantic information, detailed information and peripheral information are fused through a multi-dimensional information fusion module.

[0019] Furthermore, the calculation formula for information fusion performed by the multi-scale feature fusion module (MSFM) in S21 is as follows:

[0020]

[0021] F D =CA(F D )=σ(β(W2δ(β(W1F D ))))

[0022] α=PA(F M )=σ((W2δ(β(W1F M ))))

[0023] Where σ,δ,β,W, These represent the Sigmoid function, ReLU function, batch normalization, convolution operation, addition operation, and multiplication operation, respectively. F DF represents a deep feature map containing semantic information. S α represents a shallow feature map containing detailed information, CA represents channel attention mechanism, PA represents pixel attention mechanism, and α represents dynamic weights fused from guiding information extracted from deep feature maps.

[0024] Furthermore, S23 includes the following steps:

[0025] S231. Calculate the horizontal gradient X i,v and vertical gradient X i,h ;

[0026] x i.v =Conv2d(x i kernel v )

[0027] x i,h =Conv2d(x h kernel h )

[0028] Where Conv2d represents a two-dimensional convolution operation, X i For the i-th input channel, kernel h and kernel v These represent the vertical convolution kernel and the horizontal convolution kernel, respectively.

[0029] S232. Through the following total gradient χ i The calculation formula yields the edge information feature map.

[0030]

[0031] Furthermore, S25 includes the following steps:

[0032] S251. After obtaining the guiding weights by global average pooling, convolutional blocks, batch normalization and the Sigmoid function, the extracted semantic information is used as the basis to assign different weights to the edge information feature map and the detail information feature map, and then multiplied into the edge information feature map and the detail information feature map.

[0033] The process formula is as follows:

[0034]

[0035] α=SW(F S )=σ((β(Wf D )))

[0036] The formula for global average pooling is as follows:

[0037]

[0038] Where σ,δ,β,W, These represent the Sigmoid function, ReLU function, batch normalization, convolution operation, addition operation, and multiplication operation, respectively. F B Represents edge information, F S Represents semantic information, F D SW represents the semantic information guiding weight generation mechanism, and α represents the dynamic weight fused from the guiding information extracted from the semantic information.

[0039] S252. After adding the product of the weights and edge information feature maps and the product of the weights and detail information feature maps, perform convolution filtering to obtain multi-dimensional information features.

[0040] The convolutional filtering formula is as follows:

[0041] SConv=Conv(δ(β(F A ))).

[0042] Where δ, β, and Conv represent the ReLU function, batch normalization, 3×3 convolution operation, and F, respectively. A The result is the sum of the product of the weight and the edge information feature map and the product of the weight and the detail information feature map.

[0043] Furthermore, S3 includes the following steps:

[0044] S31. The multi-dimensional information fusion features are processed by the segmentation module to output the detection map;

[0045] S32. Perform binary processing on each value of the output detection image to output a grayscale image;

[0046] S33. In the grayscale image, obtain the infrared small target or the background of the infrared image at the corresponding position in the different grayscale value areas;

[0047] S34. Obtain the infrared small target mask image.

[0048] Furthermore, S33 includes the following steps:

[0049] S331. In the grayscale image, the area with a grayscale value of 1 is a white pixel, and the corresponding position is detected as a small infrared target;

[0050] S332. In a grayscale image, the area with a grayscale value of 0 is a black pixel, and the corresponding position is the background of the infrared image.

[0051] By adopting the above technical solution, the present invention has the following advantages:

[0052] This invention provides a method for detecting small infrared targets in seascape scenes based on multi-dimensional feature fusion. This method introduces an edge information extraction module, which not only compensates for the loss of target information caused by downsampling but also provides edge information, achieving more accurate target detection. To further reduce the loss caused by downsampling, inspired by the DeepLab network structure, a shallow feature map with richer detail information is introduced in the final stage. This method proposes a fusion mechanism that combines semantic information with detail and edge information. This mechanism first extracts semantic information from an improved FPN baseline network, and then organically integrates all three components using a semantic information-driven dynamic weight allocation mechanism, effectively improving the detection accuracy of small infrared targets in seascape scenes. Attached Figure Description

[0053] Figure 1 This is a flowchart of the infrared small target detection method for sea and sky scenes according to the present invention;

[0054] Figure 2 The infrared image is the input for this invention;

[0055] Figure 3 This is a schematic diagram of the improved FPN network architecture of the present invention;

[0056] Figure 4 This is a structural diagram of the multi-scale feature fusion module of the present invention;

[0057] Figure 5 This is the structure of the multi-dimensional information extraction module of the present invention;

[0058] Figure 6 This is the annotation mask diagram of the present invention;

[0059] Figure 7 This is a comparison chart of ROC curves obtained from experiments on a test set between the infrared small target detection method for sea and sky scenes of the present invention and other advanced infrared small target detection algorithms. Detailed Implementation

[0060] The technical solution of the present invention will be specifically described below with reference to the accompanying drawings. It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0061] Figure 1 A flowchart of the infrared small target detection method for seascape scenes according to the present invention is shown. A specific method for infrared small target detection in seascape scenes based on multi-dimensional feature fusion is as follows: Figure 1 As shown, it includes the following steps:

[0062] S1. Extract information from the input infrared image using semantic information, edge information, and detail information modules;

[0063] S1 includes the following specific steps:

[0064] S11. Input an infrared image with dimensions of 640×512×3 (length×width×height). Figure 2 As shown.

[0065] S12. A ResNet-20 backbone FPN architecture is used to extract semantic information features from the input infrared image. After processing by the first module, a feature map x1 with a size of 120×120×16 is obtained. After passing through the second ResNet block, a feature map x2 with a size of 60×60×32 is obtained. After passing through the third ResNet block, a feature map x3 with a size of 30×30×64 is obtained. The specific schematic diagram of the backbone FPN architecture is shown below. Figure 3 As shown.

[0066] S2. Multi-dimensional information is optimized and fused using a dynamic weight allocation mechanism dominated by semantic information;

[0067] S2 includes the following specific steps:

[0068] S21. The feature map x4 with length × width × height of 60 × 60 × 64 and the feature map x2 with length × width × height of 60 × 60 × 32 are fused through the multi-scale feature fusion module MSFM, and after upsampling, the feature map x5 with length × width × height of 120 × 120 × 32 is obtained.

[0069] The calculation formula for information fusion in the Multi-Scale Feature Fusion (MSFM) module of S21 is as follows:

[0070]

[0071] F D =CA(F D )=σ(β(Wδ(β(WF D ))))

[0072] α=PA(F M )=σ((Wδ(β(WF M ))))

[0073] Where σ,δ,β,W, These represent the Sigmoid function, ReLU function, batch normalization, convolution operation, addition operation, and multiplication operation, respectively. F D F represents a deep feature map containing semantic information. S S22. The shallow feature map containing detailed information is represented by CA, channel attention mechanism, pixel attention mechanism, and α, which represents the dynamic weights for fusing guided information extracted from the deep feature map. The feature map x5 (length × width × height 120 × 120 × 32) and feature map x1 (length × width × height 120 × 120 × 16) are fused using the multi-scale feature fusion module MSFM to obtain feature map x6 (length × width × height 120 × 120 × 16). The multi-scale feature fusion module is described in detail below. Figure 4 As shown.

[0074] S23. Obtain an edge information feature map x7 with a size of 480×480×1 through the first-order edge information processing module.

[0075] S23 includes the following specific steps:

[0076] S231. Calculate the horizontal gradient x i.v and vertical gradient x i,h ;

[0077] x i.v =Conv2d(x i kernel v )

[0078] x i,h =Conv2d(x h kernel h )

[0079] Where Conv2d represents a two-dimensional convolution operation, X i For the i-th input channel, kernel h and kernel v These represent the vertical convolution kernel and the horizontal convolution kernel, respectively.

[0080] S232. After calculating the horizontal and vertical gradients separately, the total gradient χ is obtained through the following... i The calculation formula yields the edge information feature map.

[0081]

[0082] S24. A detail information feature map x8 with a size of 480×480×3 is obtained by a detail information processing module composed of 3×3 convolutional kernels;

[0083] S25. Semantic information, detailed information, and edge information are fused through a multi-dimensional information fusion module. The multi-dimensional information fusion module is specifically as follows: Figure 5 As shown, the specific operation is as follows: edge information is extracted through the first-order gradient extraction module, and detail information is extracted through the convolution module. The extracted semantic information is then processed by global average pooling, convolutional blocks, batch normalization, and the Sigmoid function to obtain guiding weights. Based on these guiding weights, different weights are assigned to the edge information feature map and the detail information feature map for multiplication. Finally, the results of the two multiplications are added together and subjected to convolutional filtering to obtain multi-dimensional information features.

[0084] The process formula is as follows:

[0085]

[0086] α=SW(F S )=σ((β(Wf D )))

[0087] The formula for global average pooling is as follows:

[0088]

[0089] Where σ,δ,β,W, These represent the Sigmoid function, ReLU function, batch normalization, convolution operation, addition operation, and multiplication operation, respectively. F B Represents edge information, F S Represents semantic information, F D SW represents the semantic information guiding weight generation mechanism, and α represents the dynamic weight fused from the guiding information extracted from the semantic information.

[0090] The convolutional filtering formula is as follows:

[0091] SConv=Conv(δ(β(F A ))).

[0092] Where δ, β, and Conv represent the ReLU function, batch normalization, 3×3 convolution operation, and F, respectively. A The result is the sum of the product of the weight and the edge information feature map and the product of the weight and the detail information feature map.

[0093] S3. Generate a mask image using the output detection image.

[0094] S3 includes the following steps:

[0095] S31. The multi-dimensional information fusion features are processed by the segmentation module to output the detection map;

[0096] S32. Perform binary processing on each value of the output detection image to output a grayscale image;

[0097] S33. In the grayscale image, obtain the infrared small target or the background of the infrared image at the corresponding position in the different grayscale value areas;

[0098] S33 includes the following specific steps:

[0099] S331. In the grayscale image, the area with a grayscale value of 1 is a white pixel, and the corresponding position is detected as a small infrared target;

[0100] S332. In a grayscale image, the area with a grayscale value of 0 is a black pixel, and the corresponding position is the background of the infrared image.

[0101] S34. The above results yield an infrared small target mask image, as shown below. Figure 6 As shown.

[0102] In one specific embodiment, the results of comparing the infrared small target detection method for sea and sky scenes of the present invention with other current advanced infrared small target detection algorithms are shown in Table 1.

[0103] Table 1: Comparison of the Infrared Small Target Detection Method of the Present Invention with Other Advanced Infrared Small Target Detection Algorithms

[0104]

[0105] As shown in Table 1, compared with other advanced infrared small target detection algorithms, the infrared small target detection method for sea and sky scenes based on multi-dimensional feature fusion of this invention achieves the best performance in terms of cross-union ratio, average cross-union ratio, weighted harmonic mean, and prediction capability evaluation value. The ROC curves obtained from experiments on the test set using the infrared small target detection method for sea and sky scenes based on multi-dimensional feature fusion of this invention and other advanced infrared small target detection algorithms are shown in the figure below. Figure 7 As shown, from Figure 7 As can be seen from the above, the performance of the infrared small target detection method for sea and sky scenes based on multi-dimensional feature fusion of the present invention is still at the optimal level.

[0106] Finally, it should be noted that although the present invention has been described with reference to specific embodiments, those skilled in the art should recognize that the above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Various equivalent changes or substitutions can be made without departing from the concept of the present invention. Therefore, any changes or modifications to the above embodiments within the essential spirit of the present invention will fall within the scope of the claims of the present invention.

Claims

1. A method for detecting an infrared small target in a sea-sky scene based on multi-dimensional feature fusion, characterized in that, It comprises the following steps: S1. Extracting information from the input infrared image through semantic information, edge information, and detail information modules; S2. Optimizing and fusing features using a dynamic weight distribution mechanism dominated by semantic information in multiple dimensions; The S2 comprises the following steps: S21. Fusing information obtained from features x4 and x3 through a multi-scale feature fusion module MSFM to obtain feature x5; The calculation formula of the information fusion of the multi-scale feature fusion module MSFM in the S21 is as follows: F M = CA(F D ) = σ(β(Wδ(β(WF D )))) a = PA(F M ) = σ((Wδ(β(WF M )))) wherein σ, δ, β, W, respectively represent Sigmoid function, ReLu function, batch normalization, convolution operation, addition operation and multiplication operation, F D represents deep feature map containing semantic information, F S represents shallow feature map containing detail information, CA represents channel attention mechanism, PA represents pixel attention mechanism, and α represents dynamic weight of guide information fusion extracted from deep feature map, F M represents F D ; S22. Fusing information obtained from features x5 and x2 through MSFM to obtain feature x6; S23. Obtaining edge information feature x7 through an edge information processing module; S24. Obtaining detail information feature x8 through a detail information processing module; S25. Fusing semantic information, detail information, and edge information through a multi-dimensional information fusion module MDFM; The S25 comprises the following steps: S251. After obtaining the guide weight by globally averaging the extracted semantic information, convolution block, batch normalization, and Sigmoid function, different weights are assigned to the edge information feature map and the detail information feature map based on the guide weight; The formula is as follows: a = SW(F S ) = σ((β(Wf D ))) The formula of the global average pooling is as follows: wherein σ, δ, β, W, respectively represent Sigmoid function, ReLu function, batch normalization, convolution operation, addition operation and multiplication operation, F B represents edge information, F S represents semantic information, F D represents detail information, SW represents semantic information guide weight generation mechanism, and α represents dynamic weight of guide information fusion extracted from semantic information. S252. After adding the weight and edge information feature map product and the weight and detail information feature map product, convolution filtering processing is performed to obtain multi-dimensional information fusion features; The convolution filtering processing formula is as follows: SConv = Conv(δ(β(F A ))) wherein, Conv represents a 3x3 convolution operation, F A represents the result of adding the product of the weight and the edge information feature map and the product of the weight and the detail information feature map; S3. Generating a mask mask image using the output detection image.

2. The sea-sky scene infrared small target detection method based on multi-dimensional feature fusion according to claim 1, characterized in that, The S1 comprises the following steps: S11. Inputting an infrared image; S12. Extracting semantic information features from the input infrared image using a FPN architecture with Resnet-20 as the backbone network to obtain features x2, x3, and x4 processed by the first, second, and third modules in sequence.

3. The sea-sky scene infrared small target detection method based on multi-dimensional feature fusion according to claim 1, characterized in that, The S23 comprises the following steps: S231. Compute horizontal gradient X i v and vertical gradient X i h; x i.v = Conv2d(x i , kernel v ) x i,h = Conv2d(x h , kernel h ) wherein Conv2d denotes a two-dimensional convolution operation, X i is the i-th input channel, kernel h and kernel ν denote a vertical and a horizontal convolution kernel, respectively. S232. The total gradient χ is calculated by i The calculation formula is used to calculate the edge information feature map.

4. The sea-sky scene infrared small target detection method based on multi-dimensional feature fusion according to claim 1, characterized in that, The S3 comprises the following steps: S31. Processing the multi-dimensional information fusion features through a segmentation module to output a detection image; S32. Performing binary processing on each value of the output detection image to output a grayscale image; S33. In different grayscale value regions in the grayscale image, the corresponding positions of the infrared small targets or the background of the infrared image are obtained; S34. Obtaining an infrared small target mask mask image.

5. The sea-sky scene infrared small target detection method based on multi-dimensional feature fusion according to claim 4, characterized in that, The S33 comprises the following steps: S331. In the grayscale image, the region with a grayscale value of 1 is a white pixel, and the corresponding position is detected as an infrared small target; S332. In the grayscale image, the region with a grayscale value of 0 is a black pixel, and the corresponding position is the background of the infrared image.

Images