Multi-scale local contrast attention infrared small target detection method

Abstract

The invention provides a multi-scale local contrast attention infrared small target detection method, and the method comprises the steps: inputting an initial infrared image input feature into a preset backbone network for feature coding, and inputting at least one obtained down-sampling infrared image feature and an integrated infrared image feature into a feature enhancement module, the feature enhancement module comprises a plurality of attention modules which are respectively used for processing each down-sampling infrared image feature and processing integrated infrared image features, and the attention modules are used for performing multi-scale channel attention feature and multi-scale space attention feature fusion on input features to obtain attention features; inputting the attention features of different scales output by each attention module into a feature fusion module to obtain fused infrared image features; and inputting the fused infrared image features into a prediction module to obtain a result image of infrared small target segmentation in the infrared image. According to the method, the infrared small target can be effectively segmented.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-scale local contrast attention infrared small target detection method. Background Art

[0002] Single-frame infrared small target (SIRST) detection is one of the key technologies for infrared search and tracking, and is widely used in early warning systems, precision-guided weapons, maritime surveillance systems, and other fields. The purpose of SIRST detection is to accurately locate the target against different infrared backgrounds. Unlike general targets, small infrared targets are usually located at a certain distance from the infrared detection system and usually occupy only a few pixels in the image. In addition, the signal-to-noise ratio of small infrared targets is relatively low and lacks sufficient texture details. This is due to atmospheric scattering, refraction, optical defocusing, and various noise sources. Therefore, how to detect small targets quickly and accurately is a challenging task.

[0003] In order to detect infrared small targets, there are currently two main methods, model-driven and data-driven. Model-driven can be further divided into three categories: filter-based, local information-based, and optimization-based. Among them, the local contrast measurement method is the most commonly used, which uses the grayscale difference between the target and the neighborhood to calculate the contrast of each pixel in the image, effectively enhancing the target while suppressing the background. Model-driven models usually combine domain expertise and mathematical models to provide good interpretability. However, these methods require a lot of experience to design hand-crafted features and adjust hyperparameters. They also often separate feature extraction, threshold segmentation, and target detection steps. Therefore, when encountering complex backgrounds and unknown target shapes, the effectiveness of model-driven is limited.

[0004] In recent years, computer vision has developed rapidly. Learning infrared small target features in a data-driven way has achieved good results. Based on data-driven, there are three main types: detection-based, generative adversarial network-based, and segmentation-based. The data-driven type automatically learns features from a large amount of training data, eliminating the need for manual feature design. Compared with manually extracted features, this method can achieve more general feature extraction in complex backgrounds in an end-to-end manner.

[0005] In many areas of machine vision, the attention mechanism enables neural networks to automatically learn and selectively focus on important information in the input, thereby improving the performance and generalization ability of the model. However, small infrared targets usually exist in complex backgrounds with low contrast, and their size is relatively small and the edges are blurred. The general attention mechanism will focus too much attention on non-small target areas, resulting in the small target information being ignored.

[0006] In order to solve the above problems, some researchers proposed to integrate the model-driven idea into the data-driven model. ALCNet first introduced local contrast metrics in the deep learning network to enhance the contrast of small targets; on the one hand, RISTDnet, RDIAN and HoLoCoNet enhance the targets in the input image by using fixed convolution kernels of different sizes or local contrast weights before the image enters the deep network; however, these methods only perform local contrast enhancement at the beginning of the network, which can be regarded as providing a priori for subsequent feature extraction, but ignore the interaction of contrast information between low-level and high-level features. On the other hand, MLCLNet, ALCLNet and LCAGNet attempt to modularize the local contrast metric and then enhance the network's ability to extract the contrast of small targets by combining shallow and deep features. However, the contrast metric module they designed failed to fully explore and utilize the contrast information on channel and spatial features. The cascade feature map generated by the multi-scale contrast measurement module contains rich image feature information, which helps the model better understand the image content. These methods choose to extract features only by maximum pooling or average pooling of multi-scale connected feature maps. This method may amplify the interference of bright noise in complex backgrounds and may lose subtle feature information that helps distinguish weak targets. Summary of the invention

[0007] In view of the above problems, the present invention proposes a multi-scale local contrast attention infrared small target detection method to overcome the above problems.

[0008] The present invention provides a multi-scale local contrast attention infrared small target detection method, the method comprising:

[0009] Inputting the initial infrared image input features into a preset backbone network for feature encoding; wherein the backbone network includes a plurality of cascaded downsampling modules and a multi-scale hierarchical feature integration module connected to the last layer of downsampling modules, so as to obtain at least one downsampled infrared image feature through at least one downsampling module in the cascaded multiple downsampling modules, and performing local feature extraction and global feature extraction of different scales on the sampling results of the last layer of downsampling modules through the multi-scale hierarchical feature integration module, and performing feature fusion on the obtained local features and global features of multiple scales to obtain integrated infrared image features;

[0010] Inputting at least one downsampled infrared image feature and the integrated infrared image feature into a feature enhancement module respectively; wherein the feature enhancement module comprises a plurality of attention modules for processing each downsampled infrared image feature and for processing the integrated infrared image feature respectively, and the attention module obtains an attention feature by fusing a multi-scale channel attention feature and a multi-scale spatial attention feature on the input feature;

[0011] The attention features of different scales output by each attention module are respectively input into the feature fusion module, and the input attention features of different scales are fused by the feature fusion module to strengthen the feature representation of the target feature and obtain the fused infrared image feature;

[0012] The fused infrared image features are input into a prediction module to obtain a result image of infrared small target segmentation in the infrared image.

[0013] Further, the feature fusion module includes at least one cascaded feature fusion submodule;

[0014] The attention features of different scales output by each attention module are input into the feature fusion module, including:

[0015] Input the attention features output by the attention module corresponding to the multi-scale hierarchical feature integration module as low-level features into the feature fusion submodule at the bottom layer of the feature fusion module, and input the output features of the feature fusion submodule at the previous upsampling layer as low-level features into the feature fusion submodule at the next upsampling layer;

[0016] The attention features output by the attention modules corresponding to each downsampling module are input as high-level feature skip layers into the feature fusion submodule corresponding to the scale.

[0017] Furthermore, the attention module is a local contrast attention module, and the local contrast attention module is used to complete the following operations:

[0018] The directional gradients of the first input feature of the local contrast attention module in eight different directions are calculated respectively by the first convolution kernel at a given scale, and the directional gradients in opposite directions are multiplied to obtain multiple difference metrics at a given scale:

[0019]

[0020] Among them, d n,Nl represents the difference metric at a given scale, l represents different scales, l = 1, 2, ..., k, F in1 Represents the first input feature, conv i represents the first convolution kernel, i=1,2,...,8;

[0021] Compute the contrast saliency map (CSM) at a given scale from the individual difference metrics at a given scale Nl :

[0022]

[0023] Perform residual connection between the contrast saliency map at different scales and the first input feature to obtain the contrast saliency feature F CSM :

[0024]

[0025] Calculate the channel attention map of the contrast-significant feature and the spatial attention map of the contrast-significant feature respectively, and calculate the local contrast attention F based on the channel attention map and the spatial attention map LCAM :

[0026]

[0027] Among them, M c Channel attention map representing contrast-significant features, M s A spatial attention map representing contrast-significant features, represents the element-wise multiplication operation, Represents element addition operation;

[0028] Wherein, the first convolution kernel is expressed as:

[0029] λ i =-1,λ else =0,i=1,2,...,8

[0030] The first convolution kernel is divided into 9 sub-blocks with the same number of rows and columns, and the number of rows and columns in the first convolution kernel at different scales is different.

[0031] Further, the respectively calculating the channel attention map of the contrast significant feature and the spatial attention map of the contrast significant feature comprises:

[0032] M c =σ(FC(GAP(con 1×1 (F CSM (F in1 )))))

[0033] M s =σ(con 1×1 (F CSM (F in1 )))

[0034] Among them, σ represents the activation function, FC is the fully connected operation, GAP represents the average pooling operation, and con 1×1 is a convolution operation with a kernel size of 1×1.

[0035] Furthermore, the multi-scale hierarchical feature integration module is used to complete the following operations:

[0036] Perform convolution operations of different scales on the second input feature input into the multi-scale hierarchical feature integration module to obtain local features of different scales:

[0037] F p =con 1×1 (F in2 )

[0038] F S =con 1×1 (con 3×3 (F in2 ))

[0039] Among them, F p represents the first convolution sub-feature, F s Represents the second convolution sub-feature, con 1×1 is a convolution operation with a kernel size of 1×1, 3×3 is a convolution operation with a kernel size of 3×3;

[0040] The second input feature is equally divided into N feature map subsets along the channel dimension:

[0041] F z ∈R C / N×H×W (where z∈{1,2,...N})

[0042] Among them, C, H and W represent channel, height and width respectively;

[0043] Perform dilated convolutions with different dilation rates on each feature map subset, and connect different groups of dilated convolutions in layers to obtain multiple convolution feature map subsets:

[0044]

[0045] Among them, d Z is a 3×3 convolution kernel with a dilation rate of 2 Z , concat represents the connection operation;

[0046] Each convolution feature map subset is concatenated along the channel dimension, and the following operations are performed to obtain the total feature F of the dilated convolution layer with global features of different scales: D :

[0047]

[0048] Among them, ReLU represents the rectified linear unit, B represents the batch normalization operation, Represents the features obtained by concatenating each convolution feature map subset along the channel dimension;

[0049] The following operations are performed on the first convolution sub-feature, the second convolution sub-feature and the total feature of the dilated convolution layer to obtain the first output feature F of the multi-scale hierarchical feature integration module: out1 :

[0050] F out1 =ReLU(B(Con 1×1 (concat(F p ,F S ,F D )))).

[0051] Furthermore, the feature fusion submodule is a dynamic feature cross perception module, and the dynamic feature cross perception module is used to complete the following operations:

[0052] The high-level features F of the dynamic feature cross-perception module are input H Do the following:

[0053]

[0054] in, Represents the high-level features after transformation, TPConv 3×3,2,C / 2 represents a transposed convolution operation to double the high-level feature space size and halve the channel size;

[0055] The low-level features F that are input into the dynamic feature cross-perception module L The global fusion feature F is obtained by concatenating the transformed high-level features in the channel dimension Cat ;

[0056] The bottleneck structure is used to perform the following operations on the global fusion feature to obtain the fusion feature F that gives global information: G :

[0057]

[0058] Among them, PBR represents the continuous point-by-point convolution layer, batch normalization layer and ReLU activation function, GAP is the global average pooling, σ represents the activation function, Represents element-wise multiplication operation;

[0059] The bottleneck structure is used to further extract spatial details and edge information from low-level features and inject them into deep features to obtain high-level features F' injected with low-level information. H :

[0060]

[0061] An element addition operation is performed on the fusion feature endowed with global information and the high-level feature injected with low-level information to obtain the output feature of the dynamic feature cross-perception module.

[0062] Furthermore, the backbone network includes a cascaded first downsampling module, a second downsampling module and a third downsampling module, the feature output by the first downsampling module is a first downsampling infrared image feature, and the feature output by the second downsampling module is a second downsampling infrared image feature.

[0063] Furthermore, the feature fusion module includes a cascaded first feature fusion submodule and a second feature fusion submodule.

[0064] The attention features of different scales output by each attention module are input into the feature fusion module, including:

[0065] The attention features output by the attention module corresponding to the multi-scale hierarchical feature integration module are used as low-level features, and the attention features output by the attention module corresponding to the second downsampling module are used as high-level features, and are input into the first feature fusion submodule;

[0066] The output features of the first feature fusion submodule are used as low-level features, and the attention features output by the attention module corresponding to the first downsampling module are used as high-level features and input into the second feature fusion submodule.

[0067] Furthermore, before inputting the initial infrared image input features into a preset backbone network for feature encoding, the method includes:

[0068] Acquire infrared images containing small targets;

[0069] The infrared image is input into a stem module, so as to perform channel expansion on the infrared image through the stem module and increase the characterization capability of the input features, thereby obtaining initial infrared image input features.

[0070] The embodiment of the present application provides a multi-scale local contrast attention infrared small target detection method, which uses a multi-scale hierarchical feature integration module to perform feature fusion on local features and global features of multiple scales to fully utilize the global context information of small targets, which helps to estimate the correlation between pixels in a large range and effectively process infrared small targets of various sizes and shapes. The initial infrared image input features, downsampled infrared image features and the integrated infrared image features obtained by the backbone network are respectively input into the feature enhancement module, and the attention features based on channels and space can be obtained through the attention module of the feature enhancement module. Finally, the fused infrared image features are obtained through the feature fusion module, which can fully utilize the global information of high-level features and the details that low-level features are sensitive to small targets, and more accurately segment infrared small targets. The method has high generalization and robustness, and effectively segments infrared small targets.

[0071] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only used for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. In the accompanying drawings:

[0073] Figure 1 A flowchart of a multi-scale local contrast attention infrared small target detection method according to an embodiment of the present invention;

[0074] Figure 2 It is a schematic diagram of the overall architecture of a multi-scale local contrast attention network according to an embodiment of the present invention;

[0075] Figure 3 A schematic diagram of a network architecture of a contrast-significant module according to an embodiment of the present invention;

[0076] Figure 4 A schematic diagram of the overall network architecture of the local contrast attention module according to an embodiment of the present invention;

[0077] Figure 5 A schematic diagram of the network architecture of a multi-scale hierarchical feature integration module according to an embodiment of the present invention;

[0078] Figure 6 Schematic diagram of the network architecture of the dynamic feature cross-perception module of an embodiment of the present invention. DETAILED DESCRIPTION

[0079] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0080] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined.

[0081] Figure 1 A flowchart of a multi-scale local contrast attention infrared small target detection method according to an embodiment of the present invention is shown, referring to Figure 1 It can be seen that a multi-scale local contrast attention infrared small target detection method of the present invention includes the following steps:

[0082] S1. Inputting the initial infrared image input features into a preset backbone network for feature encoding; wherein the backbone network includes a plurality of cascaded downsampling modules and a multi-scale hierarchical feature integration module connected to the last layer of downsampling modules, so as to obtain at least one downsampled infrared image feature through at least one downsampling module in the cascaded multiple downsampling modules, and performing local feature extraction and global feature extraction of different scales on the sampling results of the last layer of downsampling modules through the multi-scale hierarchical feature integration module, and performing feature fusion on the obtained local features and global features of multiple scales to obtain integrated infrared image features;

[0083] In a specific embodiment of the present invention, before inputting the obtained initial infrared image input features into a preset backbone network for feature encoding, the method includes: acquiring an infrared image containing a small target; inputting the infrared image into a stem module to perform channel expansion on the infrared image through the stem module and increase the representation capability of the input features to obtain the initial infrared image input features.

[0084] S2, respectively inputting at least one downsampled infrared image feature and the integrated infrared image feature into a feature enhancement module; wherein the feature enhancement module comprises a plurality of attention modules for processing each downsampled infrared image feature and for processing the integrated infrared image feature, and the attention module obtains an attention feature by fusing a multi-scale channel attention feature and a multi-scale spatial attention feature on the input feature;

[0085] In an embodiment of the present invention, the downsampled infrared image features and the integrated infrared image features are connected to a feature fusion module through a feature enhancement module in a skip-layer connection manner so that high-level features can be fused with low-level features, making full use of the global information of high-level features and the details of low-level features that are sensitive to small targets, and more accurately dividing infrared small targets.

[0086] S3, respectively inputting the attention features of different scales output by each attention module into the feature fusion module, fusing the input attention features of different scales through the feature fusion module to strengthen the feature representation of the target feature, and obtaining the fused infrared image feature;

[0087] The embodiment of the present invention performs iterative upsampling level by level through a feature fusion module, and fuses high-level features and low-level features in each level of upsampling to strengthen the feature representation of the target feature, thereby obtaining a fused infrared image feature;

[0088] S4, inputting the fused infrared image features into a prediction module to obtain a result image of infrared small target segmentation in the infrared image.

[0089] In the embodiment of the present invention, the prediction module uses 1x1 convolution to perform prediction, and obtains a result image of fine segmentation of infrared small targets through a convolution segmentation head. The number of output image channels is 1, and the size is the original image size.

[0090] Furthermore, in an embodiment of the present invention, the feature fusion module includes at least one cascaded feature fusion submodule; the inputting the attention features of different scales output by each attention module into the feature fusion module specifically includes: inputting the attention features output by the attention module corresponding to the multi-scale hierarchical feature integration module as low-level features into the bottom-level feature fusion submodule in the feature fusion module, and inputting the output features of the feature fusion submodule of the previous upsampling level as low-level features into the feature fusion submodule of the next upsampling level; and inputting the attention features output by the attention modules corresponding to each downsampling module as high-level features to the feature fusion submodule corresponding to the scale. In an embodiment of the present invention, the number of feature fusion submodules based on the scale of the feature can be the total number of downsampling modules minus one.

[0091] Furthermore, in order to better implement the infrared small target detection method of the present invention, the present invention proposes a multi-scale local contrast attention network (MLCMANet), such as Figure 2 As shown in Figure 2, a local contrastive attention module (LCAM) is designed as Figure 4 As shown; the multi-scale hierarchical feature integration module (MHFI) is as follows Figure 5 As shown in the figure, the dynamic feature cross perception module (DFCS) is as follows Figure 6 The following is a further detailed introduction of a multi-scale local contrast attention infrared small target detection method of the present invention in conjunction with the accompanying drawings.

[0092] Further, by Figure 2 It can be seen that the multi-scale local contrast attention network achieves target segmentation through a U-shaped encoder-decoder structure, which consists of five parts: Stem block, backbone network, feature enhancement module, feature fusion module, and prediction module.

[0093] Among them, the backbone network in the specific embodiment of the present invention includes a cascaded first downsampling module, a second downsampling module and a third downsampling module, the feature output by the first downsampling module is the first downsampling infrared image feature, and the feature output by the second downsampling module is the second downsampling infrared image feature.

[0094] The corresponding feature fusion module includes a cascaded first feature fusion submodule and a second feature fusion submodule. The process of inputting the attention features of different scales output by each attention module into the feature fusion module includes: taking the attention features output by the attention module corresponding to the multi-scale hierarchical feature integration module as low-level features, and taking the attention features output by the attention module corresponding to the second downsampling module as high-level features, and inputting them into the first feature fusion submodule; taking the output features of the first feature fusion submodule as low-level features, and taking the attention features output by the attention module corresponding to the first downsampling module as high-level features, and inputting them into the second feature fusion submodule.

[0095] The following is combined with Figure 3 -Attached Figure 6 Each functional module in the embodiment of the present invention is further introduced in detail.

[0096] Furthermore, in a specific embodiment of the present invention, the attention module is a local contrast attention module, and the network architecture of the local contrast attention module is as follows: Figure 3 , Figure 4 As shown in Figure 2, local contrast measurement has been shown to be effective in enhancing small target features and suppressing background. We introduce contrast measurement into the attention we designed to calculate multi-scale contrast saliency maps.

[0097] Specifically, the local contrast attention module is used to complete the following operations:

[0098] S11, respectively calculating the directional gradients of the first input feature of the local contrast attention module in eight different directions through the first convolution kernel at a given scale, and multiplying the directional gradients in opposite directions to obtain multiple difference metrics at a given scale:

[0099]

[0100] Among them, d n,Nl represents the difference metric at a given scale, l represents different scales, l = 1, 2, ..., k, F in1 Represents the first input feature, conv i represents the first convolution kernel, i=1,2,...,8;

[0101] It should be noted that, in an embodiment of the present invention, the first input feature of the first attention module is the initial infrared image input feature, the first input feature of the second attention module is the downsampled infrared image feature, and the first input feature of the third attention module is the integrated infrared image feature.

[0102] Furthermore, the present invention determines the scale of the convolution according to the number of rows and columns of the first convolution kernel. Specifically, the first convolution kernel is expressed as:

[0103]

[0104] It can be seen from the formula of the first convolution kernel that the first convolution kernel is divided into 9 sub-blocks with the same number of rows and columns, which has eight convolution kernels with fixed parameters. The convolution kernel is equally divided into nine sub-blocks, which are arranged clockwise, and the number of rows and columns in the first convolution kernel at different scales is different. The middle value of each convolution kernel is 1. The present invention uses the scale of the first convolution kernel with different numbers of rows and columns to perform the convolution operation, and uses the convolution kernels at multiple different scales to respectively calculate the difference metric at a given scale.

[0105] S12. Calculate the contrast saliency map CSM at a given scale from each difference metric at a given scale Nl :

[0106]

[0107] Perform residual connection between the contrast saliency map at different scales and the first input feature to obtain the contrast saliency feature F CSM :

[0108]

[0109] S13, respectively calculating the channel attention map of the contrast-significant feature and the spatial attention map of the contrast-significant feature, and calculating the local contrast attention F based on the channel attention map and the spatial attention map LCAM :

[0110]

[0111] Among them, M c Channel attention map representing contrast-significant features, Ms A spatial attention map representing contrast-significant features, represents the element-wise multiplication operation, represents element addition operation, local contrast attention F LCAM This is the output feature of the local contrastive attention module.

[0112] Among them, the contrast-significant features contain specific contrast information in different channels and positions. In order to properly focus on and mine these channel and spatial information and improve the network's ability to extract key features, the present invention specifically designs a local contrast attention module. Figure 4 As shown in Figure 1, it consists of a contrast channel attention module and a contrast space attention module. Figure 3 The contrast salient module CSMblock shown in computes the contrast salient feature FCSM.

[0113] Specifically, the first input feature is:

[0114] F in1 ∈R C×H×W (6)

[0115] Where C, H, and W represent channel, height, and width, respectively. The contrast channel attention module provides a one-dimensional channel attention map M c ∈R C×1×1 , the contrast spatial attention module provides a two-dimensional spatial attention map M s ∈R 1×H×W , and multiply the two attention weights by the original image respectively and then add them together to output the total attention. The specific steps are as follows:

[0116]

[0117] Therefore, the local contrastive attention F LCAM It can also be expressed as:

[0118]

[0119] In an embodiment of the present invention, for an input feature map, a local contrastive attention module (LCAM) dynamically learns the attention weights of the channel dimension and the spatial dimension of the input feature map simultaneously by explicitly designing a parallel attention method, thereby obtaining an enhanced feature representation.

[0120] Specifically, each channel of the multi-channel feature map is a detector of features of different scales of contrast. The attention mechanism designed in the present invention explores the relationship between channels and identifies the most influential feature maps. Specifically, for the first input feature of the input, the number of channels of the cascade map obtained by the contrast significant module CSMblock is first compressed to C using a 1*1 convolution, and then global average pooling (GAP) is used to obtain channel-level attention. After obtaining the channel description, the channel description is placed in a fully connected network (FC) to obtain a channel attention map M of the contrast significant feature. c In order to reduce the parameters in the fully connected network (FC), the hidden layer activation size is set to R C / r*1*1 , where r is the reduction rate.

[0121] Therefore, the channel attention map for calculating the contrast salient feature is specifically:

[0122] M c =σ(FC(GAP(con 1×1 (F CSM (F in1 ))))) (10)

[0123] Among them, σ represents the activation function, FC is the fully connected operation, GAP represents the average pooling operation, and con 1×1 is a convolution operation with a kernel size of 1×1.

[0124] Furthermore, in infrared small target detection, the detailed location of small targets plays a vital role. Compared with channel attention, spatial attention focuses on spatial information, which is equally important as another branch. The present invention further designs a spatial attention module to refine the feature map from the pixel level to improve the feature representation of small target contrast in the feature map. As with channel attention, for the first input feature of the input, the cascade map obtained by the contrast significant module CSMblock uses a 1*1 convolution to compress the number of channels to one dimension to obtain a one-dimensional feature map, and the sigmoid function is used to scale it to obtain a spatial attention map M of the contrast significant feature. s The calculation process can be summarized as follows:

[0125] Therefore, the spatial attention map of the contrast salient feature is calculated as follows:

[0126] M s =σ(con 1×1 (F CSM (F in1 ))) (11)

[0127] Further, refer to Figure 5The multi-scale hierarchical feature integration module (MHFI) of the embodiment of the present invention is used to complete the following operations:

[0128] S21, performing convolution operations of different scales on the second input feature input to the multi-scale hierarchical feature integration module to obtain local features of different scales:

[0129] F p =con 1×1 (F in2 ) (12)

[0130] F S =con 1×1 (con 3×3 (F in2 )) (13)

[0131] Among them, F p represents the first convolution sub-feature, F s represents the second convolution sub-feature, F p represents the first convolution sub-feature and F s The second convolution sub-feature is a local feature with different scales. 1×1 is a convolution operation with a kernel size of 1×1, 3×3 is a convolution operation with a kernel size of 3×3; in addition, in a specific embodiment of the present invention, the second input feature is the infrared image feature input by the last layer downsampling module to the multi-scale hierarchical feature integration module,

[0132] S22, dividing the second input feature into N feature map subsets equally along the channel dimension:

[0133] F z ∈R C / N×ZH×W (where z∈{1,2,...N}) (14)

[0134] Specifically, the present invention uses the Split operator to convert F in2 ∈R C×W×H The feature subsets are divided into N subsets along the channel dimension. In the specific embodiment of the present invention, N is 4.

[0135] S23, performing dilated convolutions with different dilation rates on each feature map subset respectively, and connecting different groups of dilated convolutions in layers to obtain multiple convolution feature map subsets:

[0136]

[0137] Wherein, concat represents a concatenation operation; in a specific embodiment of the present invention, each set of feature graph subsets can be denoted as F z ∈R C / N×H×W , each group Fz Corresponding to a dilated convolution, the dilation rate is set to 2, 4, 8 and 16, gradually expanding the receptive field without increasing the number of parameters. Different groups of dilated convolutions are layered and connected in a residual-like manner. In terms of structure, the first group of dilated convolutions extracts features from F1 and obtains Next, the second set of feature map subsets F2 and the output of the previous set The concatenation is then fed into the second set of dilated convolutions. This process is repeated many times until all input features are processed. This process is shown in formula (15) and the attached Figure 5 shown.

[0138] S24, concatenate each convolution feature map subset along the channel dimension, and perform the following operations to obtain the total feature F of the dilated convolution layer with global features of different scales: D :

[0139]

[0140] Among them, ReLU represents the rectified linear unit, B represents the batch normalization operation, Represents the features obtained by concatenating each convolution feature map subset along the channel dimension;

[0141] S25, performing the following operations on the first convolution sub-feature, the second convolution sub-feature and the total feature of the dilated convolution layer to obtain a first output feature F of the multi-scale hierarchical feature integration module out1 :

[0142] F out1 =ReLU(B(Con 1×1 (concat(F p ,F S ,F D )))) (17)

[0143] Furthermore, in infrared small target detection, high-level feature maps carry rich semantic information, while low-level feature maps contain key details and edge information. Due to multiple downsampling processes, high-level features may lose details that are sensitive to small targets, and low-level features may lack sufficient global context. In order to promote the effective interaction of high-level and low-level features, the present invention proposes a module called dynamic feature cross-perception (DFCS). Among them, the feature fusion submodule is the dynamic feature cross-perception module. The dynamic feature cross-perception module (DFCS) uses global information to adaptively fuse the spatial details in the low-level features with the semantic information in the high-level features. Figure 6 The dynamic feature cross-sensing module (DFCS) of the embodiment of the present invention is used to complete the following operations:

[0144] S31, input the high-level feature F of the dynamic feature cross perception module H Do the following:

[0145]

[0146] in, Represents the high-level features after transformation, TPConv 3×3,2,C / 2 Represents a transposed convolution operation to double the spatial size of high-level features and halve the channel size; in an embodiment of the present invention, in order to match low-level features, high-level features first pass through a 3×3 transposed convolution layer to double the spatial size and halve the channel size.

[0147] S32, inputting the low-level feature F of the dynamic feature cross perception module L The global fusion feature F is obtained by concatenating the transformed high-level features in the channel dimension Cat ;

[0148]

[0149] S33, using the bottleneck structure to perform the following operations on the global fusion feature to obtain a fusion feature F that is endowed with global information G :

[0150]

[0151] Among them, PBR represents the continuous point-by-point convolution layer, batch normalization layer and ReLU activation function, GAP is the global average pooling, σ represents the activation function, represents an element-wise multiplication operation; in the embodiment of the present invention, the global fusion feature F Cat It is input into the bottleneck structure composed of two point-by-point convolution layers, which reduces the number of parameters and the amount of computation while maintaining the efficient nonlinear feature extraction capability of the fused features. The fused features are calibrated by global channel information to strengthen the feature representation. Subsequently, through point-by-point convolution, features are selected according to the importance of the feature map to obtain F. G , guiding the convolutional layer to retain important features while removing parts with less information.

[0152] S34, use the bottleneck structure to further extract spatial details and edge information from low-level features, inject them into deep features, and obtain high-level features F' injected with low-level information H :

[0153]

[0154] In the embodiment of the present invention, the bottleneck structure is used to further extract spatial details and edge information from the low-level features and inject them into the deep features to obtain the high-level features F' injected with the low-level information.H .

[0155] S35, performing element addition operation on the fusion feature endowed with global information and the high-level feature injected with low-level information to obtain the output feature of the dynamic feature cross-perception module:

[0156]

[0157] Among them, F out2 Represents the output features of the dynamic feature cross-perception module.

[0158] The embodiment of the present invention proposes an MLCMANet to realize SIRST detection. In order to further overcome the challenges brought by small infrared targets, we design and utilize three key modules in the proposed MLCMANet: Local Contrast Attention Module (LCAM), Multi-scale Hierarchical Feature Integration Module (MHFI), Dynamic Feature Cross-perception Module (DFCS). LCAM aims to enhance small target areas and weaken uninteresting areas. MHFI fuses contextual information from multiple scales for better feature representation. DFCS fuses low-level and deep semantics to retain more information about small targets. Comparison with the most advanced methods and ablation studies verify the superiority of the constructed model and the effectiveness of each component.

[0159] Furthermore, the embodiment of the present invention also experimentally verifies the above-mentioned multi-scale local contrast attention infrared small target detection method. Specifically, it is implemented using PyTorch. Initially, the image and label are resized to 512×512, and then 480×480 crops are randomly cropped as network input. In addition, all networks are trained using the Soft-IoU loss function, and Adam is selected as the optimizer. The initial learning rate is set to 0.001, and the weight decay coefficient is set to 1×10-4. The batch size is set to 8, and the number of iterations is 300 epochs. In terms of hardware, all methods are implemented on Windows 10 equipped with Intel i7-12700 CPU and GeForce RTX 4090D. For the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited to the order of the actions described, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0160] Furthermore, the input image is firstly passed through the stem to expand the image channels to 16 to increase the network's ability to represent the input features. The stem block consists of two 3x3 convolutions, the first of which has an input channel of 1, an output channel of 16, and an output size of 480×480; the second of which has an input channel of 16, an output channel of 16, and an output size of 480×480.

[0161] The backbone network is divided into three downsampling stages and a multi-scale hierarchical feature integration module, which is connected to the last stage. Each downsampling stage contains four residual blocks, and each residual block contains two 3×3 convolutions. The number of input channels in stage one is 16, the number of output channels is 16, and the output size is 480×480; the number of input channels in stage two is 16, the number of output channels is 32, and the output size is 240×240; the number of input channels in stage three is 32, the number of output channels is 64, and the output size is 120×120;

[0162] The feature enhancement module contains three local contrast attention modules, which are connected to the backbone network stage 1, stage 2, and hierarchical feature integration module respectively, and are used to extract deep target features of the cross-layer guidance network.

[0163] The feature fusion module contains two dynamic feature cross-perception modules. The first dynamic feature cross-perception module receives the feature map from the multi-scale hierarchical feature integration module and the local contrast attention module after stage 2. The second dynamic feature cross-perception module receives the feature map from the previous dynamic feature cross-perception module and the local contrast attention module after stage 1.

[0164] Finally, the prediction module uses 1x1 convolution to perform prediction, and obtains the fine segmentation result image of the infrared small target through the convolution segmentation head. The number of output image channels is 1, and the size is the original image size.

[0165] This method achieves more accurate segmentation because the local contrast attention module (LCAM) is specially designed to enhance the representation of contrast features for small infrared and dim targets. It focuses on the target contrast features in the network and adaptively extracts spatial and channel contrast information features.

[0166] A multi-scale local contrast attention infrared small target detection method provided by an embodiment of the present application integrates the advantages of end-to-end convolutional neural network (CNN) and interpretable local contrast measurement, and can effectively extract the global semantic information and features of small targets. Specifically, a local contrast attention module (LCAM) is designed to make full use of the spatial and channel contrast information of the image, enhance the contrast information of small targets, and suppress areas of no interest. Secondly, we designed a multi-scale hierarchical feature integration module (MHFI) to make full use of the global contextual information of small targets, which helps to estimate the correlation between pixels in a large range and effectively handle infrared small targets of various sizes and shapes. Finally, we designed a dynamic feature cross-perception module (DFCS) to guide the expression of deep features through shallow features, which helps to promote the bottom-up interactive fusion of low-level detail information and high-level semantic information. This method has high generalization and robustness, and effectively segments small infrared targets.

[0167] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, any one of the claimed embodiments may be used in any combination.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-scale local contrast attention infrared small target detection method, characterized in that: The method comprises: Inputting the initial infrared image input features into a preset backbone network for feature encoding; wherein the backbone network includes a plurality of cascaded downsampling modules and a multi-scale hierarchical feature integration module connected to the last layer of downsampling modules, so as to obtain at least one downsampled infrared image feature through at least one downsampling module in the cascaded multiple downsampling modules, and performing local feature extraction and global feature extraction of different scales on the sampling results of the last layer of downsampling modules through the multi-scale hierarchical feature integration module, and performing feature fusion on the obtained local features and global features of multiple scales to obtain integrated infrared image features; Inputting at least one downsampled infrared image feature and the integrated infrared image feature into a feature enhancement module respectively; wherein the feature enhancement module comprises a plurality of attention modules for processing each downsampled infrared image feature and for processing the integrated infrared image feature respectively, and the attention module obtains an attention feature by fusing a multi-scale channel attention feature and a multi-scale spatial attention feature on the input feature; The attention features of different scales output by each attention module are respectively input into the feature fusion module, and the input attention features of different scales are fused by the feature fusion module to strengthen the feature representation of the target feature and obtain the fused infrared image feature; The fused infrared image features are input into a prediction module to obtain a result image of infrared small target segmentation in the infrared image.

2. The method according to claim 1, characterized in that The feature fusion module includes at least one cascaded feature fusion submodule; The attention features of different scales output by each attention module are input into the feature fusion module, including: Input the attention features output by the attention module corresponding to the multi-scale hierarchical feature integration module as low-level features into the feature fusion submodule at the bottom layer of the feature fusion module, and input the output features of the feature fusion submodule at the previous upsampling layer as low-level features into the feature fusion submodule at the next upsampling layer; The attention features output by the attention modules corresponding to each downsampling module are input as high-level feature skip layers into the feature fusion submodule corresponding to the scale.

3. The method according to claim 1 or 2, characterized in that: The attention module is a local contrast attention module, which is used to complete the following operations: The directional gradients of the first input feature of the local contrast attention module in eight different directions are calculated respectively by the first convolution kernel at a given scale, and the directional gradients in opposite directions are multiplied to obtain multiple difference metrics at a given scale: Among them, d n,Nl represents the difference metric at a given scale, l represents different scales, l = 1, 2, ..., k, F in1 Represents the first input feature, conv i represents the first convolution kernel, i=1,2,...,8; Compute the contrast saliency map (CSM) at a given scale from the individual difference metrics at a given scale Nl : Perform residual connection between the contrast saliency map at different scales and the first input feature to obtain the contrast saliency feature F CSM : Calculate the channel attention map of the contrast-significant feature and the spatial attention map of the contrast-significant feature respectively, and calculate the local contrast attention F based on the channel attention map and the spatial attention map LCAM : Among them, M c Channel attention map representing contrast-significant features, M s A spatial attention map representing contrast-significant features, represents the element-wise multiplication operation, Represents element addition operation; Wherein, the first convolution kernel is expressed as: The first convolution kernel is divided into 9 sub-blocks with the same number of rows and columns, and the number of rows and columns in the first convolution kernel at different scales is different.

4. The method according to claim 3, characterized in that The respectively calculating the channel attention map of the contrast-significant feature and the spatial attention map of the contrast-significant feature comprises: M c =σ(FC(GAP(con 1×1 (F CSM (F in 1))))) M s =σ(con 1×1 (F CSM (F in1 ))) Among them, σ represents the activation function, FC is the fully connected operation, GAP represents the average pooling operation, and con 1×1 is a convolution operation with a kernel size of 1×1.

5. The method according to claim 1 or 2, characterized in that: The multi-scale hierarchical feature integration module is used to complete the following operations: Perform convolution operations of different scales on the second input feature input into the multi-scale hierarchical feature integration module to obtain local features of different scales: F p =con 1×1 (F in2 ) F S =with 1×1 (with 3×3 (F in2 )) Among them, F p represents the first convolution sub-feature, F s Represents the second convolution sub-feature, con 1×1 is a convolution operation with a kernel size of 1×1, 3×3 is a convolution operation with a kernel size of 3×3; The second input feature is equally divided into N feature map subsets along the channel dimension: F z ∈R C / N×H×W (where z∈{1,2,...N}) Among them, C, H and W represent channel, height and width respectively; Perform dilated convolutions with different dilation rates on each feature map subset, and connect different groups of dilated convolutions in layers to obtain multiple convolution feature map subsets: Among them, d Z is a 3×3 convolution kernel with a dilation rate of 2 Z , concat represents the connection operation; Each convolution feature map subset is concatenated along the channel dimension, and the following operations are performed to obtain the total feature F of the dilated convolution layer with global features of different scales: D : Among them, ReLU represents the rectified linear unit, B represents the batch normalization operation, Represents the features obtained by concatenating each convolution feature map subset along the channel dimension; The following operations are performed on the first convolution sub-feature, the second convolution sub-feature and the total feature of the dilated convolution layer to obtain the first output feature F of the multi-scale hierarchical feature integration module: out1 : F out1 =ReLU(B(With 1×1 (concat(F p ,F S ,F D ))))。 6. The method according to claim 2, characterized in that The feature fusion submodule is a dynamic feature cross perception module, and the dynamic feature cross perception module is used to complete the following operations: The high-level features F of the dynamic feature cross-perception module are input H Do the following: in, Represents the high-level features after transformation, TPConv 3×3,2,C / 2 represents a transposed convolution operation to double the high-level feature space size and halve the channel size; The low-level features F that are input into the dynamic feature cross-perception module L The global fusion feature F is obtained by concatenating the transformed high-level features in the channel dimension Cat ; The bottleneck structure is used to perform the following operations on the global fusion feature to obtain the fusion feature F that gives global information: G : Among them, PBR represents the continuous point-by-point convolution layer, batch normalization layer and ReLU activation function, GAP is the global average pooling, σ represents the activation function, Represents element-wise multiplication operation; The bottleneck structure is used to further extract spatial details and edge information from low-level features and inject them into deep features to obtain high-level features F' injected with low-level information. H : An element addition operation is performed on the fusion feature endowed with global information and the high-level feature injected with low-level information to obtain the output feature of the dynamic feature cross-perception module.

7. The method according to claim 1, characterized in that The backbone network includes a cascaded first down-sampling module, a second down-sampling module and a third down-sampling module. The features output by the first down-sampling module are first down-sampled infrared image features, and the features output by the second down-sampling module are second down-sampled infrared image features.

8. The method according to claim 7, characterized in that The feature fusion module includes a cascaded first feature fusion submodule and a second feature fusion submodule. The attention features of different scales output by each attention module are input into the feature fusion module, including: The attention features output by the attention module corresponding to the multi-scale hierarchical feature integration module are used as low-level features, and the attention features output by the attention module corresponding to the second downsampling module are used as high-level features, and are input into the first feature fusion submodule; The output features of the first feature fusion submodule are used as low-level features, and the attention features output by the attention module corresponding to the first downsampling module are used as high-level features and input into the second feature fusion submodule.

9. The method according to claim 1, characterized in that: Before inputting the initial infrared image input features into a preset backbone network for feature encoding, the method comprises: Acquire infrared images containing small targets; The infrared image is input into a stem module, so as to perform channel expansion on the infrared image through the stem module and increase the characterization capability of the input features, thereby obtaining initial infrared image input features.

Images