A multiscale local contrast attention infrared small target detection method

Abstract

The application provides a multiscale local contrast attention infrared small target detection method, which comprises the following steps: inputting an initial infrared image feature into a preset backbone network for feature coding, inputting at least one obtained down-sampling infrared image feature and an integrated infrared image feature into a feature enhancement module, wherein the feature enhancement module comprises a plurality of attention modules respectively used for processing each down-sampling infrared image feature and used for processing the integrated infrared image feature, the attention module obtains an attention feature by fusing a multiscale channel attention feature and a multiscale spatial attention feature of the input feature; inputting different scale attention features output by each attention module into a feature fusion module respectively to obtain a fused infrared image feature; and inputting the fused infrared image feature into a prediction module to obtain a result image of infrared small target segmentation in the infrared image. The method can effectively segment the infrared small target.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a multi-scale local contrast attention infrared small target detection method. BACKGROUND

[0002] Single-frame infrared small target (SIRST) detection is one of the key technologies of infrared search and tracking, and is widely used in early warning systems, precision guided weapons, maritime surveillance systems and other fields. The purpose of SIRST detection is to accurately locate the target in different infrared backgrounds. Unlike general targets, infrared small targets are usually located at a certain distance from the infrared detection system and usually occupy only a few pixels in the image. In addition, the signal-to-noise ratio of the infrared small target is relatively low, and there is a lack of sufficient texture details. This is due to atmospheric scattering, refraction, optical defocus and various noise sources. Therefore, how to quickly and accurately detect small targets is a challenging task.

[0003] In order to detect infrared small targets, there are currently two main methods, model-driven and data-driven. Model-driven can be further divided into three categories based on filters, local information and optimization. Among them, the local contrast measurement method is the most commonly used, which uses the gray level difference between the target and the neighborhood to calculate the contrast of each pixel in the image, effectively enhancing the target while suppressing the background. The model based on model-driven usually combines the professional knowledge of the field and the mathematical model, providing good interpretability. However, these methods require a lot of experience to design handcrafted features and adjust hyperparameters. They also often separate feature extraction, threshold segmentation and target detection steps. Therefore, the effectiveness of model-driven based methods is limited when encountering complex backgrounds and unknown target shapes.

[0004] In recent years, computer vision has seen rapid development. Learning infrared small target features in a data-driven manner has achieved good results. On the basis of data-driven, there are three types: detection-based, generative adversarial network-based and segmentation-based. Data-driven automatically learns features from a large amount of training data, eliminating the need for manual feature design. Compared with handcrafted features, this method can achieve more general feature extraction in complex backgrounds in an end-to-end manner.

[0005] Attention mechanisms can automatically learn and selectively focus on important information in the input in many areas of machine vision, thereby improving the performance and generalization ability of the model. However, infrared small targets usually exist in complex backgrounds, have low contrast, and their size is relatively small and the edges are blurred. The general attention mechanism will focus too much attention on non-small target areas, causing small target information to be ignored.

[0006] To solve the above problems, some researchers propose to integrate the model-driven idea into the data-driven model. ALCNet first introduces a local contrast measure in a deep learning network to enhance the contrast of small targets; on the one hand, RISTDnet, RDIAN and HoLoCoNet enhance the targets in the input image by using fixed convolution kernels of different sizes or local contrast weights before the image enters the deep network; however, these methods only perform local contrast enhancement at the beginning of the network, which can be regarded as providing priori for subsequent feature extraction, but ignores the interaction of contrast information between low-level and high-level features. On the other hand, MLCLNet, ALCLNet and LCAGNet try to modularize the local contrast measure, and then enhance the ability of the network to extract the contrast of small targets by combining shallow and deep features. However, the contrast measure module designed by them fails to fully exploit and utilize the contrast information on the channel and spatial features. The cascaded feature maps generated by the multi-scale contrast measurement module contain rich image feature information, helping the model better understand the image content. These methods choose to extract features only by maximum pooling or average pooling of multi-scale connected feature maps. This method may amplify the interference of bright noise in a complex background, and may lose subtle feature information that helps to distinguish weak small targets. SUMMARY

[0007] In view of the above problems, the present application is proposed to overcome the above problems.

[0008] The present application provides a multi-scale local contrast attention infrared small target detection method, the method comprising:

[0009] inputting an initial infrared image feature into a preset backbone network for feature encoding; wherein the backbone network comprises a plurality of cascaded down-sampling modules and a multi-scale hierarchical feature integration module connected to the last down-sampling module, so as to obtain at least one down-sampled infrared image feature through at least one down-sampling module in the plurality of cascaded down-sampling modules, and to respectively perform local feature extraction and global feature extraction of different scales on the sampling results of the last down-sampling module through the multi-scale hierarchical feature integration module, and to obtain integrated infrared image features by feature fusion of the obtained local features and global features of multiple scales;

[0010] respectively inputting at least one down-sampled infrared image feature and the integrated infrared image feature into a feature enhancement module; wherein the feature enhancement module comprises a plurality of attention modules respectively for processing each down-sampled infrared image feature and for processing the integrated infrared image feature, and the attention module obtains attention features by fusing multi-scale channel attention features and multi-scale spatial attention features of the input features;

[0011] The different scale attention features output by the respective attention modules are input into the feature fusion module, the different scale attention features input into the feature fusion module are fused by the feature fusion module to strengthen the feature representation of the target feature, and a fused infrared image feature is obtained;

[0012] The fused infrared image feature is input into the prediction module to obtain a result image of infrared small target segmentation in the infrared image.

[0013] Further, the feature fusion module comprises at least one feature fusion sub-module in cascade;

[0014] The respective attention modules output different scale attention features, which are input into the feature fusion module, including:

[0015] The attention features output by the attention modules corresponding to the multi-scale hierarchical feature integration module are input into the feature fusion sub-module at the bottom layer of the feature fusion module as low-level features, and the output features of the feature fusion sub-module at the previous up-sampling level are input into the feature fusion sub-module at the next up-sampling level as low-level features.

[0016] The attention features output by the attention modules corresponding to the respective down-sampling modules are input into the feature fusion sub-module corresponding in scale as high-level features.

[0017] Further, the attention module is a local contrast attention module, and the local contrast attention module is used to complete the following operations:

[0018] The direction gradients of the first input feature input into the local contrast attention module in eight different directions are calculated respectively by the first convolution kernel at a given scale, and the direction gradients located in opposite directions are multiplied to obtain a plurality of difference measures at the given scale:

[0019]

[0020] wherein d n,Nl represents the difference measure at the given scale, l represents different scales, l = 1, 2,..., k, F in1 represents the first input feature, conv i represents the first convolution kernel, i = 1, 2,..., 8.

[0021] The contrast saliency map CSM Nl at the given scale is calculated from each difference measure at the given scale.

[0022]

[0023] The contrast saliency feature F is obtained by connecting the contrast saliency map of different scales and the first input feature in residual connection CSM :

[0024]

[0025] The channel attention map of the contrast saliency feature and the spatial attention map of the contrast saliency feature are calculated respectively, and the local contrast attention F is calculated based on the channel attention map and the spatial attention map LCAM :

[0026]

[0027] Wherein, M c represents the channel attention map of the contrast saliency feature, M s represents the spatial attention map of the contrast saliency feature, represents element multiplication operation, represents element addition operation;

[0028] Wherein, the first convolution kernel is represented as:

[0029] λ i =-1, λ else =0, i=1, 2,..., 8

[0030] Wherein, the first convolution kernel is divided into 9 subblocks with the same row and column number, and the row and column numbers in the first convolution kernel of different scales are different.

[0031] Further, the channel attention map of the contrast saliency feature and the spatial attention map of the contrast saliency feature are calculated respectively, including:

[0032] M c =σ(FC(GAP(con 1×1 (F CSM (F in1 )))))

[0033] M s =σ(con 1×1 (F CSM (F in1 )))

[0034] Wherein, σ represents an activation function, FC is a full connection operation, GAP represents an average pooling operation, and con 1×1 is a convolution operation with a kernel size of 1x1.

[0035] Further, the multi-scale hierarchical feature integration module is used to complete the following operations:

[0036] performing convolution operations of different scales on the second input feature input to the multi-scale hierarchical feature integration module to obtain local features of different scales:

[0037] F p = con 1×1 (F in2 )

[0038] F S = con 1×1 (con 3×3 (F in2 )

[0039] wherein F p represents a first convolutional sub-feature, F s represents a second convolutional sub-feature, con 1×1 is a convolution operation with a kernel size of 1x1, and con 3×3 is a convolution operation with a kernel size of 3x3;

[0040] equally dividing the second input feature along the channel dimension into N feature map subsets:

[0041] F z ∈R C / N×H×W (where z∈{1,2,...N})

[0042] wherein C, H and W represent the channel, height and width, respectively;

[0043] respectively performing dilated convolution with different dilation rates on each feature map subset, and hierarchically connecting different groups of dilated convolution to obtain a plurality of convolution feature map subsets:

[0044]

[0045] wherein d Z is a 3x3 convolution kernel with a dilation rate of 2 Z , and concat represents a concatenation operation;

[0046] splicing each convolution feature map subset along the channel dimension, and performing the following operation to obtain a dilated convolution layer total feature F D with different scales:

[0047]

[0048] wherein ReLU represents a rectified linear unit, B represents a batch normalization operation, represents a feature obtained by splicing each convolution feature map subset along the channel dimension;

[0049] performing the following operation on the first convolutional sub-feature, the second convolutional sub-feature and the expanded convolutional layer total feature to obtain a first output feature F of the multi-scale hierarchical feature integration module out1 :

[0050] F out1 =ReLU(B(Con 1×1 (concat(F p ,F S ,F D ))).

[0051] Further, the feature fusion sub-module is a dynamic feature cross perception module, and the dynamic feature cross perception module is used to complete the following operation:

[0052] performing the following operation on the high-level feature F input into the dynamic feature cross perception module H :

[0053]

[0054] wherein, represents a modified high-level feature, TPConv 3×3,2,C / 2 represents a transpose convolution operation, so as to double the spatial size of the high-level feature and halve the channel size;

[0055] performing the following operation on the low-level feature F input into the dynamic feature cross perception module L and the modified high-level feature in the channel dimension to obtain a global fusion feature F Cat :

[0056] performing the following operation on the global fusion feature by using a bottleneck structure to obtain a fusion feature F endowed with global information G :

[0057]

[0058] wherein, PBR represents a continuous point-wise convolution layer, a batch normalization layer and a ReLU activation function, GAP is a global average pooling, σ represents an activation function, represents an element multiplication operation;

[0059] further extracting spatial details and edge information from the low-level feature by using a bottleneck structure, and injecting the spatial details and edge information into the deep-level feature to obtain a high-level feature F' injected with low-level information H :

[0060]

[0061] An element addition operation is performed on the fusion feature with global information and the high-level feature with injected low-level information, to obtain an output feature of the dynamic feature cross-perception module.

[0062] Further, the backbone network includes cascaded first, second and third down-sampling modules, the feature output by the first down-sampling module is a first down-sampled infrared image feature, and the feature output by the second down-sampling module is a second down-sampled infrared image feature.

[0063] Further, the feature fusion module includes cascaded first and second feature fusion sub-modules,

[0064] The different scale attention features output by the respective attention modules are input into the feature fusion module, including:

[0065] The attention features output by the attention modules corresponding to the multi-scale hierarchical feature integration module are taken as low-level features, and the attention features output by the attention modules corresponding to the second down-sampling module are taken as high-level features, which are input into the first feature fusion sub-module.

[0066] The output feature of the first feature fusion sub-module is taken as a low-level feature, and the attention feature output by the attention module corresponding to the first down-sampling module is taken as a high-level feature, which are input into the second feature fusion sub-module.

[0067] Further, before inputting the initial infrared image input feature into the preset backbone network for feature encoding, the method includes:

[0068] An infrared image containing small targets is acquired.

[0069] The infrared image is input into a stem module, so as to expand the channel of the infrared image through the stem module and increase the representation ability of the input feature, to obtain an initial infrared image input feature.

[0070] The method provided by the embodiment of the application is a multi-scale local contrast attention infrared small target detection method, which uses a multi-scale hierarchical feature integration module to perform feature fusion on local features and global features of multiple scales, fully utilizes the global context information of the small target, which helps to estimate the correlation between pixels in a large range and effectively process infrared small targets of various sizes and shapes, and the initial infrared image features obtained by the backbone network, the down-sampled infrared image features and the integrated infrared image features are input into a feature enhancement module respectively, the channel-based and spatial-based attention features can be obtained through the attention module of the feature enhancement module, and finally the fused infrared image features can be obtained through a feature fusion module, so that the global information of high-level features and the details of low-level features sensitive to small targets can be fully utilized to more accurately divide the infrared small targets. The method has high generalization and robustness, and effectively segments the infrared small targets.

[0071] The above description is only a summary of the technical scheme of the application, in order to enable the technical means of the application to be more clearly understood, and to be implemented according to the content of the specification, and in order to enable the above and other purposes, features and advantages of the application to be more obvious and easy to understand, the following specific embodiments of the application are described. BRIEF DESCRIPTION OF DRAWINGS

[0072] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not meant to limit the present application. In the drawings:

[0073] Figure 1 A flowchart of a multi-scale local contrast attention infrared small target detection method according to an embodiment of the application;

[0074] Figure 2 A schematic diagram of the overall architecture of a multi-scale local contrast attention network according to an embodiment of the application;

[0075] Figure 3 A schematic diagram of the network architecture of a contrast saliency module according to an embodiment of the application;

[0076] Figure 4 A schematic diagram of the overall network architecture of a local contrast attention module according to an embodiment of the application;

[0077] Figure 5 A schematic diagram of the network architecture of a multi-scale hierarchical feature integration module according to an embodiment of the application;

[0078] Figure 6 A schematic diagram of the network architecture of a dynamic feature cross perception module according to an embodiment of the application. DETAILED DESCRIPTION

[0079] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms without being limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thoroughly and completely understood, and will fully convey the scope of the present disclosure to those skilled in the art.

[0080] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. It should also be understood that terms such as those defined in general dictionaries should be understood as having meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined.

[0081] Figure 1 A flow chart of a multi-scale local contrast attention infrared small target detection method according to an embodiment of the present application is shown, referring to Figure 1 It can be seen that the multi-scale local contrast attention infrared small target detection method according to the present application comprises the following steps:

[0082] S1, input the initial infrared image feature into a preset backbone network for feature encoding; wherein the backbone network comprises a plurality of cascaded down-sampling modules and a multi-scale hierarchical feature integration module connected to the last down-sampling module, so as to obtain at least one down-sampled infrared image feature through at least one down-sampling module in the plurality of cascaded down-sampling modules, and to respectively perform local feature extraction and global feature extraction of different scales on the sampling results of the last down-sampling module through the multi-scale hierarchical feature integration module, and to obtain integrated infrared image features by feature fusion of the obtained local features and global features of multiple scales;

[0083] In a specific embodiment of the present application, before inputting the obtained initial infrared image feature into the preset backbone network for feature encoding, the method comprises: acquiring an infrared image containing a small target; inputting the infrared image into a stem module to expand the channels of the infrared image through the stem module and increase the representation ability of the input feature, and obtaining the initial infrared image feature.

[0084] S2, respectively input at least one down-sampled infrared image feature and the integrated infrared image feature into a feature enhancement module; wherein the feature enhancement module comprises a plurality of attention modules respectively used for processing each down-sampled infrared image feature and used for processing the integrated infrared image feature, and the attention module obtains attention features by fusing multi-scale channel attention features and multi-scale spatial attention features of the input features;

[0085] In the embodiment of the present application, the respective down-sampled infrared image features and the integrated infrared image features are connected to the feature fusion module through the feature enhancement module in a skip-layer connection manner, which can fuse high-level features and low-level features, fully utilize the global information of the high-level features and the details of the low-level features sensitive to small targets, and more accurately divide the infrared small targets.

[0086] S3, input the attention features of different scales output by the respective attention modules into the feature fusion module, fuse the input attention features of different scales through the feature fusion module to strengthen the feature representation of the target features, and obtain fused infrared image features;

[0087] In the embodiment of the present application, the feature fusion module is iteratively up-sampled, and the high-level features and the low-level features are fused in each up-sampling process to strengthen the feature representation of the target features, and the fused infrared image features are obtained.

[0088] S4, input the fused infrared image features into the prediction module to obtain a result image of the infrared small target segmentation in the infrared image.

[0089] In the embodiment of the present application, the prediction module uses 1x1 convolution for prediction, and the fine segmentation result image of the infrared small target is obtained through the convolution segmentation head. The output image channel number is 1, and the size is the original image size.

[0090] Further, in the embodiment of the present application, the feature fusion module includes at least one cascaded feature fusion sub-module; the input of the attention features output by the attention modules corresponding to the multi-scale hierarchical feature integration module into the feature fusion module as low-level features into the bottommost feature fusion sub-module of the feature fusion module, and the output features of the feature fusion sub-module of the previous up-sampling level as low-level features into the feature fusion sub-module of the next up-sampling level; the attention features output by the attention modules corresponding to the respective down-sampling modules as high-level features are input into the scale corresponding feature fusion sub-module in a skip-layer manner. In the embodiment of the present application, the number of feature fusion sub-modules based on the scale of the features can be the total number of down-sampling modules minus one.

[0091] Further, in order to better realize the infrared small target detection method of the present application, the present application proposes a multi-scale local contrast attention network (MLCMANet), as shown in Figure 2 , a local contrast attention module (LCAM) is designed, as shown in Figure 4 , a multi-scale hierarchical feature integration module (MHFI) is designed, as shown in Figure 5 , and a dynamic feature cross perception module (DFCS) is designed, as shown inFigure 6 A multiscale local contrast attention infrared small target detection method is further described in detail below with reference to the drawings.

[0092] Further, the multiscale local contrast attention network realizes target segmentation through a U-shaped encoder-decoder structure, which comprises five parts, namely, a stem block, a backbone network, a feature enhancement module, a feature fusion module and a prediction module. Figure 2 It can be known that the multiscale local contrast attention network realizes target segmentation through a U-shaped encoder-decoder structure, which comprises five parts, namely, a stem block, a backbone network, a feature enhancement module, a feature fusion module and a prediction module.

[0093] The backbone network in the embodiment of the present application comprises a first downsampling module, a second downsampling module and a third downsampling module connected in cascade, wherein the feature output by the first downsampling module is a first down-sampled infrared image feature, and the feature output by the second downsampling module is a second down-sampled infrared image feature.

[0094] The feature fusion module comprises a first feature fusion submodule and a second feature fusion submodule connected in cascade, wherein the attention features output by the attention modules corresponding to the multiscale hierarchical feature integration module are input into the first feature fusion submodule as low-level features, and the attention features output by the attention modules corresponding to the second downsampling module are input into the first feature fusion submodule as high-level features; the output features of the first feature fusion submodule are input into the second feature fusion submodule as low-level features, and the attention features output by the attention modules corresponding to the first downsampling module are input into the second feature fusion submodule as high-level features.

[0095] The various functional modules in the embodiment of the present application are further described in detail below. Figure 3 -Appendix Figure 6 The various functional modules in the embodiment of the present application are further described in detail below.

[0096] Further, the attention module in the embodiment of the present application is a local contrast attention module, and the network architecture of the local contrast attention module is as shown in Figure 3 、 Figure 4 It can be known that the multiscale local contrast attention network realizes target segmentation through a U-shaped encoder-decoder structure, which comprises five parts, namely, a stem block, a backbone network, a feature enhancement module, a feature fusion module and a prediction module.

[0097] Specifically, the local contrast attention module is used to complete the following operations.

[0098] S11, the direction gradients of the first input features input into the local contrast attention module in eight different directions are calculated through a first convolution kernel at a given scale, and the direction gradients located in opposite directions are multiplied to obtain a plurality of difference measures at the given scale:

[0099]

[0100] wherein d n,Nl denotes the difference measure under a given scale, l denotes different scales, l = 1, 2, …, k, F in1 denotes the first input feature, conv i denotes the first convolution kernel, i = 1, 2, …, 8.

[0101] wherein it is to be noted that, in the embodiment of the present application, the first input feature of the first attention module is the initial infrared image input feature, the first input feature of the second attention module is the down-sampled infrared image feature, and the first input feature of the third attention module is the integrated infrared image feature.

[0102] Further, the present application determines the scale of convolution according to the number of rows and columns of the first convolution kernel. Specifically, the first convolution kernel is represented as:

[0103]

[0104] As can be seen from the formula of the first convolution kernel, the first convolution kernel is divided into 9 sub-blocks with the same number of rows and columns, which has eight fixed parameter convolution kernels, and the convolution kernel is equally divided into nine sub-blocks, sorted in clockwise order, and the number of rows and columns in the first convolution kernel under different scales is different. The value in the middle of each convolution kernel is 1. The present application adopts the scale of convolution operation with the first convolution kernel with different number of rows and columns, and calculates the difference measure under a given scale using the convolution kernel under multiple different scales.

[0105] S12, calculating the contrast saliency map CSM under a given scale from each difference measure under the given scale Nl

[0106]

[0107] Residually connecting the contrast saliency map under different scales with the first input feature to obtain the contrast saliency feature F CSM

[0108]

[0109] S13, respectively calculating the channel attention map of the contrast saliency feature and the spatial attention map of the contrast saliency feature, and calculating the local contrast attention F LCAM

[0110]

[0111] wherein M c denotes the channel attention map of the contrast saliency feature, M​​​s a spatial attention map representing a contrast salient feature, represents an element multiplication operation, represents an element addition operation, local contrast attention F LCAM is the output feature of the local contrast attention module.

[0112] The contrast salient feature contains specific contrast information in different channels and positions. In order to pay attention to and mine these channel and spatial information, and improve the network's ability to extract key features, the local contrast attention module is designed. As shown in Figure 4 , it is composed of a contrast channel attention module and a contrast spatial attention module. The contrast channel attention module and the contrast spatial attention module both calculate the contrast salient feature FCSM through the contrast salient module CSMblock shown in Figure 3

[0113] Specifically, the first input feature is:

[0114] F in1 ∈R C×H×W (6)

[0115] Wherein, C, H and W represent channel, height and width respectively. The contrast channel attention module provides a one-dimensional channel attention map M c ∈R C×1×1 The contrast spatial attention module provides a two-dimensional spatial attention map M s ∈R 1×H×W , and the two attention weights are multiplied by the original image and then added to output the total attention. The specific steps are as follows:

[0116]

[0117] Therefore, the local contrast attention F LCAM can also be represented as:

[0118]

[0119] In the embodiment of the application, for the input feature map, the local contrast attention module (LCAM) dynamically learns the attention weights of the channel dimension and the spatial dimension of the input feature map at the same time by explicitly designing parallel attention, so as to obtain an enhanced feature representation.

[0120] ​Specifically, each channel of the multi-channel feature map is a detector of different scale contrast features, and the attention mechanism designed by the application explores the relationship between channels and identifies the most influential feature map. Specifically, for the first input feature, the channel number of the concatenated graph obtained by the contrast saliency module CSMblock is compressed to C using 1*1 convolution, and then global average pooling (GAP) is used to obtain channel-level attention. After obtaining the channel description, the channel description is put into a fully connected network (FC) to obtain the channel attention map M of the contrast salient feature c . In order to reduce the parameters in the fully connected network (FC), the hidden layer activation size is set to R C / r*1*1 , where r is the reduction rate.

[0121] Therefore, the calculation of the channel attention map of the contrast salient feature is specifically as follows:

[0122] M c =σ(FC(GAP(con 1×1 (F CSM (F in1 ))))) (10)

[0123] Where, σ represents an activation function, FC is a fully connected operation, GAP represents an average pooling operation, con 1×1 is a convolution operation with a kernel size of 1*1.

[0124] Further, in infrared small target detection, the detailed position of the small target plays a crucial role. Compared with channel attention, spatial attention focuses on spatial information, which is also important as another branch. The application further designs a spatial attention module to refine the feature map at the pixel level to improve the feature representation of the small target contrast in the feature map. Like channel attention, for the first input feature, 1*1 convolution is used on the concatenated graph obtained by the contrast saliency module CSMblock to compress the channel number to one dimension to obtain a one-dimensional feature map, and a sigmoid function is used to scale it to obtain a spatial attention map M of the contrast salient feature s . The calculation process can be summarized as follows:

[0125] Therefore, the calculation of the spatial attention map of the contrast salient feature is specifically as follows:

[0126] M s =σ(con 1×1 (F CSM (F in1 ))) (11)

[0127] Further, with reference to Figure 5The multi-scale hierarchical feature integration module (MHFI) of the embodiment of the present application is used to complete the following operations:

[0128] S21, performing convolution operations of different scales on the second input feature input into the multi-scale hierarchical feature integration module to obtain local features of different scales:

[0129] F p =con 1×1 (F in2 ) (12)

[0130] F S =con 1×1 (con 3×3 (F in2 )) (13)

[0131] Wherein, F p represents the first convolutional sub-feature, F s represents the second convolutional sub-feature, F p represents the first convolutional sub-feature and F s represents the second convolutional sub-feature, i.e. local features with different scales, con 1×1 is a convolution operation with a kernel size of 1x1, and con 3×3 is a convolution operation with a kernel size of 3x3; in addition, in the specific embodiment of the present application, the second input feature is an infrared image feature input into the multi-scale hierarchical feature integration module by the last down-sampling module,

[0132] S22, dividing the second input feature into N feature map subsets along the channel dimension:

[0133] F z ∈R C / N×ZH×W (where z∈{1,2,...N}) (14)

[0134] Specifically, the present application divides F in2 ∈R C×W×H into N feature subsets along the channel dimension by a Split operator, and N is 4 in the specific embodiment of the present application.

[0135] S23, respectively performing dilated convolution with different dilation rates on each feature map subset, and hierarchically connecting different groups of dilated convolution to obtain a plurality of convolution feature map subsets:

[0136]

[0137] Wherein, concat represents a connection operation; in the specific embodiment of the present application, each group of feature map subsets can be denoted as F z ∈R C / N×H×W , and each group Fz Corresponding to one dilated convolution, the dilation rate is set to 2, 4, 8 and 16, and the receptive field is gradually enlarged without increasing the number of parameters. Different groups of dilated convolutions are hierarchically connected in a residual-like manner. Structurally, the first group of dilated convolutions extracts features from F1, obtaining Next, the second group of feature map subsets F2 is concatenated with the output of the previous group , and then sent to the second group of dilated convolutions. This process is repeated several times until all input features are processed, as shown in formula (15) and the accompanying Figure 5 .

[0138] S24, splicing each convolutional feature map subset along the channel dimension, and performing the following operation to obtain the total feature F of the dilated convolution layer with different scales D :

[0139]

[0140] where ReLU represents a rectified linear unit, B represents a batch normalization operation, represents the feature obtained by splicing each convolutional feature map subset along the channel dimension;

[0141] S25, performing the following operation on the first convolutional sub-feature, the second convolutional sub-feature and the total feature of the dilated convolution layer to obtain the first output feature F of the multi-scale hierarchical feature integration module out1 :

[0142] F out1 = ReLU (B (Con 1×1 (concat (F p ,F S ,F D ))) (17)

[0143] Further, in infrared small target detection, high-level feature maps carry rich semantic information, while low-level feature maps contain key details and edge information. Due to multiple downsampling processes, high-level features may lose details sensitive to small targets, while low-level features may lack sufficient global context. To facilitate effective interaction between high-level and low-level features, the present application proposes a dynamic feature cross perception module (DFCS). The feature fusion sub-module is a dynamic feature cross perception module (DFCS), which adaptively fuses spatial details in low-level features and semantic information in high-level features using global information. Referring to Figure 6 , the dynamic feature cross perception module (DFCS) of the embodiment of the present application is used to perform the following operations:

[0144] S31, input the high-level features F into the dynamic feature cross perception module H The following operations are performed:

[0145]

[0146] Wherein, represents the modified high-level features, TPConv 3×3,2,C / 2 represents a transpose convolution operation to double the spatial dimension and halve the channel dimension; in the embodiment of the application, in order to match the low-level features, the high-level features are first passed through a 3x3 transpose convolution layer to double the spatial dimension and halve the channel dimension.

[0147] S32, input the low-level features F into the dynamic feature cross perception module L and the modified high-level features are spliced in the channel dimension to obtain global fusion features F Cat ;

[0148]

[0149] S33, the global fusion features are processed by the bottleneck structure as follows to obtain the fusion features F endowed with global information G :

[0150]

[0151] Wherein, PBR represents a continuous point-wise convolution layer, a batch normalization layer and a ReLU activation function, GAP is a global average pooling, and σ represents an activation function, represents an element multiplication operation; in the embodiment of the application, the global fusion features F Cat are input into a bottleneck structure composed of two point-wise convolution layers to reduce the number of parameters and the amount of calculation while maintaining the efficient nonlinear feature extraction capability of the fusion features. The fusion features are calibrated by global channel information to strengthen the feature representation. Subsequently, by point-wise convolution, the features are selected according to the importance of the feature maps to obtain F G , guiding the convolution layer to retain important features while removing the part with less information.

[0152] S34, the low-level features are further extracted by the bottleneck structure to extract spatial details and edge information, which are injected into the deep features to obtain the high-level features F' injected with low-level information H :

[0153]

[0154] In the embodiment of the application, the low-level features are further extracted by the bottleneck structure to extract spatial details and edge information, which are injected into the deep features to obtain the high-level features F' injected with low-level informationH .

[0155] S35, performing element addition operation on the fusion feature of the global information and the high-level feature of the injected low-level information, to obtain an output feature of the dynamic feature cross perception module:

[0156]

[0157] wherein F out2 represents the output feature of the dynamic feature cross perception module.

[0158] The embodiment of the application proposes an MLCMANet to realize SIRST detection. In order to further overcome the challenges brought by infrared small targets, three key modules are designed and utilized in the proposed MLCMANet: a local-contrast attention module (LCAM), a multi-scale hierarchical feature integration module (MHFI), and a dynamic feature cross perception module (DFCS). The LCAM aims to enhance the small target region and weaken the uninteresting region. The MHFI fuses the context information from multiple scales to achieve better feature representation. The DFCS fuses low-level and deep-level semantics to retain more information about small targets. Comparison with state-of-the-art methods and ablation studies verify the superiority of the constructed model and the effectiveness of each component.

[0159] Further, the embodiment of the application also experimentally verifies the above-mentioned multi-scale local-contrast attention infrared small target detection method. Specifically, it is implemented using PyTorch. Initially, the size of the image and label is adjusted to 512x512, and then a 480x480 crop is randomly cropped as the network input. In addition, all networks are trained using a Soft-IoU loss function, and Adam is selected as the optimizer. The initial learning rate is set to 0.001, and the weight decay coefficient is set to 1x10-4. The batch size is set to 8, and the number of iterations is 300 epochs. In terms of hardware, all methods are implemented on a Windows 10 computer equipped with an Intel i7-12700 CPU and a GeForce RTX 4090D. For the method embodiments, they are all described as a series of action combinations for simplicity, but those skilled in the art should know that the application is not limited by the described action sequence, because according to the application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the application.

[0160] Further, the input image is first input through a stem, which expands the image channel to 16 to increase the representation ability of the network for input features. The stem block is two 3x3 convolutions, the first of which has an input channel of 1 and an output channel of 16, and an output size of 480x480; the second has an input channel of 16, an output channel of 16, and an output size of 480x480.

[0161] The backbone network is divided into three down-sampling stages and a multi-scale hierarchical feature integration module, which is connected after the last stage. Each down-sampling stage contains four residual blocks, and each residual block contains two 3x3 convolutions. The first stage has an input channel of 16, an output channel of 16, and an output size of 480x480; the second stage has an input channel of 16, an output channel of 32, and an output size of 240x240; the third stage has an input channel of 32, an output channel of 64, and an output size of 120x120.

[0162] The feature enhancement module contains three local contrast attention modules, which are connected after the first stage, the second stage, and the hierarchical feature integration module of the backbone network, respectively. It is used to guide the extraction of target features in the deep layers of the network.

[0163] The feature fusion module contains two dynamic feature cross perception modules. The first dynamic feature cross perception module receives feature maps from the multi-scale hierarchical feature integration module and the local contrast attention module after the second stage; the second dynamic feature cross perception module receives feature maps from the previous dynamic feature cross perception module and the local contrast attention module after the first stage.

[0164] Finally, the prediction module uses a 1x1 convolution for prediction, and the fine segmentation result image of the infrared small target is obtained through the convolution segmentation head. The output image has a channel number of 1 and a size of the original image size.

[0165] This method achieves relatively accurate segmentation because the local contrast attention module (LCAM) is specifically designed to enhance the representation ability of contrast features for small and dim infrared targets. It focuses on target contrast features in the network and adaptively extracts spatial and channel contrast information features.

[0166] The method provided by the embodiment of the application is a multi-scale local contrast attention infrared small target detection method, which integrates the advantages of convolutional neural network (CNN) end-to-end and local contrast measurement and can effectively extract global semantic information and features of small targets. Specifically, a local contrast attention module (LCAM) is designed to fully utilize the contrast information of space and channels of an image, enhance the contrast information of small targets, and suppress regions of no interest. Secondly, a multi-scale hierarchical feature integration module (MHFI) is designed to fully utilize the global context information of small targets, which helps to estimate the correlation between pixels in a large range and effectively process infrared small targets of various sizes and shapes. Finally, a dynamic feature cross-sensing module (DFCS) is designed to guide deep feature expression through shallow feature, which helps to promote the bottom-up interaction and fusion of low-level detailed information and high-level semantic information. The method has high generalization and robustness and effectively segments infrared small targets.

[0167] Furthermore, those skilled in the art will appreciate that the features of the various embodiments are combinable in different ways, and that the combinations of features from different embodiments are within the scope of the application and form different embodiments. For example, any of the claimed embodiments can be used in any combination.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-scale local contrast attention infrared small target detection method, characterized by: The method comprises: Inputting the initial infrared image input features into a preset backbone network for feature encoding; wherein the backbone network includes a plurality of cascaded downsampling modules and a multi-scale hierarchical feature integration module connected to the last downsampling module, so as to obtain at least one downsampled infrared image feature through at least one downsampling module in the cascaded plurality of downsampling modules, and performing local feature extraction and global feature extraction of different scales on the sampling results of the last downsampling module through the multi-scale hierarchical feature integration module, and performing feature fusion on the obtained local features and global features of multiple scales to obtain an integrated infrared image feature; Inputting at least one downsampled infrared image feature and the integrated infrared image feature into a feature enhancement module respectively; wherein the feature enhancement module includes a plurality of attention modules for processing each downsampled infrared image feature and for processing the integrated infrared image feature respectively, and the attention modules obtain attention features by fusing multi-scale channel attention features and multi-scale spatial attention features on the input features; The attention features of different scales output by each attention module are input into the feature fusion module respectively. The feature fusion module fuses the input attention features of different scales to enhance the feature representation of the target feature and obtain the fused infrared image feature. The fused infrared image features are input into the prediction module to obtain the result image of infrared small target segmentation in the infrared image; wherein, The attention module is a local contrast attention module, which is used to complete the following operations: The directional gradients of the first input feature of the local contrast attention module in eight different directions are calculated by the first convolution kernel at a given scale, and the directional gradients in the opposite directions are multiplied to obtain multiple difference metrics at a given scale: n =1,2,3,4 l =1,2,3,...,k in, represents the difference measure at a given scale, l Represents different scales, F in1 represents the first input feature, Conv i represents the first convolution kernel, i =1,2,...,8; Calculate the contrast saliency map at a given scale from each difference measure at a given scale : Perform residual connection between the contrast saliency map at different scales and the first input feature to obtain the contrast saliency feature F CSM : in, concat Indicates a connection operation; Calculate the channel attention map of the contrast salient feature and the spatial attention map of the contrast salient feature respectively, and calculate the local contrast attention map based on the channel attention map and the spatial attention map F LCAM : in, M c Channel attention maps representing contrast-significant features, M s The spatial attention map representing contrast salient features, represents element-wise multiplication operation, Represents element-wise addition: Wherein, the first convolution kernel is expressed as: The first convolution kernel is divided into 9 sub-blocks with the same number of rows and columns, and the number of rows and columns in the first convolution kernel at different scales is different.

2. The method according to claim 1, characterized in that The feature fusion module includes at least one cascaded feature fusion submodule; The attention features of different scales output by each attention module are input into the feature fusion module, including: Input the attention features output by the attention module corresponding to the multi-scale hierarchical feature integration module as low-level features into the feature fusion submodule at the bottom layer of the feature fusion module, and input the output features of the feature fusion submodule at the previous upsampling layer as low-level features into the feature fusion submodule at the next upsampling layer; The attention features output by the attention modules corresponding to each downsampling module are input as high-level feature skip layers into the feature fusion submodule corresponding to the scale.

3. The method according to claim 1, characterized in that The separately calculating the channel attention map of the contrast salient feature and the spatial attention map of the contrast salient feature comprises: in, M c Channel attention maps representing contrast-significant features, M s Represents the spatial attention map of the contrast salient feature, σ represents the activation function, FC is the fully connected operation, GAP represents the average pooling operation, con 1×1 is a convolution operation with a kernel size of 1×1.

4. The method according to claim 1 or 2, characterized in that The multi-scale hierarchical feature integration module is used to complete the following operations: Perform convolution operations of different scales on the second input feature of the multi-scale hierarchical feature integration module to obtain local features of different scales: in, F p represents the first convolution sub-feature, F in2 represents the second input feature, F s represents the second convolution sub-feature, con 1×1 is a convolution operation with a kernel size of 1×1, con 3×3 is a convolution operation with a kernel size of 3×3; The second input feature is divided equally along the channel dimension into N Feature map subsets: in, F Z represents a subset of feature maps, C, H and W represent channel, height and width respectively; Perform dilated convolutions with different dilation rates on each feature map subset, and connect the dilated convolutions of different groups in layers to obtain multiple convolution feature map subsets: in, represents the subset of convolutional feature maps, d Z The convolution kernel is 3×3 and the expansion rate is 2 Z , concat represents the connection operation; Each convolution feature map subset is spliced ​​along the channel dimension, and the following operations are performed to obtain the total features of the dilated convolution layer with global features of different scales: F D : in, represents the calibrated linear unit, represents the batch normalization operation, Represents the features obtained by concatenating each convolutional feature map subset along the channel dimension; Perform the following operations on the first convolution sub-feature, the second convolution sub-feature and the total feature of the dilated convolution layer to obtain the first output feature of the multi-scale hierarchical feature integration module: F out1 : 。 5. The method according to claim 2, characterized in that The feature fusion submodule is a dynamic feature cross perception module, which is used to complete the following operations: High-level features of the dynamic feature cross-perception module are input F H Perform the following operations: ， in, Represents the high-level features after transformation, TPConv 3×3,2,C / 2 represents a transposed convolution operation to double the high-level feature space size and halve the channel size; The low-level features that are input into the dynamic feature cross-perception module F L The global fusion feature is obtained by splicing the transformed high-level features in the channel dimension F Cat ; The bottleneck structure is used to perform the following operations on the global fusion feature to obtain the fusion feature with global information: F G : ， Among them, PBR represents the continuous point-by-point convolution layer, batch normalization layer and ReLU activation function, GAP is the global average pooling, σ represents the activation function, Represents element-wise multiplication operation; The bottleneck structure is used to further extract spatial details and edge information from low-level features, and then inject them into deep features to obtain high-level features that have been injected with low-level information. : ， An element-wise addition operation is performed on the fusion features imparted with global information and the high-level features injected with low-level information to obtain output features of the dynamic feature cross-perception module.

6. The method according to claim 1, characterized in that The backbone network includes a cascaded first downsampling module, a second downsampling module and a third downsampling module. The features output by the first downsampling module are first downsampling infrared image features, and the features output by the second downsampling module are second downsampling infrared image features.

7. The method according to claim 6, characterized in that The feature fusion module includes a cascaded first feature fusion submodule and a second feature fusion submodule. The attention features of different scales output by each attention module are input into the feature fusion module, including: The attention features output by the attention module corresponding to the multi-scale hierarchical feature integration module are used as low-level features, and the attention features output by the attention module corresponding to the second downsampling module are used as high-level features, and are input into the first feature fusion submodule; The output features of the first feature fusion submodule are used as low-level features, and the attention features output by the attention module corresponding to the first downsampling module are used as high-level features and input into the second feature fusion submodule.

8. The method according to claim 1, characterized in that Before inputting the initial infrared image input features into a preset backbone network for feature encoding, the method includes: Acquire infrared images containing small targets; The infrared image is input into a stem module, so as to perform channel expansion on the infrared image through the stem module and increase the representation capability of the input features, thereby obtaining initial infrared image input features.

Images