Infrared image small target segmentation method based on space channel attention network

Abstract

The invention provides an infrared image small target segmentation method based on a space channel attention network, and the method comprises the steps: inputting a to-be-processed infrared image into a preset encoder, carrying out the step-by-step iterative feature extraction and down-sampling, and extracting different-scale low-level semantic features of a target region at different feature processing levels, the encoder comprises a plurality of cascaded sub-encoders, each sub-encoder is composed of a preset MCFA module and a down-sampling module, and the output feature of the MCFA module of each sub-encoder is the low-level semantic feature of the current feature processing level; and respectively inputting the low-level semantic features output by each feature processing level in the encoder into sub-decoders of corresponding feature reconstruction levels in a preset decoder, and executing step-by-step iterative up-sampling, feature fusion and feature reconstruction so as to finally obtain features of a target area segmented from the infrared image. According to the invention, the target area in the infrared image can be accurately identified.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a small target segmentation method for infrared images based on a spatial channel attention network. Background Art

[0002] Infrared small target segmentation has a wide range of applications in maritime rescue, maritime surveillance, infrared guidance, forest fire prevention and other fields. However, the infrared small target segmentation task has various challenges. First, due to the long imaging distance, the target size of infrared small targets is usually less than 30×30 pixels. Second, infrared small targets are often submerged in complex backgrounds, resulting in low contrast of small targets. Finally, infrared small targets lack shape, texture and color information.

[0003] In response to the above challenges, a large number of model-driven methods have been proposed, including local contrast-based methods, morphology-based methods, low-rank sparse matrix-based methods, and filter-based methods. The local contrast-based method uses local differences to suppress the background to enhance the target. The morphology-based method extracts and adjusts the morphological features of the target and background areas to achieve the segmentation and extraction of small infrared targets. The low-rank sparse matrix-based method uses the sparse properties of small infrared targets and the low-rank properties of the background to convert the segmentation task into a classification task. The filter-based method uses the designed filter to estimate the background of the infrared image to achieve the effect of suppressing the background. However, the model-driven method relies on parameter selection and manual debugging, and the segmentation performance is poor in scenes with complex backgrounds and low signal-to-noise ratios.

[0004] Unlike model-driven methods, data-driven methods based on deep learning can automatically learn features from a large number of data sets in complex scenes, making the network model have certain adaptability and generalization ability when dealing with small infrared targets in complex scenes. Therefore, various networks specifically used for infrared small target detection have been proposed. These box-level methods use bounding boxes to a certain extent to solve the problem of poor segmentation performance in scenes with complex backgrounds and low signal-to-noise ratios, and are more resistant to interference from complex backgrounds than traditional methods. However, target detection methods usually cannot describe the shape and size of the target well, and cannot accurately detect the target when the target size is small.

[0005] In fact, semantic segmentation can provide pixel-level classification results and better handle small-sized targets in complex backgrounds. If infrared small target detection is modeled as a semantic segmentation problem, the model detection performance loss caused by the small size of the target can be better solved. However, the convolution operation of the semantic segmentation method can only capture local features and lacks the ability to perceive global information. It is easy to segment the noise in the background as a target. In addition, due to the small size of infrared small targets, it is easy to cause the loss of small targets after multiple downsampling, and this process is irreversible.

[0006] In addition, Transformer can capture long-range dependencies in images by introducing a self-attention mechanism to promote global learning of correlations between different regions, which can better represent high-level semantic information in images. However, since small infrared targets have different shapes and sizes, multi-scale feature extraction is crucial. However, the Transformer designed by these methods only has single-scale feature extraction capabilities, and has poor generalization capabilities for segmenting small targets of different shapes and sizes. Summary of the invention

[0007] The present invention proposes an infrared image small target segmentation method based on spatial channel attention network to overcome or partially overcome the above problems.

[0008] The present invention provides a method for segmenting small targets in infrared images based on a spatial channel attention network, the method comprising:

[0009] The infrared image to be processed is input into a preset encoder to perform iterative feature extraction and downsampling step by step, and low-level semantic features of different scales of the infrared image are extracted at different feature processing levels, wherein the encoder includes a plurality of cascaded sub-encoders, each sub-encoder corresponds to a feature processing level, each sub-encoder is composed of a preset MCFA module and a downsampling module, the output feature of the MCFA module of each sub-encoder is the low-level semantic feature of the current feature processing level, and the MCFA module represents a multi-scale convolution fusion attention module, which is used to extract low-level semantic features of the target area by combining the global information and local information of the infrared image;

[0010] The low-level semantic features output by each feature processing level in the encoder are respectively input into the sub-decoder of the corresponding feature reconstruction level in the preset decoder, and upsampling, feature fusion and feature reconstruction are performed step by step to fuse the low-level semantic features of the infrared image at different scales into the high-level semantic features of the infrared image at the corresponding scale obtained in the upsampling process of the sub-decoder, so as to obtain the features of the target area in the infrared image, wherein the decoder includes multiple cascaded sub-decoders, each of which corresponds to a feature reconstruction level.

[0011] Furthermore, the step of inputting the low-level semantic features outputted by each feature processing level in the encoder into the sub-decoders of the corresponding feature reconstruction levels in the preset decoder respectively includes:

[0012] The low-level semantic features output from the second feature processing level to the last feature processing level are respectively input into the HGCA module of the sub-decoder in the corresponding feature reconstruction level, wherein the sub-decoders from the second level to the last level are respectively composed of an upsampling module, an HGCA module and an ARU module, the HGCA module represents a hierarchical guided channel attention module, and the ARU module represents a hole residual module;

[0013] The low-level semantic features of the first-level feature processing layer are input into the feature splicing module in the sub-decoder of the first-level feature reconstruction layer, wherein the first-level sub-decoder includes an upsampling module, a feature splicing module and an ARU module, and the ARU module of the first-level sub-decoder outputs the features of the target area.

[0014] Furthermore, in the process of respectively inputting the low-level semantic features output by each feature processing level in the encoder into the sub-decoder of the corresponding feature reconstruction level in the preset decoder, the method further includes:

[0015] The image features output by the downsampling module of the last-level sub-encoder in the encoder are input into the transition ARU module between the encoder and the decoder, and the image features output by the transition ARU module are input into the decoder to perform step-by-step iterative upsampling, wherein the transition ARU module and the ARU module in the sub-decoder have the same structure.

[0016] Furthermore, the HGCA module is used to complete the following operations:

[0017] Obtaining high-level semantic features of the input HGCA module, wherein the high-level semantic features are semantic features output by the upsampling module in the sub-encoder to which the HGCA module belongs,

[0018] After performing an average pooling operation and a maximum pooling operation on the high-level semantic features respectively, performing an addition operation on the features obtained by performing the average pooling operation and the features obtained by performing the maximum pooling operation to obtain a first high-level image feature;

[0019] A convolution operation, a batch normalization operation, and a ReLu operation are sequentially performed on the first high-level image feature to obtain a second high-level image feature;

[0020] A convolution operation, a batch normalization operation, and a Sigmoid operation are sequentially performed on the second high-level image feature to obtain a third high-level image feature;

[0021] Obtaining low-level semantic features input into the HGCA module, performing a matrix multiplication operation on the low-level semantic features and the third high-level image features to obtain fused image features;

[0022] The fused image features and the high-level semantic features are concatenated to obtain output features of the HGCA module.

[0023] Furthermore, the ARU module is used to complete the following operations:

[0024] Obtain image features input to the ARU module;

[0025] Performing step-by-step iterative residual downsampling on the image features of the input ARU module to obtain residual features of different iteration levels, wherein in each iteration level of residual downsampling, a hole residual block is used to obtain the residual features of the current iteration level, and the residual features of the current iteration level are downsampled and input into the next iteration level;

[0026] After downsampling the residual features of the last iteration level, the upsampled image features are obtained;

[0027] Performing iterative residual upsampling on the upsampled image features step by step to obtain residual upsampling features of different iteration levels, wherein residual upsampling is performed in each iteration level of residual upsampling using a cascaded hole residual block and an upsampling module, the upsampling module outputs the residual upsampling features of the current iteration level, and the residual features of the corresponding iteration levels from the second level to the last level are respectively spliced ​​with the residual upsampling features and input into the previous upsampling iteration level;

[0028] The residual features of the first level and the residual up-sampled features of the first level are added to obtain the image features output by the ARU module.

[0029] Furthermore, the structures of each hole residual block in the step-by-step iterative residual downsampling process and each hole residual block in the step-by-step iterative residual upsampling process are the same, and different features of the target area are captured by setting different hole rates for the hole residual blocks in different iteration levels.

[0030] Furthermore, the hole residual block is used to perform the following operations:

[0031] Get the features of the input hole residual block;

[0032] Performing hole convolution, batch normalization and ReLu operations on the features of the input hole residual block in sequence to obtain a first hole feature;

[0033] Performing convolution operation and batch normalization operation on the first hole feature in sequence to obtain a second hole image feature;

[0034] Performing an addition budget on the feature of the input hole residual block and the feature of the second hole image to obtain a third hole feature;

[0035] After performing the ReLu operation on the third hole feature, the feature of the output hole residual block is obtained.

[0036] Furthermore, the MCFA module is used to complete the following operations:

[0037] Obtain image features for input into the MCFA module;

[0038] Extract the feature matrix with local information of the image features input to the MCFA module, and extract the attention matrix of the image features input to the MCFA module,

[0039] Performing a multiplication operation on the feature matrix with local information and the attention matrix to obtain a feature matrix with global information;

[0040] The feature matrix with global information, the feature matrix with local information and the image features input to the MCFA module are fused to obtain the low-level semantic features output by the MCFA module:

[0041] F O =Conv 1×1 (αF av +F v +F)

[0042] Among them, F O is the low-level semantic feature output by the MCFA module, F is the image feature input to the MCFA module, and F v is the feature matrix with local information, F av is the feature matrix with global information, F a is the attention matrix, and α represents the learnable weight.

[0043] Furthermore, the feature matrix with local information for extracting image features input to the MCFA module includes:

[0044] The following operations are performed on the image features input to the MCFA module to map the image features input to the MCFA module to a Query vector:

[0045] F q1 = reshape(FC(reshape(Conv 1×1 (F))))

[0046] The following operations are performed on the image features input to the MCFA module to map the image features input to the MCFA module into a Key vector:

[0047] Fk1 = reshape(FC(reshape(Conv 1×1 (F))))

[0048] The following operations are performed on the Query vector and the Key vector to obtain the attention matrix:

[0049] F a =Softmax(reshape(F q1 ×F k1 ))

[0050] Among them, Conv 1×1 Represents a 1×1 convolution operation, reshape represents a tensor reshaping operation, FC represents a fully connected operation, and Softmax represents a normalization operation.

[0051] Furthermore, the feature matrix with local information for extracting image features input to the MCFA module includes:

[0052] Perform two convolution operations on the image features input to the MCFA module to obtain the first image features;

[0053] Performing two dilated convolution operations with a first dilated rate on the image features input to the MCFA module to obtain a second image feature;

[0054] Performing two dilated convolution operations with a second dilated rate on the image features input to the MCFA module to obtain a third image feature;

[0055] A splicing operation is performed on the first image feature, the second image feature and the third image feature, and a convolution operation is performed on the obtained spliced ​​image feature to obtain a feature matrix with local information.

[0056] The embodiment of the present application provides a method for small target segmentation in infrared images based on a spatial channel attention network. The present invention simultaneously obtains global information and local information of the target area in the infrared image based on a multi-scale convolutional fusion attention module, better understands the overall morphology and background of the target area in the infrared image, and uses a skip-layer link method to hierarchically integrate the low-level semantic features of the infrared image in the downsampling process into the high-level semantic features of the infrared image in the upsampling process, which further helps to select the features of the target area and retain the key features of the target area. At the same time, through step-by-step upsampling, the feature reconstruction of the target area is achieved, and finally the pixel-level semantic segmentation result of the infrared image is generated, and the target area in the infrared image is accurately identified.

[0057] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Various other advantages and benefits will become apparent to those skilled in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only used for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. In the accompanying drawings:

[0059] Figure 1 It is a flow chart of a method for small target segmentation in infrared images based on a spatial channel attention network according to an embodiment of the present invention;

[0060] Figure 2 A schematic diagram of the network architecture of a method for segmenting small targets in infrared images based on a spatial channel attention network according to an embodiment of the present invention;

[0061] Figure 3 Schematic diagram of the architecture of the MCFA module according to an embodiment of the present invention;

[0062] Figure 4 Schematic diagram of the HGCA module of an embodiment of the present invention;

[0063] Figure 5 Schematic diagram of the architecture of the ARU module according to an embodiment of the present invention;

[0064] Figure 6 Schematic diagram of segmentation results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0066] Before introducing the infrared image small target segmentation method based on the spatial channel attention network according to the embodiment of the present invention, the principles to be used in the present invention are briefly introduced:

[0067] (1)CNN

[0068] The network structure of CNN is similar to that of artificial neural networks, and is currently widely used in the fields of image segmentation, face recognition, and target detection. CNN uses a weight sharing method, which can greatly reduce the parameters of neurons, thereby simplifying the model. For multi-dimensional images, CNN can identify spatial features and relationships in multi-dimensional directions, a property that other neural networks do not have. In an image, CNN can extract underlying image features such as the edges or angles of the image, and then use convolutional layers to repeatedly extract higher-level image features, making CNN's feature extraction ability better than manually operated features. CNN mainly includes the following six network layers:

[0069] 1) Input layer: The first layer of CNN, used to input image data.

[0070] 2) Convolution layer: As the most important layer of CNN, its main function is to learn and extract the features of the input image. Convolution kernels of different sizes extract different image features. Usually, the convolution of the previous layer is used to extract the underlying features. After obtaining the underlying features of the image, the subsequent convolution repeatedly extracts high-level features. These high-level features can realize the segmentation and classification of the image.

[0071] 3) Activation function layer: The activation function layer is used in CNN to improve the nonlinearity of the network to improve the data fitting performance of the network. In addition, it can also prevent the network gradient from disappearing during the process of extracting image features. The activation functions commonly used in CNN are:

[0072] Sigmoid function: Mainly used for binary classification problems. The Sigmoid function maps the input to (0,1), and the expression is as follows:

[0073]

[0074] The Sigmoid activation function will make the network gradient disappear. As can be seen from the formula, when the input is infinite or infinitely small, its derivative is infinitely close to 0, which makes the network gradient update slow or disappear. In addition, its output mean is not 0, which makes the calculation of the next layer complicated, so it is usually only applied to the output layer.

[0075] Corrected Linear Unit: Converges faster than the hyperbolic tangent function. When the input value is less than 0, the input is mapped to 0. When the input value is greater than or equal to 0, the output value is equal to the input value. The function expression is as follows:

[0076]

[0077] 4) Pooling layer: The pooling layer is also an important component layer in CNN. Its main function is to realize network downsampling and upsampling. During the downsampling process, the image resolution will be reduced, which will greatly reduce the number of network parameters and avoid network overfitting. Its translation invariance and the role of increasing image resolution are of great significance for segmentation or classification. The pooling layer is an important means often used by neural networks to avoid overfitting.

[0078] 5) Fully connected layer: After the image data is automatically extracted through multiple convolutions and pooling, the features extracted at this time are usually deeper features, and then pass through the fully connected layer to finally output the classification results.

[0079] 6) Output layer: The last layer of CNN, which is used to output the classifier of the number of categories.

[0080] (2) Transformer

[0081] Unlike traditional RNN and CNN, the Transformer network has no loop and convolution structure, so it can process each element in the sequence in parallel, speeding up training and reasoning. The Transformer consists of two parts: an encoder and a decoder, where the encoder converts the input sequence into a series of high-dimensional vectors, and the decoder gradually generates the target sequence based on the encoder's output and previous predictions. Each encoder and decoder consists of multiple layers of Transformer blocks, each of which consists of two sub-layers, namely a multi-head self-attention machine and a feedforward neural network.

[0082] Position encoding: Position encoding is a very important step in Transformer, which can give each element in the input sequence a position information, so that Transformer can process the position information of the elements in the sequence.

[0083] Multi-head self-attention mechanism: The self-attention mechanism is a mechanism that can learn the relationship between different positions in the input sequence. In each Transformer block of the encoder and decoder, the multi-head self-attention mechanism maps the input sequence into a triple of a key, a query, and a value. An attention score vector is obtained by calculating the dot product of the query and the key, and then the attention score vector is multiplied by the value vector to obtain the self-attention vector. In this process, the multi-head self-attention mechanism divides the attention mechanism into multiple heads, calculates the attention score vectors separately, and concatenates multiple self-attention vectors, and then performs dimensionality reduction through the fully connected layer. The specific calculation formula is as follows:

[0084]

[0085] MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W o (5) Where Q, K, and V represent the amount of data mapped from the input sequence to Query, Key, and Value, respectively. W o Represents different weight matrices, head i Represents the score vector calculated by the i-th head attention module.

[0086] The multi-head attention mechanism can be understood as a mechanism that maps the original vector set into multiple subspaces and calculates the attention weights independently in each subspace. Specifically, the input Query, Key, and Value vectors are linearly transformed into vectors in multiple subspaces, and then the dot product attention weights are calculated in each subspace. Finally, the attention information obtained in each subspace is concatenated and linearly transformed again to obtain the final output vector.

[0087] Feedforward Neural Network: The feedforward neural network is another important component in Transformer. In each Transformer block, the feedforward neural network accepts the output from the multi-head self-attention mechanism as input and maps it into a new representation vector through two linear transformations and a nonlinear activation function. It can be expressed as:

[0088] FFN(x)=max(0,xW1+b1)W2+b2 (6)

[0089] W1 and W2 represent two different linear functions, b1 and b2 are two bias constants, and the activation function used is the ReLU function.

[0090] (3) Dilated Convolution

[0091] For images where the target area is smaller than the global image, a multi-scale feature extractor is needed to expand the receptive field. Therefore, based on Unet, the first convolution of each layer of the encoding module and the convolution of the bottom transition layer are replaced with dilated convolutions with different dilation rates to expand the receptive field and capture the feature information of the surrounding objects in the image.

[0092] Atrous convolution theory is widely used in computer vision tasks. It keeps the feature resolution unchanged by increasing the receptive field. Atrous convolution can be defined as follows:

[0093]

[0094] where f out [i] is the output feature, f in[i] is the input feature, d is the hole rate, ω[k] is the kth parameter of the filter, and k is the filter size.

[0095] Furthermore, f out [i] represents the value of the output feature at position i. It is calculated by taking the weighted sum of the input feature and the filter. in [i+d*k]: The value of the input feature at position i+d*k. Here d represents the dilation rate, which controls the position interval of sampling in the convolution operation. By increasing d, the input features can be sampled in a larger range without increasing the size of the convolution kernel. ω[k] represents the weight parameter of the convolution kernel at position k. It is used to weight the input features at the corresponding position.

[0096] Unlike traditional convolution, Atrous convolution introduces a dilation rate d, which allows the convolution kernel to be sampled at intervals of d, thereby expanding the receptive field while maintaining the resolution of the feature. Specifically: Keep the feature resolution unchanged: Atrous convolution can expand the receptive field without increasing the size of the feature map. Enhance multi-scale feature extraction: By setting different dilation rates d, Atrous convolution can extract features at different scales, so it is particularly suitable for tasks that require multi-scale information such as semantic segmentation and object detection. In this formula, d controls the sampling interval, that is, the sampling interval of the convolution kernel on the input feature. The role of the dilation rate is to determine the distance of the "jump" sampling during the convolution operation. When d=1, the formula is equivalent to a normal convolution operation, the sampling interval is 1, and the convolution kernel acts continuously on the input feature. When d>1, the convolution kernel jumps sampling on the input feature with an interval of d, which expands the receptive field of the convolution kernel.

[0097] Further, Figure 1 A method for segmenting small targets in infrared images based on a spatial channel attention network is shown in FIG. Figure 1 It can be seen that the infrared image small target segmentation method based on the spatial channel attention network of the present invention includes the following steps:

[0098] S1. Input the infrared image to be processed into a preset encoder to perform iterative feature extraction and downsampling step by step, and extract low-level semantic features of the infrared image at different scales at different feature processing levels, wherein the encoder includes a plurality of cascaded sub-encoders, each sub-encoder corresponds to a feature processing level, each sub-encoder is composed of a preset MCFA module and a downsampling module, the output feature of the MCFA module of each sub-encoder is the low-level semantic feature of the current feature processing level, and the MCFA module represents a multi-scale convolution fusion attention module, which is used to extract low-level semantic features of the target area by combining the global information and local information of the infrared image;

[0099] The present invention reconsiders the relationship between local and global information of the image, improves the Transformer architecture, and designs the MCFA module, so that the MCFA module can pay attention to the global and local information of the infrared image by calculating the attention matrix, which can effectively enhance the features of small targets and suppress noise.

[0100] S2. The low-level semantic features output by each feature processing level in the encoder are respectively input into the sub-decoders of the corresponding feature reconstruction levels in the preset decoder, and upsampling, feature fusion and feature reconstruction are performed step by step to fuse the low-level semantic features of the infrared image at different scales into the high-level semantic features of the infrared image at corresponding scales obtained in the upsampling process of the sub-decoder, so as to obtain the features of the target area in the infrared image, wherein the decoder includes a plurality of cascaded sub-decoders, and each sub-decoder corresponds to a feature reconstruction level.

[0101] In the process of iterative upsampling, the present invention uses a skip connection method to map and reconstruct the low-level semantic features extracted in the downsampling process with the high-level semantic features of the infrared image, and gradually restores the resolution of the infrared image to retain the key features of the target image, and finally accurately identifies the target area in the image. It should be noted here that the present invention regards the levels at the same feature processing scale in the decoder and the encoder as the same level.

[0102] In addition, in the embodiment of the present invention, the fusion of low-level semantic features of different scales of the infrared image into the high-level semantic features of the corresponding scale of the infrared image obtained in the upsampling process of the sub-decoder is implemented by the HGCA module. The HGCA module implements the fusion of the first semantic features and the high-level semantic features in the corresponding feature processing levels in the encoder and the decoder.

[0103] Figure 2 A network architecture diagram of a method for segmenting small targets in infrared images based on a spatial channel attention network is shown in another specific embodiment of the present invention. Figure 2 The present invention specifically shows how the encoding process and the decoding process of the infrared image are jump-connected, and the detailed implementation method of the decoding process of the infrared image. Figure 2 The specific network architecture shown adopts an encoder-decoder structure design similar to the classic UNet. This design aims to promote effective interaction between low-level and high-level feature information through a specific hierarchical structure, so that the model can better combine the details and global information in the image.

[0104] The encoder part consists of five MCFA modules, which are used to extract multi-level features of the image. In the decoder part, the model contains five ARUs, which are responsible for mapping the encoded features back to the resolution of the original image. In addition, there is an additional ARU as a transition layer at the bottom. In order to maintain the feature transfer and consistency between the encoder and decoder, jump connection layers are used. These connections are implemented through HGCA modules to ensure that features at different levels can be effectively fused. In the first stage of the model, the encoder module is responsible for extracting the key features of the input image, which includes global and local key information to ensure that the model can capture important patterns in the image. Next, these extracted features will be mapped and reconstructed in the decoder, gradually restoring the resolution of the image while retaining key features. Finally, after multi-level feature reconstruction, the model generates pixel-level semantic segmentation results for infrared images, accurately identifying the target area in the image.

[0105] Specifically, in combination with Figure 1 and attached Figure 2 It can be seen that in step S2, a specific embodiment of inputting the low-level semantic features output by each feature processing level in the encoder into the sub-decoder of the corresponding feature reconstruction level in the preset decoder includes the following steps:

[0106] S21, respectively inputting the low-level semantic features output from the second feature processing level to the last feature processing level into the HGCA module of the sub-decoder in the corresponding feature reconstruction level, wherein the sub-decoders from the second level to the last level are respectively composed of an upsampling module, an HGCA module and an ARU module, the HGCA module represents a hierarchical guided channel attention module, and the ARU module represents a hole residual module;

[0107] S22. Input the low-level semantic features of the first-level feature processing layer into the feature splicing module in the sub-decoder of the first-level feature reconstruction layer, wherein the first-level sub-decoder includes an upsampling module, a feature splicing module and an ARU module, and the ARU module of the first-level sub-decoder outputs the features of the target area.

[0108] Further, by Figure 2It can be known that in the process of inputting the low-level semantic features output by each feature processing level in the encoder into the sub-decoder of the corresponding feature reconstruction level in the preset decoder in step S2, the present invention also includes the following method: inputting the image features output by the downsampling module of the last level sub-encoder in the encoder into the transition ARU module between the encoder and the decoder, and inputting the image features output by the transition ARU module into the decoder to perform step-by-step iterative upsampling, wherein the transition ARU module and the ARU module in the sub-decoder have the same structure. Among them, it can be understood that the image features output by the transition ARU module are input into the upsampling module of the bottom feature reconstruction level in the decoder.

[0109] In another specific embodiment of the present invention, a hierarchical guided channel attention (HGCA) module and an atrous residual U-block (ARU) are also designed, wherein the HGCA module can realize high-level features guiding low-level features, weighting decisive channels, and reducing redundant information, which can reduce background noise in infrared images and make small targets stand out more clearly in the background. The ARU is located in a deeper layer of the network and extracts more refined features from the feature map, so that the model can identify the features of small infrared targets in more detail and comprehensively. The various modules of the present invention are further introduced below in conjunction with the accompanying drawings.

[0110] like Figure 3 As shown, the MCFA module of the present invention is used to complete the following operations:

[0111] S11, obtaining image features input into the MCFA module;

[0112] S12, extracting a feature matrix with local information of the image features input to the MCFA module, and extracting an attention matrix of the image features input to the MCFA module,

[0113] S13, performing a multiplication operation on the feature matrix with local information and the attention matrix to obtain a feature matrix with global information;

[0114] S14, the feature matrix with global information, the feature matrix with local information and the image features input to the MCFA module are fused to obtain the low-level semantic features output by the MCFA module:

[0115] F O =Conv 1×1 (αF av +F v +F) (8)

[0116] Among them, F O is the low-level semantic feature output by the MCFA module, F is the image feature input to the MCFA module, and F v is the feature matrix with local information, Fav is the feature matrix with global information, F a is the attention matrix, and α represents the learnable weight.

[0117] Furthermore, the feature matrix with local information for extracting image features input to the MCFA module includes:

[0118] The following operations are performed on the image features input to the MCFA module to map the image features input to the MCFA module to a Query vector:

[0119] F q1 = reshape(FC(reshape(Conv 1×1 (F)))) (9)

[0120] The following operations are performed on the image features input to the MCFA module to map the image features input to the MCFA module into a Key vector:

[0121] F k1 = reshape(FC(reshape(Conv 1×1 (F)))) (10)

[0122] The following operations are performed on the Query vector and the Key vector to obtain the attention matrix:

[0123] F a =Softmax(reshape(F q1 ×F k1 )) (11)

[0124] Among them, Conv 1×1 Represents a 1×1 convolution operation, reshape represents a tensor reshaping operation, FC represents a fully connected operation, and Softmax represents a normalization operation.

[0125] Furthermore, the feature matrix with local information for extracting image features input to the MCFA module includes:

[0126] Perform two convolution operations on the image features input to the MCFA module to obtain the first image features;

[0127] Performing two dilated convolution operations with a first dilated rate on the image features input to the MCFA module to obtain a second image feature;

[0128] Performing two dilated convolution operations with a second dilated rate on the image features input to the MCFA module to obtain a third image feature;

[0129] A splicing operation is performed on the first image feature, the second image feature and the third image feature, and a convolution operation is performed on the obtained spliced ​​image feature to obtain a feature matrix with local information.

[0130] In a specific embodiment of the present invention, the first void ratio can be set to 2, and the second void ratio can be set to 4, so that and Then, F1, F2 and F4 are concatenated and then 1×1Conv is performed to obtain F v H×W×C .

[0131] The MCFA module of the present invention redesigns the self-attention architecture and introduces convolution and dilated convolution. By calculating the attention matrix, it is integrated with the features extracted by multi-scale convolution to simultaneously obtain local and global information of the image, which can effectively enhance the features of small targets and suppress noise. The MCFA module is mainly composed of convolution, dilated convolution and fully connected layers, and is used to extract multi-level features of the image.

[0132] Furthermore, in addition to extracting the global information of small objects, channel attention plays an important role in the small object segmentation network, which can help the network better capture the target features, reduce information redundancy, and improve the adaptability and generalization ability of the network. In the HGCA-Former model, we introduce a hierarchical guided channel attention (HGCA) module between the encoder and the decoder. Its architecture is as follows: Figure 4 shown.

[0133] The decoder contains rich high-level semantic information, which helps guide the selection of low-level information. Through the HGCA module, the HGCA-Former model can adaptively weight the features of different channels and effectively utilize the information of different channels, so that the network pays more attention to the small target itself and improves the generalization ability of the model for infrared small target segmentation in different scenarios. After the image is extracted by the MCFA block, the features are input into the HGCA block. In addition, the features processed by ARU are also input into the HGCA block to guide the selection of low-level information, further better capture the target features, and reduce information redundancy.

[0134] Specifically, the HGCA module of the present invention is used to complete the following operations:

[0135] S31, obtaining high-level semantic features input to the HGCA module, wherein the high-level semantic features are semantic features output by an upsampling module in a sub-encoder to which the HGCA module belongs;

[0136] S32, after performing an average pooling operation and a maximum pooling operation on the high-level semantic features respectively, performing an addition operation on the features obtained by performing the average pooling operation and the features obtained by performing the maximum pooling operation to obtain a first high-level image feature;

[0137] S33, performing a convolution operation, a batch normalization operation, and a ReLu operation on the first high-level image feature in sequence to obtain a second high-level image feature;

[0138] S34, performing a convolution operation, a batch normalization operation, and a Sigmoid operation on the second high-level image feature in sequence to obtain a third high-level image feature;

[0139] S35, obtaining low-level semantic features input into the HGCA module, performing a matrix multiplication operation on the low-level semantic features and the third high-level image features to obtain fused image features;

[0140] S36, performing a splicing operation on the fused image features and the high-level semantic features to obtain output features of the HGCA module.

[0141] like Figure 4 As shown, the HGCA module has two inputs, namely, high-level semantic features Coarse H×W×C and low-level semantic features Fine H×W×C First, Coarse performs AvgPool and MaxPool operations respectively and then adds the two together to get Then, F h1 Convolution, batch normalization and ReLu operations are performed in sequence to obtain Finally, F h2 Convolution, batch normalization and Sigmoid operations are performed in sequence to obtain Then, the high-level feature F h3 Multiplying with the low-level feature Fine, we get Finally, F hl Concatenate with Coarse to get the final output of the HGCA block

[0142] Furthermore, after the input image passes through the MCFA module, most of the noise is filtered out, and the target features are more prominent when passed to deeper layers of the network. In order to better capture multi-scale features, we designed the atrous residual U-block (ARU), as shown in Figure 5As shown. We use ARU in the bottom transition layer and decoder part between the network encoder and decoder. ARU is an encoder-decoder structure similar to U-Net, and the backbone is a hollow residual block. ARU performs downsampling and upsampling, and can obtain multi-scale feature maps from the input feature map. This further extracts the global and local features of the image, which is conducive to separating the noise component from the background branch. We use the residual connection of the residual block to perform feature compensation to prevent the convolution operation from causing the loss of small target information. The hollow convolution of the residual block is used to improve the depth feature resolution of each layer. Different hollow rates are set for each layer to capture the multi-scale features of small targets.

[0143] Furthermore, the ARU module of the present invention is used to complete the following operations:

[0144] S41, obtaining the image features input to the ARU module,

[0145] S42, performing step-by-step iterative residual downsampling on the image features input to the ARU module to obtain residual features of different iteration levels, wherein, in each iteration level of residual downsampling, a hole residual block is used to obtain the residual features of the current iteration level, and the residual features of the current iteration level are downsampled and input into the next iteration level;

[0146] S43, downsampling the residual features of the last iteration level to obtain upsampled image features;

[0147] S44, performing iterative residual upsampling on the upsampled image features step by step to obtain residual upsampling features of different iteration levels, wherein residual upsampling is performed in each iteration level of residual upsampling using a cascaded hole residual block and an upsampling module, the upsampling module outputs the residual upsampling features of the current iteration level, and the residual features of the corresponding iteration levels from the second level to the last level are respectively spliced ​​with the residual upsampling features and input into the previous upsampling iteration level;

[0148] S45. Perform an addition operation on the residual features of the first level and the residual up-sampling features of the first level to obtain the image features output by the ARU module.

[0149] It should be noted that the structures of each hole residual block in the step-by-step iterative residual downsampling process and each hole residual block in the step-by-step iterative residual upsampling process are the same, and different features of the target area are captured by setting different hole rates for the hole residual blocks in different iteration levels.

[0150] Furthermore, the hole residual block is used to perform the following operations:

[0151] Get the features of the input hole residual block;

[0152] Performing hole convolution, batch normalization and ReLu operations on the features of the input hole residual block in sequence to obtain a first hole feature;

[0153] Performing convolution operation and batch normalization operation on the first hole feature in sequence to obtain a second hole image feature;

[0154] Performing an addition budget on the feature of the input hole residual block and the feature of the second hole image to obtain a third hole feature;

[0155] After performing the ReLu operation on the third hole feature, the feature of the output hole residual block is obtained.

[0156] The ARU module provided in the embodiment of the present invention makes full use of the characteristics of deep network information, so the residual block in the ARU adopts dilated convolution, which can help extract more refined features and help capture the details of small targets more accurately. The design of ARU can better process a wider range of pixels and effectively filter out the noise around small infrared targets. After processing by ARU, we can get a more refined small target feature map.

[0157] In a specific embodiment of the present invention, various SOTA methods are used to segment small targets on five infrared images of the SIRST public dataset. The segmentation results are as follows: Figure 5 We magnify the target area and place it at the top, and use orange, yellow, and green circles to represent accurately detected targets, undetected targets, and false alarms, respectively. Figure 5 The experimental results of the present invention on the segmentation of small targets in infrared images are shown in FIG. Figure 5 It can be seen that the method of the present invention can accurately obtain the small target segmentation result by segmenting the small target in the infrared image.

[0158] The infrared image small target segmentation method based on the spatial channel attention network provided by the embodiment of the present invention is based on the infrared small target segmentation model of the improved Transformer: HGCA-Former. HGCA-Former improves the Transformer structure, redesigns the self-attention architecture and introduces convolution and dilated convolution. By calculating the attention matrix, it is fused with the features extracted by multi-scale convolution to simultaneously obtain local and global information of the image, which can effectively enhance the features of small targets and suppress noise, thereby avoiding the loss of information of small targets during downsampling. In addition, by introducing hierarchical guided channel attention, high-level features can guide low-level features, adaptively weight different channel features, effectively utilize different channel information, and reduce redundant information. This can reduce background noise in infrared images and make small targets stand out more clearly in the background, thereby improving the generalization ability of the model for infrared small target segmentation in different scenarios. The design of MCFA and HGCA enables HGCA-Former to suppress noise, enhance small target features, and accurately segment small targets. HGCA-Former brings a new perspective to infrared small target segmentation. Its robustness and stability make it a popular choice for real-world applications and can better support real-time applications across various devices.

[0159] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined.

[0160] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0161] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, any one of the claimed embodiments may be used in any combination.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for small target segmentation in infrared images based on spatial channel attention network, characterized in that: The method comprises: The infrared image to be processed is input into a preset encoder to perform iterative feature extraction and downsampling step by step, and low-level semantic features of different scales of the infrared image are extracted at different feature processing levels, wherein the encoder includes a plurality of cascaded sub-encoders, each sub-encoder corresponds to a feature processing level, each sub-encoder is composed of a preset MCFA module and a downsampling module, the output feature of the MCFA module of each sub-encoder is the low-level semantic feature of the current feature processing level, and the MCFA module represents a multi-scale convolution fusion attention module, which is used to extract low-level semantic features of the target area by combining the global information and local information of the infrared image; The low-level semantic features output by each feature processing level in the encoder are respectively input into the sub-decoder of the corresponding feature reconstruction level in the preset decoder, and upsampling, feature fusion and feature reconstruction are performed step by step to fuse the low-level semantic features of the infrared image at different scales into the high-level semantic features of the infrared image at the corresponding scale obtained in the upsampling process of the sub-decoder, so as to obtain the features of the target area in the infrared image, wherein the decoder includes multiple cascaded sub-decoders, each of which corresponds to a feature reconstruction level.

2. The method according to claim 1, characterized in that The step of inputting the low-level semantic features outputted by each feature processing level in the encoder into the sub-decoders of the corresponding feature reconstruction levels in the preset decoder comprises: The low-level semantic features output from the second feature processing level to the last feature processing level are respectively input into the HGCA module of the sub-decoder in the corresponding feature reconstruction level, wherein the sub-decoders from the second level to the last level are respectively composed of an upsampling module, an HGCA module and an ARU module, the HGCA module represents a hierarchical guided channel attention module, and the ARU module represents a hole residual module; The low-level semantic features of the first-level feature processing layer are input into the feature splicing module in the sub-decoder of the first-level feature reconstruction layer, wherein the first-level sub-decoder includes an upsampling module, a feature splicing module and an ARU module, and the ARU module of the first-level sub-decoder outputs the features of the target area.

3. The method according to claim 2, characterized in that In the process of respectively inputting the low-level semantic features output by each feature processing level in the encoder into the sub-decoder of the corresponding feature reconstruction level in the preset decoder, the method further includes: The image features output by the downsampling module of the last-level sub-encoder in the encoder are input into the transition ARU module between the encoder and the decoder, and the image features output by the transition ARU module are input into the decoder to perform step-by-step iterative upsampling, wherein the transition ARU module and the ARU module in the sub-decoder have the same structure.

4. The method according to claim 2, characterized in that: The HGCA module is used to complete the following operations: Obtaining high-level semantic features of the input HGCA module, wherein the high-level semantic features are semantic features output by the upsampling module in the sub-encoder to which the HGCA module belongs, After performing an average pooling operation and a maximum pooling operation on the high-level semantic features respectively, performing an addition operation on the features obtained by performing the average pooling operation and the features obtained by performing the maximum pooling operation to obtain a first high-level image feature; A convolution operation, a batch normalization operation, and a ReLu operation are sequentially performed on the first high-level image feature to obtain a second high-level image feature; A convolution operation, a batch normalization operation, and a Sigmoid operation are sequentially performed on the second high-level image feature to obtain a third high-level image feature; Obtaining low-level semantic features input into the HGCA module, performing a matrix multiplication operation on the low-level semantic features and the third high-level image features to obtain fused image features; The fused image features and the high-level semantic features are concatenated to obtain output features of the HGCA module.

5. The method according to claim 2 or 3, characterized in that: The ARU module is used to complete the following operations: Obtain image features input to the ARU module; Performing step-by-step iterative residual downsampling on the image features of the input ARU module to obtain residual features of different iteration levels, wherein in each iteration level of residual downsampling, a hole residual block is used to obtain the residual features of the current iteration level, and the residual features of the current iteration level are downsampled and input into the next iteration level; After downsampling the residual features of the last iteration level, the upsampled image features are obtained; Performing iterative residual upsampling on the upsampled image features step by step to obtain residual upsampling features of different iteration levels, wherein residual upsampling is performed in each iteration level of residual upsampling using a cascaded hole residual block and an upsampling module, the upsampling module outputs the residual upsampling features of the current iteration level, and the residual features of the corresponding iteration levels from the second level to the last level are respectively spliced ​​with the residual upsampling features and input into the previous upsampling iteration level; The residual features of the first level and the residual up-sampled features of the first level are added to obtain the image features output by the ARU module.

6. The method according to claim 5, characterized in that The structures of each hole residual block in the step-by-step iterative residual downsampling process and each hole residual block in the step-by-step iterative residual upsampling process are the same. Different features of the target area are captured by setting different hole rates for the hole residual blocks in different iteration levels.

7. The method according to claim 6, characterized in that The hole residual block is used to perform the following operations: Get the features of the input hole residual block; The features of the input hole residual block are sequentially subjected to hole convolution, batch normalization and ReL. u After the operation, the first cavity feature is obtained; Performing convolution operation and batch normalization operation on the first hole feature in sequence to obtain a second hole image feature; Performing an addition budget on the feature of the input hole residual block and the feature of the second hole image to obtain a third hole feature; After performing the ReLu operation on the third hole feature, the feature of the output hole residual block is obtained.

8. The method according to claim 1, characterized in that The MCFA module is used to complete the following operations: Obtain image features for input into the MCFA module; Extract the feature matrix with local information of the image features input to the MCFA module, and extract the attention matrix of the image features input to the MCFA module, Performing a multiplication operation on the feature matrix with local information and the attention matrix to obtain a feature matrix with global information; The feature matrix with global information, the feature matrix with local information and the image features input to the MCFA module are fused to obtain the low-level semantic features output by the MCFA module: F O =Conv 1×1 (αF av +F v +F) Among them, F O is the low-level semantic feature output by the MCFA module, F is the image feature input to the MCFA module, and F v is the feature matrix with local information, F av is the feature matrix with global information, F a is the attention matrix, and α represents the learnable weight.

9. The method according to claim 8, characterized in that The feature matrix with local information for extracting image features input into the MCFA module includes: The following operations are performed on the image features input to the MCFA module to map the image features input to the MCFA module to a Query vector: F q1 =reshape(FC(reshape(Conv 1×1 (F)))) The following operations are performed on the image features input to the MCFA module to map the image features input to the MCFA module into a Key vector: F k1 =reshape(FC(reshape(Conv 1×1 (F)))) The following operations are performed on the Query vector and the Key vector to obtain the attention matrix: F a =Softmax(reshape(F q1 ×F k1 )) Among them, Conv 1×1 Represents a 1×1 convolution operation, reshape represents a tensor reshaping operation, FC represents a fully connected operation, and Softmax represents a normalization operation.

10. The method according to claim 8, characterized in that The feature matrix with local information for extracting image features input into the MCFA module includes: Perform two convolution operations on the image features input to the MCFA module to obtain the first image features; Performing two dilated convolution operations with a first dilated rate on the image features input to the MCFA module to obtain a second image feature; Performing two dilated convolution operations with a second dilated rate on the image features input to the MCFA module to obtain a third image feature; A splicing operation is performed on the first image feature, the second image feature and the third image feature, and a convolution operation is performed on the obtained spliced ​​image feature to obtain a feature matrix with local information.

Images