Infrared image small target segmentation method based on spatial channel attention network

Abstract

The application provides an infrared image small target segmentation method based on a spatial channel attention network, which comprises the following steps: inputting an infrared image to be processed into a preset encoder to perform step-by-step iterative feature extraction and down-sampling, and extracting low-level semantic features of different scales of a target region at different feature processing levels, wherein the encoder comprises a plurality of cascaded sub-encoders, each of which is composed of a preset MCFA module and a down-sampling module, and the output feature of the MCFA module of each sub-encoder is a low-level semantic feature of the current feature processing level; inputting the low-level semantic features output by each feature processing level in the encoder into the sub-decoder of the corresponding feature reconstruction level in the preset decoder respectively, performing step-by-step iterative up-sampling, fused feature and feature reconstruction, and finally obtaining the features of the target region segmented from the infrared image. The application can accurately identify the target region in the infrared image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a small target segmentation method for infrared images based on a spatial channel attention network. Background Art

[0002] Infrared small target segmentation has widespread applications in maritime rescue, maritime surveillance, infrared guidance, forest fire prevention, and other fields. However, this task presents various challenges. First, due to the long imaging distance, small infrared targets are typically smaller than 30×30 pixels. Second, small infrared targets are often submerged in complex backgrounds, resulting in low contrast. Finally, small infrared targets lack shape, texture, and color information.

[0003] To address the above challenges, a large number of model-driven methods have been proposed, including local contrast-based methods, morphology-based methods, low-rank sparse matrix-based methods, and filter-based methods. Local contrast-based methods use local differences to suppress the background and enhance the target. Morphology-based methods extract and adjust the morphological features of the target and background regions to achieve segmentation and extraction of small infrared targets. Low-rank sparse matrix-based methods use the sparse nature of small infrared targets and the low-rank properties of the background to transform the segmentation task into a classification task. Filter-based methods use designed filters to estimate the background of infrared images and achieve background suppression. However, model-driven methods rely on parameter selection and manual debugging, and their segmentation performance is poor in scenes with complex backgrounds and low signal-to-noise ratios.

[0004] Unlike model-driven approaches, data-driven methods based on deep learning can automatically learn features from large datasets of complex scenes, giving the network model a certain degree of adaptability and generalization when dealing with small infrared targets in complex scenes. Consequently, various networks specifically designed for small infrared target detection have been proposed. These box-level methods use bounding boxes to address, to a certain extent, the problem of poor segmentation performance in scenes with complex backgrounds and low signal-to-noise ratios, and are more resistant to interference from complex backgrounds than traditional methods. However, target detection methods generally cannot well describe the shape and size of the target, and cannot accurately detect small targets.

[0005] In practice, semantic segmentation provides pixel-level classification results and is better suited to handling small objects in complex backgrounds. Modeling infrared small target detection as a semantic segmentation problem can better address the performance loss of models due to the small size of the objects. However, the convolution operation in semantic segmentation methods can only capture local features and lacks global information perception, making it prone to segmenting background noise as objects. Furthermore, due to the small size of small infrared objects, multiple downsampling processes can easily result in the loss of small objects, a process that is irreversible.

[0006] Furthermore, the Transformer introduces a self-attention mechanism to promote global learning of correlations between different regions, enabling it to capture long-range dependencies in images, which can better represent high-level semantic information. However, due to the diverse shapes and sizes of small infrared targets, multi-scale feature extraction is crucial. However, the Transformers designed by these methods only have single-scale feature extraction capabilities, resulting in poor generalization for the segmentation of small targets of varying shapes and sizes. Summary of the Invention

[0007] The present invention proposes a small target segmentation method for infrared images based on spatial channel attention network to overcome or partially overcome the above problems.

[0008] The present invention provides a method for small target segmentation in infrared images based on a spatial channel attention network, the method comprising:

[0009] Inputting the infrared image to be processed into a preset encoder to perform iterative feature extraction and downsampling step by step, extracting low-level semantic features of the infrared image at different scales at different feature processing levels, wherein the encoder includes multiple cascaded sub-encoders, each sub-encoder corresponds to a feature processing level, and each sub-encoder is composed of a preset MCFA module and a downsampling module. The output features of the MCFA module of each sub-encoder are the low-level semantic features of the current feature processing level. The MCFA module represents a multi-scale convolutional fusion attention module, which is used to extract low-level semantic features of the target area by combining global information and local information of the infrared image;

[0010] The low-level semantic features output by each feature processing level in the encoder are respectively input into the sub-decoder of the corresponding feature reconstruction level in the preset decoder, and upsampling, feature fusion and feature reconstruction are performed step by step to fuse the low-level semantic features of the infrared image at different scales into the high-level semantic features of the infrared image at corresponding scales obtained in the upsampling process of the sub-decoder, so as to obtain the features of the target area in the infrared image, wherein the decoder includes multiple cascaded sub-decoders, each sub-decoder corresponding to a feature reconstruction level.

[0011] Furthermore, the step of inputting the low-level semantic features outputted by each feature processing level in the encoder into the sub-decoders of the corresponding feature reconstruction levels in the preset decoder comprises:

[0012] The low-level semantic features output from the second feature processing layer to the last feature processing layer are respectively input into the HGCA module of the sub-decoder in the corresponding feature reconstruction layer, wherein the sub-decoders from the second level to the last level are respectively composed of an upsampling module, an HGCA module and an ARU module. The HGCA module represents the hierarchical guided channel attention module, and the ARU module represents the void residual module.

[0013] The low-level semantic features of the first-level feature processing layer are input into the feature splicing module in the sub-decoder of the first-level feature reconstruction layer, wherein the first-level sub-decoder includes an upsampling module, a feature splicing module and an ARU module, and the ARU module of the first-level sub-decoder outputs the features of the target area.

[0014] Furthermore, in the process of inputting the low-level semantic features output by each feature processing level in the encoder into the sub-decoder of the corresponding feature reconstruction level in the preset decoder, the method further includes:

[0015] The image features output by the downsampling module of the last-level sub-encoder in the encoder are input into the transition ARU module between the encoder and the decoder, and the image features output by the transition ARU module are input into the decoder to perform step-by-step iterative upsampling, wherein the transition ARU module and the ARU module in the sub-decoder have the same structure.

[0016] Furthermore, the HGCA module is used to complete the following operations:

[0017] Obtain high-level semantic features input to the HGCA module, wherein the high-level semantic features are semantic features output by the upsampling module in the sub-encoder to which the HGCA module belongs,

[0018] After performing an average pooling operation and a maximum pooling operation on the high-level semantic features, respectively, performing an addition operation on the features obtained by performing the average pooling operation and the features obtained by performing the maximum pooling operation to obtain a first high-level image feature;

[0019] Performing a convolution operation, a batch normalization operation, and a ReLu operation on the first high-level image feature in sequence to obtain a second high-level image feature;

[0020] The third high-level image feature is obtained by sequentially performing a convolution operation, a batch normalization operation, and a Sigmoid operation on the second high-level image feature;

[0021] Obtaining low-level semantic features input to the HGCA module, performing a matrix multiplication operation on the low-level semantic features and the third high-level image features to obtain fused image features;

[0022] The fused image features and the high-level semantic features are concatenated to obtain output features of the HGCA module.

[0023] Furthermore, the ARU module is used to complete the following operations:

[0024] Obtain image features input to the ARU module;

[0025] Performing step-by-step iterative residual downsampling on the image features input to the ARU module to obtain residual features of different iteration levels, wherein, in each iteration level of residual downsampling, a hole residual block is used to obtain the residual features of the current iteration level, and the residual features of the current iteration level are downsampled and input into the next iteration level;

[0026] After downsampling the residual features of the last iterative level, the upsampled image features are obtained;

[0027] Performing iterative residual upsampling on the upsampled image features to obtain residual upsampling features of different iteration levels, wherein residual upsampling is performed in each iteration level of residual upsampling using a cascaded hole residual block and an upsampling module, the upsampling module outputs the residual upsampling features of the current iteration level, and the residual features of the corresponding iteration levels from the second level to the last level are respectively spliced ​​with the residual upsampling features and input into the previous upsampling iteration level;

[0028] The residual features of the first level and the residual up-sampled features of the first level are added to obtain the image features output by the ARU module.

[0029] Furthermore, the structures of the various hole residual blocks in the step-by-step iterative residual downsampling process and the various hole residual blocks in the step-by-step iterative residual upsampling process are the same, and different features of the target area are captured by setting different hole rates for the hole residual blocks in different iteration levels.

[0030] Furthermore, the hole residual block is used to perform the following operations:

[0031] Get the features of the input hole residual block;

[0032] Performing dilated convolution, batch normalization, and ReLu operations on the features of the input dilated residual block in sequence to obtain a first dilated feature;

[0033] Perform convolution and batch normalization operations on the first hole feature to obtain the second hole image feature;

[0034] Performing an addition budget on the features of the input hole residual block and the second hole image features to obtain a third hole feature;

[0035] After performing the ReLu operation on the third hole feature, the features of the output hole residual block are obtained.

[0036] Furthermore, the MCFA module is used to complete the following operations:

[0037] Obtain image features input to the MCFA module;

[0038] Extract the feature matrix with local information of the image features input to the MCFA module, and extract the attention matrix of the image features input to the MCFA module,

[0039] Performing a multiplication operation on the feature matrix with local information and the attention matrix to obtain a feature matrix with global information;

[0040] The feature matrix with global information, the feature matrix with local information and the image features input to the MCFA module are fused to obtain the low-level semantic features output by the MCFA module:

[0041] F O =Conv 1×1 (αF av +F v +F)

[0042] Among them, F O is the low-level semantic feature output by the MCFA module, F is the image feature input to the MCFA module, and F v is the feature matrix with local information, F av is the feature matrix with global information, F a is the attention matrix, and α represents the learnable weight.

[0043] Furthermore, the feature matrix with local information for extracting image features input to the MCFA module includes:

[0044] The following operations are performed on the image features input to the MCFA module to map the image features input to the MCFA module to a query vector:

[0045] F q1 =reshape(FC(reshape(Conv 1×1 (F))))

[0046] The following operations are performed on the image features input to the MCFA module to map the image features input to the MCFA module to a Key vector:

[0047] Fk1 =reshape(FC(reshape(Conv 1×1 (F))))

[0048] Perform the following operations on the Query vector and the Key vector to obtain the attention matrix:

[0049] F a =Softmax(reshape(F q1 ×F k1 ))

[0050] Among them, Conv 1×1 Represents a 1×1 convolution operation, reshape represents a tensor reshaping operation, FC represents a fully connected operation, and Softmax represents a normalization operation.

[0051] Furthermore, the feature matrix with local information for extracting image features input to the MCFA module includes:

[0052] Perform two convolution operations on the image features input to the MCFA module to obtain the first image features;

[0053] Perform two dilated convolution operations with a first dilation rate on the image features input to the MCFA module to obtain a second image feature;

[0054] Perform two dilated convolution operations with a second dilation rate on the image features input to the MCFA module to obtain a third image feature;

[0055] A splicing operation is performed on the first image feature, the second image feature, and the third image feature, and a convolution operation is performed on the obtained spliced ​​image feature to obtain a feature matrix with local information.

[0056] An embodiment of the present application provides a method for segmenting small targets in infrared images based on a spatial channel attention network. The present invention simultaneously obtains global information and local information of the target area in the infrared image based on a multi-scale convolutional fusion attention module, better understands the overall morphology and background of the target area in the infrared image, and adopts a skip-layer link method to hierarchically integrate the low-level semantic features of the infrared image in the downsampling process into the high-level semantic features of the infrared image in the upsampling process, which further helps to select the features of the target area and retain the key features of the target area. At the same time, through step-by-step upsampling, the feature reconstruction of the target area is realized, and finally the pixel-level semantic segmentation result of the infrared image is generated, and the target area in the infrared image is accurately identified.

[0057] The above description is only a summary of the technical solutions of the present application, in order to enable the technical means of the present application to be more clearly understood, and can be implemented according to the content of the specification, and in order to enable the above and other purposes, characteristics and advantages of the present application to be more apparent and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0058] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not meant to limit the present application. In the drawings:

[0059] Figure 1 A flow chart of an infrared image small target segmentation method based on a spatial channel attention network according to an embodiment of the present application;

[0060] Figure 2 A network architecture schematic diagram of an infrared image small target segmentation method based on a spatial channel attention network according to an embodiment of the present application;

[0061] Figure 3 An architecture schematic diagram of an MCFA module according to an embodiment of the present application;

[0062] Figure 4 An architecture schematic diagram of an HGCA module according to an embodiment of the present application;

[0063] Figure 5 An architecture schematic diagram of an ARU module according to an embodiment of the present application;

[0064] Figure 6 A segmentation result schematic diagram according to an embodiment of the present application. DETAILED DESCRIPTION

[0065] Exemplary embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms without being limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0066] Before introducing the infrared image small target segmentation method based on a spatial channel attention network according to an embodiment of the present application, the principles needed in the present application are briefly introduced:

[0067] (1) CNN

[0068] The network structure of CNN is similar to that of artificial neural networks and is currently widely used in the fields of image segmentation, face recognition, and target detection. CNN uses a weight sharing method, which can greatly reduce the parameters of neurons, thereby simplifying the model. For multidimensional images, CNN can identify spatial features and relationships in multidimensional directions, a property that other neural networks do not have. In an image, CNN can extract low-level image features such as edges or angles of the image, and then use convolutional layers to repeatedly extract higher-level image features, making CNN's feature extraction ability better than manually operated features. CNN mainly includes the following six network layers:

[0069] 1) Input layer: The first layer of CNN, used to input image data.

[0070] 2) Convolutional Layer: As the most important layer in a CNN, its primary function is to learn and extract features from the input image. Convolution kernels of different sizes extract different image features. Typically, the early convolution layers extract low-level features. After acquiring these low-level features, subsequent convolution layers repeatedly extract higher-level features. These higher-level features can then be used to segment and classify the image.

[0071] 3) Activation function layer: The activation function layer is used in CNN to improve the nonlinearity of the network to improve the data fitting performance of the network. In addition, it can also prevent the network gradient from disappearing during the process of extracting image features. The activation functions commonly used in CNN are:

[0072] Sigmoid function: Mainly used for binary classification problems. The Sigmoid function maps the input to (0,1) and is expressed as follows:

[0073]

[0074] The Sigmoid activation function causes network gradients to vanish. As can be seen from the formula, when the input is infinite or infinitely small, its derivative approaches 0, causing network gradient updates to slow down or vanish. Furthermore, its output mean is non-zero, complicating the calculation of the next layer. Therefore, it is usually only used in the output layer.

[0075] Corrected Linear Unit: Converges faster than the hyperbolic tangent function. When the input value is less than 0, the input is mapped to 0. When the input value is greater than or equal to 0, the output value is equal to the input value. The function expression is as follows:

[0076]

[0077] 4) Pooling Layer: The pooling layer is also a key component of CNNs. Its primary function is to implement both downsampling and upsampling. Downsampling reduces image resolution, significantly reducing the number of network parameters and preventing overfitting. Its translational invariance and increased image resolution are crucial for both segmentation and classification. Pooling layers are a crucial tool often used in neural networks to prevent overfitting.

[0078] 5) Fully connected layer: After the image data is automatically extracted through multiple convolutions and pooling, the features extracted at this time are usually deeper features. Then they pass through the fully connected layer and finally output the classification results.

[0079] 6) Output layer: The last layer of CNN, used to output the classifier of the number of categories.

[0080] (2) Transformer

[0081] Unlike traditional RNNs and CNNs, Transformer networks lack recurrent and convolutional structures, enabling them to process each element in a sequence in parallel, accelerating training and inference. Transformers consist of two parts: an encoder and a decoder. The encoder converts the input sequence into a series of high-dimensional vectors, while the decoder incrementally generates the target sequence based on the encoder's output and previous predictions. Each encoder and decoder consists of multiple layers of Transformer blocks, each composed of two sublayers: a multi-head self-attention machine and a feedforward neural network.

[0082] Position encoding: Position encoding is a very important step in Transformer. It can assign a position information to each element in the input sequence, so that Transformer can process the position information of the elements in the sequence.

[0083] Multi-head self-attention mechanism: The self-attention mechanism is a mechanism that can learn the relationship between different positions in the input sequence. In each Transformer block of the encoder and decoder, the multi-head self-attention mechanism maps the input sequence into a triple of a key, a query, and a value. By calculating the dot product of the query and the key, an attention score vector is obtained, and then the attention score vector is multiplied by the value vector to obtain the self-attention vector. In this process, the multi-head self-attention mechanism divides the attention mechanism into multiple heads, calculates the attention score vectors separately, and splices multiple self-attention vectors together, and then performs dimensionality reduction processing through the fully connected layer. The specific calculation formula is as follows:

[0084]

[0085] MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W o (5) Where Q, K, and V represent the amount of data mapped from the input sequence to Query, Key, and Value, respectively. W o Represents different weight matrices, head i Represents the score vector calculated by the i-th head attention module.

[0086] The multi-head attention mechanism can be understood as a mechanism that maps the original vector set into multiple subspaces and independently calculates attention weights in each subspace. Specifically, the input query, key, and value vectors are linearly transformed into vectors in multiple subspaces, and then dot-product attention weights are calculated in each subspace. Finally, the attention information obtained in each subspace is concatenated and subjected to another linear transformation to obtain the final output vector.

[0087] Feedforward Neural Network: The feedforward neural network is another important component in the Transformer. In each Transformer block, the feedforward neural network accepts the output from the multi-head self-attention mechanism as input and maps it into a new representation vector through two linear transformations and a nonlinear activation function. Specifically, it can be expressed as:

[0088] FFN(x)=max(0,xW1+b1)W2+b2 (6)

[0089] Where W1 and W2 represent two different linear functions, b1 and b2 are two bias constants, and the activation function used is the ReLU function.

[0090] (3) Dilated Convolution

[0091] For images where the target area is small relative to the global image, a multi-scale feature extractor is needed to expand the receptive field. Therefore, based on the UNet, the first convolution of each layer in the encoding module and the convolution of the bottom transition layer are replaced with dilated convolutions of different dilation rates. This expands the receptive field and captures the characteristic information of objects around the image.

[0092] Atrous convolution theory is widely used in computer vision tasks. It maintains feature resolution by increasing the receptive field. Atrous convolution can be defined as follows:

[0093]

[0094] where f out [i] is the output feature, f in[i] is the input feature, d is the dilation rate, and ω[k] is the kth parameter of the filter, and k is the filter size.

[0095] Further, f out [i] represents the value of the output feature at position i. It is calculated by taking the weighted sum of the input feature and the filter. f in [i+d*k]: the value of the input feature at position i+d*k. Here, d represents the dilation rate, which controls the interval between the sampled positions in the convolution operation. By increasing d, the input feature can be sampled over a larger range without increasing the size of the convolution kernel. ω[k] represents the weight parameter of the convolution kernel at position k. It is used to weight the input feature at the corresponding position.

[0096] Unlike traditional convolution, Atrous convolution introduces a dilation rate d, allowing the convolution kernel to sample at intervals of d, thus expanding the receptive field while maintaining the resolution of the features. Specifically: maintain the resolution of the features: Atrous convolution can expand the receptive field without increasing the size of the feature map. Enhance multi-scale feature extraction: by setting different dilation rates d, Atrous convolution can extract features at different scales, so it is particularly suitable for tasks such as semantic segmentation, target detection, etc. that require multi-scale information. In this formula, d controls the interval of sampling, i.e. the sampling interval of the convolution kernel on the input feature. The role of the dilation rate is to determine the "jump" sampling distance during convolution operation. When d = 1, the formula is equivalent to the ordinary convolution operation, the sampling interval is 1, and the convolution kernel acts continuously on the input feature. When d > 1, the convolution kernel samples at intervals of d on the input feature, which expands the receptive field of the convolution kernel.

[0097] Further, Figure 1 An infrared image small target segmentation method based on a spatial channel attention network is shown in an embodiment of the present application, referring to Figure 1 It can be seen that the infrared image small target segmentation method based on the spatial channel attention network comprises the following steps:

[0098] S1, input the infrared image to be processed into a preset encoder to perform step-by-step iterative feature extraction and down-sampling, and extract low-level semantic features of different scales of the infrared image at different feature processing levels, wherein the encoder comprises a plurality of cascaded sub-encoders, each sub-encoder corresponds to a feature processing level, and each sub-encoder is composed of a preset MCFA module and a down-sampling module. The output feature of the MCFA module of each sub-encoder is the low-level semantic feature of the current feature processing level, the MCFA module represents a multi-scale convolution fusion attention module, and is used for combining global information and local information of the infrared image to extract low-level semantic features of the target region;

[0099] This paper rethinks the relationship between local and global information of the image, improves the Transformer architecture, and designs the MCFA module, so that the MCFA module can focus on the global and local information of the infrared image by calculating the attention matrix, which can effectively enhance the features of small targets and suppress noise.

[0100] S2. Input the low-level semantic features output by each feature processing level in the encoder into the sub-decoder of the corresponding feature reconstruction level in the preset decoder, perform iterative upsampling, feature fusion and feature reconstruction step by step, so as to fuse the low-level semantic features of the infrared image at different scales into the high-level semantic features of the infrared image at corresponding scales obtained in the upsampling process of the sub-decoder, and obtain the features of the target area in the infrared image, wherein the decoder includes multiple cascaded sub-decoders, and each sub-decoder corresponds to a feature reconstruction level.

[0101] During the iterative upsampling process, the present invention uses skip connections to map and reconstruct the low-level semantic features extracted during the downsampling process with the high-level semantic features of the infrared image, and gradually restores the resolution of the infrared image to preserve the key features of the target image and ultimately accurately identify the target area in the image. It should be noted that the present invention considers layers at the same feature processing scale in the decoder and encoder to be the same layer.

[0102] Furthermore, in an embodiment of the present invention, the fusion of low-level semantic features of infrared images at different scales into high-level semantic features of the corresponding scales of the infrared images obtained during the upsampling process of the sub-decoder is implemented by the HGCA module. The HGCA module implements the fusion of the first semantic features and high-level semantic features in the corresponding feature processing levels of the encoder and decoder.

[0103] Figure 2 FIG2 shows a network architecture diagram of a method for segmenting small targets in infrared images based on a spatial channel attention network according to another specific embodiment of the present invention. Figure 2 Specifically shows how the encoding process and decoding process of infrared images are jump-connected, as well as the detailed implementation of the decoding process of infrared images. Figure 2 The specific network architecture shown uses an encoder-decoder structure similar to the classic UNet. This design aims to promote effective interaction between low-level and high-level feature information through a specific hierarchical structure, enabling the model to better integrate details and global information in the image.

[0104] The encoder part consists of five MCFA modules, which are used to extract multi-level features of the image. In the decoder part, the model contains five ARUs, which are responsible for mapping the encoded features back to the resolution of the original image. In addition, there is an additional ARU as a transition layer at the bottom. In order to maintain the feature transfer and consistency between the encoder and decoder, jump connection layers are used. These connections are implemented through HGCA modules to ensure that features at different levels can be effectively fused. In the first stage of the model, the encoder module is responsible for extracting the key features of the input image. These features include global and local key information to ensure that the model can capture important patterns in the image. Next, these extracted features will be mapped and reconstructed in the decoder, gradually restoring the resolution of the image while retaining key features. Finally, after multi-level feature reconstruction, the model generates pixel-level semantic segmentation results for infrared images, accurately identifying the target area in the image.

[0105] Specifically, in conjunction with Figure 1 and attached Figure 2 It can be seen that in step S2, a specific embodiment of inputting the low-level semantic features output by each feature processing level in the encoder into the sub-decoder of the corresponding feature reconstruction level in the preset decoder includes the following steps:

[0106] S21, respectively inputting the low-level semantic features output from the second feature processing level to the last feature processing level into the HGCA module of the sub-decoder in the corresponding feature reconstruction level, wherein the sub-decoders from the second level to the last level are respectively composed of an upsampling module, an HGCA module and an ARU module, the HGCA module represents the hierarchical guided channel attention module, and the ARU module represents the void residual module;

[0107] S22. Input the low-level semantic features of the first-level feature processing layer into the feature splicing module in the sub-decoder of the first-level feature reconstruction layer, wherein the first-level sub-decoder includes an upsampling module, a feature splicing module and an ARU module, and the ARU module of the first-level sub-decoder outputs the features of the target area.

[0108] Further, by Figure 2It can be seen that in the process of inputting the low-level semantic features output by each feature processing level in the encoder into the sub-decoder of the corresponding feature reconstruction level in the preset decoder in step S2, the present invention also includes the following method: inputting the image features output by the downsampling module of the last sub-encoder in the encoder into the transition ARU module between the encoder and the decoder, and inputting the image features output by the transition ARU module into the decoder to perform step-by-step iterative upsampling, wherein the transition ARU module and the ARU module in the sub-decoder have the same structure. It can be understood that the image features output by the transition ARU module are input into the upsampling module of the bottom feature reconstruction level in the decoder.

[0109] Another specific embodiment of the present invention also includes a hierarchical guided channel attention (HGCA) module and an atrous residual U-block (ARU). The HGCA module enables high-level features to guide low-level features, assigns weights to decisive channels, and reduces redundant information. This can reduce background noise in infrared images and make small targets stand out more clearly against the background. The ARU, located in a deeper layer of the network, extracts more refined features from the feature map, enabling the model to identify the features of small infrared targets in more detail and comprehensively. The various modules of the present invention are further described below in conjunction with the accompanying drawings.

[0110] like Figure 3 As shown, the MCFA module of the present invention is used to perform the following operations:

[0111] S11, obtaining image features input to the MCFA module;

[0112] S12, extracting the feature matrix with local information of the image features input to the MCFA module, and extracting the attention matrix of the image features input to the MCFA module,

[0113] S13, performing a multiplication operation on the feature matrix with local information and the attention matrix to obtain a feature matrix with global information;

[0114] S14. Fusing the feature matrix with global information, the feature matrix with local information, and the image features input to the MCFA module to obtain low-level semantic features output by the MCFA module:

[0115] F O =Conv 1×1 (αF av +F v +F) (8)

[0116] Among them, F O is the low-level semantic feature output by the MCFA module, F is the image feature input to the MCFA module, and F v is the feature matrix with local information, Fav is the feature matrix with global information, F a is the attention matrix, and α represents the learnable weight.

[0117] Furthermore, the feature matrix with local information for extracting image features input to the MCFA module includes:

[0118] The following operations are performed on the image features input to the MCFA module to map the image features input to the MCFA module to a query vector:

[0119] F q1 =reshape(FC(reshape(Conv 1×1 (F)))) (9)

[0120] The following operations are performed on the image features input to the MCFA module to map the image features input to the MCFA module to a Key vector:

[0121] F k1 =reshape(FC(reshape(Conv 1×1 (F)))) (10)

[0122] Perform the following operations on the Query vector and the Key vector to obtain the attention matrix:

[0123] F a =Softmax(reshape(F q1 ×F k1 )) (11)

[0124] Among them, Conv 1×1 Represents a 1×1 convolution operation, reshape represents a tensor reshaping operation, FC represents a fully connected operation, and Softmax represents a normalization operation.

[0125] Furthermore, the feature matrix with local information for extracting image features input to the MCFA module includes:

[0126] Perform two convolution operations on the image features input to the MCFA module to obtain the first image features;

[0127] Perform two dilated convolution operations with a first dilation rate on the image features input to the MCFA module to obtain a second image feature;

[0128] Perform two dilated convolution operations with a second dilation rate on the image features input to the MCFA module to obtain a third image feature;

[0129] A stitching operation is performed on the first image feature, the second image feature and the third image feature, and a convolution operation is performed on the obtained stitched image feature, to obtain a feature matrix with local information.

[0130] In specific embodiments of the application, the first hole rate can be set to 2, and the second hole rate can be set to 4, to obtain and Then, F1, F2 and F4 are concatenated after 1x1 Conv to obtain F v H×W×C .

[0131] The MCFA module of the application redesigns the self-attention architecture and introduces convolution and dilated convolution, fuses the attention matrix obtained by calculation with the features extracted by multi-scale convolution to obtain local and global information of the image at the same time, which can effectively enhance the small target features and suppress noise. The MCFA module is mainly composed of convolution, dilated convolution and fully connected layer, which is used to extract multi-level features of the image.

[0132] Further, in addition to extracting global information of small targets, channel attention plays an important role in the small target segmentation network, which can help the network better capture target features, reduce information redundancy, and improve the adaptability and generalization ability of the network. In the HGCA-Former model, we introduce a hierarchical guided channel attention (HGCA) module between the encoder and the decoder, and the architecture is as shown in Figure 4 .

[0133] The decoder contains rich high-level semantic information, which helps to guide the selection of low-level information. Through the HGCA module, the HGCA-Former model can adaptively weight different channel features and effectively utilize different channel information, so that the network pays more attention to the small target itself and improves the generalization ability of the model in different scenes. After the image is extracted by the MCFA block, the features are input into the HGCA block. In addition, the features processed by the ARU are also input into the HGCA block to guide the selection of low-level information, further better capture target features and reduce information redundancy.

[0134] Specifically, the HGCA module of the application is used to complete the following operations:

[0135] S31, obtaining high-level semantic features input into the HGCA module, wherein the high-level semantic features are semantic features output by an up-sampling module in a sub-encoder to which the HGCA module belongs;

[0136] S32, after performing an average pooling operation and a maximum pooling operation on the high-level semantic features, performing an addition operation on the features obtained by performing the average pooling operation and the features obtained by performing the maximum pooling operation to obtain a first high-level image feature;

[0137] S33, sequentially performing a convolution operation, a batch normalization operation, and a ReLu operation on the first high-level image feature to obtain a second high-level image feature;

[0138] S34, performing a convolution operation, a batch normalization operation, and a Sigmoid operation on the second high-level image feature in sequence to obtain a third high-level image feature;

[0139] S35, obtaining low-level semantic features input to the HGCA module, performing a matrix multiplication operation on the low-level semantic features and the third high-level image features to obtain fused image features;

[0140] S36: performing a splicing operation on the fused image features and the high-level semantic features to obtain output features of the HGCA module.

[0141] like Figure 4 As shown, the HGCA module has two inputs, namely, high-level semantic features Coarse H×W×C and low-level semantic features Fine H×W×C First, Coarse performs AvgPool and MaxPool operations respectively and then adds them together to get Then, F h1 Convolution, batch normalization and ReLu operations are performed in sequence to obtain Finally, F h2 Convolution, batch normalization and Sigmoid operations are performed in sequence to obtain Then, the high-level features F h3 Multiplying with the low-level feature Fine, we get Finally, F hl Concatenate with Coarse to get the final output of the HGCA block

[0142] Furthermore, after the input image passes through the MCFA module, most of the noise is filtered out, and the target features are more prominent when passed to deeper layers of the network. In order to better capture multi-scale features, we designed the Atrous Residual U-block (ARU), as shown in Figure 5As shown. We use ARU for the bottom transition layer between the network encoder and decoder and the decoder part. ARU is a U-Net-like encoder-decoder structure, and the backbone is a residual block with holes. ARU performs down-sampling and up-sampling, and can obtain multi-scale feature maps from the input feature maps. This further extracts the global and local features of the image, which is beneficial to separate the noise component from the background branch. We use the residual connection of the residual block to compensate for the features, preventing the loss of small target information caused by convolution operations. The residual block uses a hole convolution to improve the depth feature resolution of each layer, and each layer sets a different hole rate to capture multi-scale features of small targets.

[0143] Further, the ARU module of the present application is used to complete the following operations:

[0144] S41, obtaining image features input into the ARU module,

[0145] S42, performing step-by-step iterative residual down-sampling on the image features input into the ARU module to obtain residual features of different iteration levels, wherein the residual features of the current iteration level are obtained by using a residual block with holes in each iteration level of residual down-sampling, and the residual features of the current iteration level are input into the next iteration level after down-sampling;

[0146] S43, obtaining up-sampled image features by down-sampling the residual features of the last iteration level;

[0147] S44, performing step-by-step iterative residual up-sampling on the up-sampled image features to obtain residual up-sampling features of different iteration levels, wherein the residual up-sampling features of the current iteration level are output by the up-sampling module in each iteration level of residual up-sampling by using a cascaded residual block with holes and an up-sampling module, and the residual features of the 2nd to last iteration level are input into the previous up-sampling iteration level after being spliced with the residual up-sampling features;

[0148] S45, performing addition operation on the residual features of the 1st level and the residual up-sampling features of the 1st level to obtain image features output by the ARU module.

[0149] It should be noted that the structures of each residual block with holes in the step-by-step iterative residual down-sampling process and each residual block with holes in the step-by-step iterative residual up-sampling process are the same, and different features of the target region are captured by setting different hole rates for the residual blocks with holes in different iteration levels.

[0150] Further, the residual block with holes is used to perform the following operations:

[0151] Obtaining features input into the residual block with holes;

[0152] Performing dilated convolution, batch normalization, and ReLu operations on the features of the input dilated residual block in sequence to obtain a first dilated feature;

[0153] Perform convolution and batch normalization operations on the first hole feature to obtain the second hole image feature;

[0154] Performing an addition budget on the features of the input hole residual block and the second hole image features to obtain a third hole feature;

[0155] After performing the ReLu operation on the third hole feature, the features of the output hole residual block are obtained.

[0156] The ARU module provided in this embodiment of the present invention fully utilizes the characteristics of deep network information. Therefore, the residual block in the ARU uses dilated convolution, which helps extract more refined features and helps more accurately capture the details of small targets. The ARU design can better process a wider range of pixels and effectively filter out noise around small infrared targets. After ARU processing, we can obtain a more refined small target feature map.

[0157] In a specific embodiment of the present invention, various SOTA methods are used to segment small targets on five infrared images of the SIRST public dataset. The segmentation results are as follows: Figure 5 As shown in the figure, we magnify the target area and place it at the top, and use orange, yellow, and green circles to represent accurately detected targets, undetected targets, and false alarms, respectively. Figure 5 The experimental results of the present invention on the segmentation of small targets in infrared images are shown. Figure 5 It can be seen that the method of the present invention can accurately obtain the small target segmentation result by segmenting the small target in the infrared image.

[0158] Embodiments of the present invention provide a method for infrared image small target segmentation based on a spatial channel attention network, and are based on an improved Transformer infrared small target segmentation model: HGCA-Former. HGCA-Former improves on the Transformer structure, redesigning the self-attention architecture and introducing convolution and dilated convolution. By calculating an attention matrix and fusing it with features extracted by multi-scale convolution, it simultaneously captures local and global image information. This effectively enhances small target features and suppresses noise, avoiding information loss during the downsampling process. Furthermore, by introducing hierarchical guided channel attention, high-level features guide low-level features, adaptively weighting features from different channels, effectively utilizing information from different channels and reducing redundant information. This reduces background noise in infrared images, making small targets stand out more clearly against the background, thereby improving the model's generalization ability for infrared small target segmentation in different scenarios. The design of MCFA and HGCA enables HGCA-Former to suppress noise, enhance small target features, and accurately segment small targets. HGCA-Former brings a new perspective to infrared small target segmentation. Its robustness and stability make it a popular choice for real-world applications and can better support real-time applications across various devices.

[0159] Those skilled in the art will understand that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art in the art to which the present invention pertains. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with those in the context of the prior art and, unless specifically defined, will not be interpreted in an idealized or overly formal sense.

[0160] For simplicity of description, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because certain steps can be performed in other orders or simultaneously according to the embodiments of the present invention. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0161] Furthermore, those skilled in the art will appreciate that although some embodiments herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, any of the claimed embodiments may be used in any combination.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for small target segmentation in infrared images based on spatial channel attention network, characterized in that: The method comprises: Inputting the infrared image to be processed into a preset encoder to perform iterative feature extraction and downsampling step by step, extracting low-level semantic features of the infrared image at different scales at different feature processing levels, wherein the encoder includes multiple cascaded sub-encoders, each sub-encoder corresponds to a feature processing level, and each sub-encoder is composed of a preset MCFA module and a downsampling module. The output features of the MCFA module of each sub-encoder are the low-level semantic features of the current feature processing level. The MCFA module represents a multi-scale convolutional fusion attention module, which is used to extract low-level semantic features of the target area by combining global information and local information of the infrared image; The low-level semantic features output by each feature processing level in the encoder are respectively input into the sub-decoder of the corresponding feature reconstruction level in the preset decoder, and upsampling, feature fusion and feature reconstruction are performed step by step to fuse the low-level semantic features of the infrared image at different scales into the high-level semantic features of the infrared image at corresponding scales obtained in the upsampling process of the sub-decoder, so as to obtain the features of the target area in the infrared image, wherein the decoder includes a plurality of cascaded sub-decoders, each sub-decoder corresponding to a feature reconstruction level; wherein, The step of inputting the low-level semantic features outputted by each feature processing level in the encoder into the sub-decoders of the corresponding feature reconstruction levels in the preset decoder comprises: The low-level semantic features output from the second feature processing layer to the last feature processing layer are respectively input into the HGCA module of the sub-decoder in the corresponding feature reconstruction layer, wherein the sub-decoders from the second level to the last level are respectively composed of an upsampling module, an HGCA module and an ARU module. The HGCA module represents the hierarchical guided channel attention module, and the ARU module represents the void residual module. The low-level semantic features of the first-level feature processing layer are input into the feature splicing module in the sub-decoder of the first-level feature reconstruction layer, wherein the first-level sub-decoder includes an upsampling module, a feature splicing module and an ARU module, and the ARU module of the first-level sub-decoder outputs the features of the target area.

2. The method according to claim 1, characterized in that In the process of inputting the low-level semantic features output by each feature processing level in the encoder into the sub-decoder of the corresponding feature reconstruction level in the preset decoder, the method further includes: The image features output by the downsampling module of the last-level sub-encoder in the encoder are input into the transition ARU module between the encoder and the decoder, and the image features output by the transition ARU module are input into the decoder to perform step-by-step iterative upsampling, wherein the transition ARU module and the ARU module in the sub-decoder have the same structure.

3. The method according to claim 1, characterized in that The HGCA module is used to complete the following operations: Obtain high-level semantic features input to the HGCA module, wherein the high-level semantic features are semantic features output by the upsampling module in the sub-encoder to which the HGCA module belongs, After performing an average pooling operation and a maximum pooling operation on the high-level semantic features, respectively, performing an addition operation on the features obtained by performing the average pooling operation and the features obtained by performing the maximum pooling operation to obtain a first high-level image feature; Performing a convolution operation, a batch normalization operation, and a ReLu operation on the first high-level image feature in sequence to obtain a second high-level image feature; The third high-level image feature is obtained by sequentially performing a convolution operation, a batch normalization operation, and a Sigmoid operation on the second high-level image feature; Obtaining low-level semantic features input to the HGCA module, performing a matrix multiplication operation on the low-level semantic features and the third high-level image features to obtain fused image features; The fused image features and the high-level semantic features are concatenated to obtain output features of the HGCA module.

4. The method according to claim 1 or 2, characterized in that The ARU module is used to complete the following operations: Obtain image features input to the ARU module; Performing step-by-step iterative residual downsampling on the image features input to the ARU module to obtain residual features of different iteration levels, wherein, in each iteration level of residual downsampling, a hole residual block is used to obtain the residual features of the current iteration level, and the residual features of the current iteration level are downsampled and input into the next iteration level; After downsampling the residual features of the last iterative level, the upsampled image features are obtained; Performing iterative residual upsampling on the upsampled image features to obtain residual upsampling features of different iteration levels, wherein residual upsampling is performed in each iteration level of residual upsampling using a cascaded hole residual block and an upsampling module, the upsampling module outputs the residual upsampling features of the current iteration level, and the residual features of the corresponding iteration levels from the second level to the last level are respectively spliced ​​with the residual upsampling features and input into the previous upsampling iteration level; The residual features of the first level and the residual up-sampled features of the first level are added to obtain the image features output by the ARU module.

5. The method according to claim 4, characterized in that The structures of the dilated residual blocks in the step-by-step iterative residual downsampling process and the dilated residual blocks in the step-by-step iterative residual upsampling process are the same. Different features of the target area are captured by setting different dilated rates for the dilated residual blocks in different iteration levels.

6. The method according to claim 5, characterized in that The hole residual block is used to perform the following operations: Get the features of the input hole residual block; Performing dilated convolution, batch normalization, and ReLu operations on the features of the input dilated residual block in sequence to obtain a first dilated feature; Perform convolution and batch normalization operations on the first hole feature to obtain the second hole image feature; Performing an addition budget on the features of the input hole residual block and the second hole image features to obtain a third hole feature; After performing the ReLu operation on the third hole feature, the features of the output hole residual block are obtained.

7. The method according to claim 1, characterized in that The MCFA module is used to complete the following operations: Obtain image features input to the MCFA module; Extract the feature matrix with local information of the image features input to the MCFA module, and extract the attention matrix of the image features input to the MCFA module, Performing a multiplication operation on the feature matrix with local information and the attention matrix to obtain a feature matrix with global information; The feature matrix with global information, the feature matrix with local information and the image features input to the MCFA module are fused to obtain the low-level semantic features output by the MCFA module: F O =Conv 1×1 (αF av +F v +F) Among them, F O The low-level semantic features output by the MCFA module, Conv 1×1 represents a 1×1 convolution operation, F is the image feature of the input MCFA module, and F v is the feature matrix with local information, F av is a feature matrix with global information, and α represents the learnable weight.

8. The method according to claim 7, characterized in that The feature matrix with local information for extracting image features input to the MCFA module includes: The following operations are performed on the image features input to the MCFA module to map the image features input to the MCFA module to a query vector: F q1 =reshape(FC(reshape(Conv 1×1 (F)))) The following operations are performed on the image features input to the MCFA module to map the image features input to the MCFA module into a Key vector: F k1 =reshape(FC(reshape(Conv 1×1 (F)))) Perform the following operations on the Query vector and the Key vector to obtain the attention matrix: F a =Softmax(reshape(F q1 ×F k1 )) Among them, Conv 1×1 Represents a 1×1 convolution operation, reshape represents a tensor reshaping operation, FC represents a fully connected operation, and Softmax represents a normalization operation.

9. The method according to claim 7, characterized in that The feature matrix with local information for extracting image features input to the MCFA module includes: Perform two convolution operations on the image features input to the MCFA module to obtain the first image features; Perform two dilated convolution operations with a first dilation rate on the image features input to the MCFA module to obtain a second image feature; Perform two dilated convolution operations with a second dilation rate on the image features input to the MCFA module to obtain third image features; A splicing operation is performed on the first image feature, the second image feature, and the third image feature, and a convolution operation is performed on the obtained spliced ​​image feature to obtain a feature matrix with local information.

Images