Infrared small target detection method of edge and semantic collaborative network

Abstract

The invention provides an infrared small target detection method based on an edge and semantic collaborative network, and the method comprises the steps: fusing dynamically focused edge information into a space attention map through explicit edge supervision by employing multi-scale edge perception attention feature extraction, and obtaining a lowest-level collaborative feature and a highest-level collaborative feature; through edge and semantic collaborative feature extraction, the lowest-level coding feature is used for guiding extraction of edge detail features of the middle-level coding features, the highest-level coding feature is used for guiding extraction of target area features of the middle-level coding features, and middle collaborative features are obtained; and guiding the decoding operation of the collaborative features of the current level by using the decoding features of the previous level and the collaborative features before the current level so as to retain the semantic information of the high-level features, and finally obtaining a small target image in the target infrared image. According to the method, the edge detail information of the low-level features and the semantic information of the high-level features are fully utilized, and the infrared small target is effectively segmented.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an infrared small target detection method based on an edge and semantic collaborative network. Background Art

[0002] Infrared Small Target Detection (IRSTD) technology, as a key research direction in the field of photoelectric detection, has wide application needs in important fields such as aerospace, disaster monitoring, etc.

[0003] Early research primarily relied on models based on filtering, local contrast, and low-rank representations, leveraging physical models and prior knowledge to extract target features. However, these approaches exhibited poor robustness in challenging scenarios (complex backgrounds and low contrast), often leading to significantly increased missed detection and false detection rates. Furthermore, their reliance on manual feature design limited their ability to mine latent information in the data, making them difficult to meet the demands of practical applications. The rapid development of convolutional neural networks (CNNs), with their ability to learn and mine deep latent information from large amounts of data, has significantly improved the performance of IRSTD methods. Many IRSTD methods based on encoder-decoder architectures have performed well. However, when targets are set against complex, brightly lit backgrounds, they often suffer from missed detections or incomplete prediction of target structure and edges due to the low contrast between the target and the background. Two main reasons have been identified for this phenomenon: 1) Insufficient cross-level feature collaboration: Low-level features are rich in detail (such as object edges) but lack semantic expression; while high-level features possess strong semantic information, their details are severely lost due to multiple downsampling. Existing methods attempt to enhance single-level features or exploit context from adjacent levels by designing enhancement modules in the encoder and decoder. While achieving satisfactory results, these two approaches ignore cues from long-range hierarchical features, resulting in insufficient cross-hierarchical feature coordination, limiting the ability to model global context, and leading to varying degrees of missed detections and false detections. 2) Insufficient utilization of edge information: Existing edge-aware methods often extract a rough edge map from the original image, input it into independent edge branches, and then enhance the edge features of the target branch through a designed fusion strategy. Due to the different semantic levels of the target branch and the edge branch, it is difficult to maintain consistency during fusion, resulting in poor fusion results and a lack of edge details in the predicted target. Summary of the Invention

[0004] The present invention proposes an infrared small target detection method based on edge and semantic collaborative network to overcome or partially overcome the above problems.

[0005] The present invention provides an infrared small target detection method based on an edge and semantic collaborative network, the method comprising:

[0006] Performing multi-level encoding on the target infrared image to extract encoding features of different levels of the target area in the target infrared image;

[0007] Multi-scale edge-aware attention feature extraction is performed on the lowest-level coding features and the highest-level coding features, respectively, so as to integrate the edge information after dynamic focus into the spatial attention map through explicit edge supervision to guide the extraction of features of the target area, and obtain the lowest-level collaborative features and the highest-level collaborative features respectively;

[0008] Edge and semantic collaborative features are extracted from the intermediate-level coding features respectively, so as to use the lowest-level coding features to guide the extraction of edge detail features of the intermediate-level coding features, and use the highest-level coding features to guide the extraction of target area features of the intermediate-level coding features, and obtain the intermediate-level collaborative features of each intermediate-level coding feature through weighted feature fusion;

[0009] The collaborative features of different levels are decoded, and in the decoding process of each level, the decoding features of the higher level and the collaborative features of each higher level before the current level are used to guide the decoding operation of the collaborative features of the current level, so as to retain the semantic information of the high-level features and finally obtain the small target image in the target infrared image.

[0010] Furthermore, the multi-scale edge-aware attention feature extraction is performed on the lowest-level coding features and the highest-level coding features, respectively, including:

[0011] Performing dilated convolution operations of different scales on the target coding features to be processed to obtain dilated convolution features of different scales, and performing cross-layer feature fusion on the dilated convolution features of different scales to obtain multi-scale context features; wherein the target coding features include the lowest-level coding features or the highest-level coding features;

[0012] Performing edge-focused embedded attention feature extraction on the multi-scale contextual features to extract an edge-enhanced attention map of the multi-scale contextual features, and locally enhancing features of a target area in the multi-scale contextual features using the edge-enhanced attention map to obtain an edge-focused embedded feature map;

[0013] The edge-focused embedded feature map is convolved and then fused with dilated convolution features of different scales to obtain a multi-scale edge-aware attention feature of the target coding feature.

[0014] Furthermore, performing edge-focused embedded attention feature extraction on the multi-scale context features includes:

[0015] The channel feature weighted optimization of multi-scale context features is performed through the preset first-channel attention mechanism to increase the weight of channels that are effective for small object extraction and obtain channel-optimized context features;

[0016] Performing edge prediction on the channel optimization context feature based on a preset target edge loss to obtain an edge prediction map, and normalizing the edge prediction map to obtain an edge weight map;

[0017] Performing spatial position prediction on the channel-optimized context features based on a preset first spatial attention mechanism to obtain a spatial attention map;

[0018] Adding the edge weight map and the spatial attention map element by element to obtain an edge enhanced attention map;

[0019] The edge-enhanced attention map and the channel-optimized context feature are element-wise multiplied to obtain an edge-focused embedding feature map.

[0020] Furthermore, said extracting edge and semantic collaborative features from the intermediate-level coding features respectively includes:

[0021] Perform multi-scale edge-aware attention feature extraction on the lowest-level coding feature to extract edge detail information of the lowest-level coding feature to obtain the target multi-scale edge-aware attention feature; perform feature fusion on the target multi-scale edge-aware attention feature and the coding feature of the target level to obtain the target edge detail feature;

[0022] Perform dual-channel attention adjustment on the high-level coding features and the target-level coding features to extract the channel attention information of the target area and obtain the target channel attention map; perform feature fusion on the high-level coding features and the target-level coding features based on the target channel attention map to obtain the target semantic guidance features;

[0023] The target edge detail features and target semantic guidance features are fused through a cross-attention mechanism to obtain the edge and semantic collaborative features of the encoding features at the target level.

[0024] Furthermore, the target multi-scale edge perception attention features and the target level encoding features are fused to obtain target edge detail features, including:

[0025] The encoding features of the target level are purified through a gating mechanism to retain edge detail features and obtain target gating features;

[0026] The target multi-scale edge-aware attention feature and the target gated feature are channel-concatenated and then convolved to obtain the first edge branch feature.

[0027] The target multi-scale edge-aware attention feature and the target gated feature are element-wise added and then convolved to obtain the second edge branch feature;

[0028] The target multi-scale edge-aware attention feature and the target gated feature are element-wise multiplied and then convolved to obtain the third edge branch feature.

[0029] The first edge branch feature, the second edge branch feature, the third edge branch feature and the encoding feature of the target level are residually connected to obtain the target edge detail feature.

[0030] Furthermore, the dual-channel attention adjustment is performed on the high-level coding features and the target-level coding features to extract the channel attention information of the target area and obtain the target channel attention map, including:

[0031] Performing global average pooling processing on the high-level coding features and the target-level coding features respectively to obtain a high-level channel description vector and a target channel description vector; splicing the high-level channel description vector and the target channel description vector to obtain a spliced ​​pooling vector;

[0032] A two-layer fully connected operation is performed on the concatenated pooled vector to extract channel interaction information through the first layer of fully connected operation, and the target channel attention map is obtained after the dimension is restored through the second layer of fully connected operation.

[0033] Furthermore, the feature fusion of the high-level coding features and the target-level coding features based on the target channel attention map includes:

[0034] Multiply the target channel attention map and the encoding features of the target layer channel by channel to obtain the channel attention features;

[0035] Based on the preset second spatial attention mechanism, feature extraction is performed on the high-level encoding features to obtain a high-level spatial attention map;

[0036] Multiply the channel attention feature and the high-level spatial attention map element-wise to obtain the first semantic branch feature;

[0037] Multiply the high-level spatial attention map and the encoding features of the target layer element by element to obtain the second semantic branch features;

[0038] The target level encoding features, the first semantic branch features and the second semantic branch features are residually connected to obtain the target semantic guidance features.

[0039] Furthermore, the feature fusion of the target edge detail features and the target semantic guidance features through the cross attention mechanism includes:

[0040] Perform channel projection on the target edge detail features through 1x1 convolution to obtain the query tensor;

[0041] The target semantic guide features are channel-projected through 1x1 convolution to obtain key tensors and value tensors;

[0042] Perform channel-level matrix multiplication on the query tensor and the key tensor to calculate the spatial position similarity of the query tensor and the key tensor on each channel to obtain a spatial position similarity matrix;

[0043] The value tensor is weightedly fused based on the spatial position similarity matrix to highlight important spatial position features, and is residually connected with the target semantic guidance features to obtain edge and semantic collaborative features.

[0044] Furthermore, in the decoding process of each level, the decoding features of the previous level and the high-level collaborative features before the current level are used to guide the decoding operation of the collaborative features of the current level, including:

[0045] After upsampling the high-level collaborative features before the current level to the spatial resolution of the previous level, they are multiplied element by element to obtain high-level guidance features.

[0046] The collaborative features of the current level and the high-level guidance features are multiplied element by element to obtain the current reference features; the decoding features of the previous level of the current level and the high-level guidance features are multiplied element by element to obtain the upper-level reference features;

[0047] After performing the convolution operation on the current reference feature, it is fused with the collaborative feature of the current level by element-by-element addition to obtain the first branch decoding feature;

[0048] After performing the convolution operation on the upper-level reference feature, it is fused with the decoding feature of the previous layer by element-by-element addition to obtain the second-branch decoding feature;

[0049] Perform a convolution operation on the high-level guidance feature and concatenate it with the first-branch decoding feature and the second-branch decoding feature to obtain a pre-decoding feature;

[0050] Perform a decoding operation on the pre-decoding features to obtain the decoding features of the current level.

[0051] Furthermore, in the decoding process of each level, the decoding features of the previous level and the high-level collaborative features before the current level are used to guide the decoding operation of the collaborative features of the current level, and further includes:

[0052] If the current level is the highest level, the highest level collaborative feature is used as the highest level decoding feature;

[0053] After concatenating the highest-level collaborative features with the second-highest-level collaborative features, a decoding operation is performed to obtain the second-highest-level decoding features.

[0054] The present invention provides a method for infrared small target detection using an edge and semantic collaborative network. After obtaining coding features at different levels, the method uses multi-scale edge-aware attention feature extraction to extract the lowest-level coding features and the highest-level coding features through explicit edge supervision, integrating the dynamically focused edge information into the spatial attention map to guide the extraction of target area features, thereby obtaining the lowest-level collaborative features and the highest-level collaborative features. The edge and semantic collaborative feature extraction respectively uses the lowest-level coding features to guide the extraction of edge detail features of intermediate-level coding features, and uses the highest-level coding features to guide the extraction of target area features of intermediate-level coding features, thereby obtaining intermediate collaborative features. During the decoding operation of each level, the decoding features of the previous level and the collaborative features before the current level are used to guide the decoding operation of the collaborative features of the current level, thereby preserving the semantic information of the high-level features and the boundary information of the low-level features, ultimately obtaining a small target image in the target infrared image. The present invention fully utilizes the edge detail information of the low-level features and the semantic information of the high-level features, so that key information of different levels can be preserved, has high generalization and robustness, and effectively segments infrared small targets.

[0055] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Various other advantages and benefits will become apparent to those skilled in the art by reading the detailed description of the preferred embodiment below. The accompanying drawings are only for the purpose of illustrating the preferred embodiment and are not to be considered as limiting the present invention. In the accompanying drawings:

[0057] Figure 1 This is a flow chart of an infrared small target detection method using an edge and semantic collaborative network according to an embodiment of the present invention;

[0058] Figure 2 Schematic diagram of the structure of an edge and semantic collaborative network (ESSNet) according to an embodiment of the present invention;

[0059] Figure 3 Schematic diagram of the structure of the multi-scale edge-aware attention module (MSEA) according to an embodiment of the present invention;

[0060] Figure 4 Schematic diagram of the structure of the edge and semantic collaboration module (ESSM) according to an embodiment of the present invention;

[0061] Figure 5 Schematic diagram of the structure of the multi-level feature fusion module (MLFF) of an embodiment of the present invention. DETAILED DESCRIPTION

[0062] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0063] It should be noted that the infrared small target detection method of the edge and semantic collaborative network in the embodiment of the present invention is based on the pre-built edge and semantic collaborative network (ESSNet), wherein: Figure 1 An infrared small target detection method based on an edge and semantic collaborative network according to an embodiment of the present invention is shown. Figure 2 This is a schematic diagram of the structure of the edge and semantic collaborative network (ESSNet) of an embodiment of the present invention, referring to Figure 1 and Figure 2 It can be seen that the infrared small target detection method of the edge and semantic collaborative network of the present invention includes the following steps:

[0064] S1. Perform multi-level encoding on the target infrared image to extract coding features of different levels of the target area in the target infrared image;

[0065] In a specific embodiment of the present invention, an encoder is designed based on ResNet, specifically comprising five stages. The first stage consists of two layers of ordinary convolution, and the remaining four stages are four residual blocks. To reduce subsequent computational costs, the present invention uniformly adjusts the number of output channels of each block to 32 by using convolutional layers. Since ResNet is a state-of-the-art technology in this field, this invention does not provide a detailed description of the encoder designed based on ResNet. The encoding process of the present invention can also be implemented using encoders in other prior art technologies, so this will not be repeated here.

[0066] S2. Perform multi-scale edge-aware attention feature extraction on the lowest-level coding features and the highest-level coding features, respectively, to integrate the edge information after dynamic focus into the spatial attention map through explicit edge supervision to guide the extraction of features of the target area, and obtain the lowest-level collaborative features and the highest-level collaborative features respectively;

[0067] The embodiment of the present invention is designed as follows Figure 3The multi-scale edge-aware attention module (MSEA) shown in the figure can directly process the lowest-level and highest-level coding features to refine the edge details of the lowest-level coding features and recover edge clues in the semantically rich but spatially coarse highest-level coding features. At the same time, the multi-scale edge-aware attention module (MSEA) can also coordinate the processing of coding features at each intermediate level to refine the edge details of each intermediate-level coding feature.

[0068] S3. Extract edge and semantic collaborative features from the intermediate-level coding features, respectively, to use the lowest-level coding features to guide the extraction of edge detail features of the intermediate-level coding features, and use the highest-level coding features to guide the extraction of target area features of the intermediate-level coding features, and obtain intermediate-level collaborative features of each intermediate-level coding feature through weighted fusion;

[0069] The embodiment of the present invention is designed as follows Figure 4 The edge and semantic collaboration module (ESSM) shown in the figure can process the coding features of each intermediate level. The characteristics of small infrared targets are blurred edges, weak features and easy to be confused with the background, which makes it difficult to achieve ideal results by relying solely on semantic information or edge information. In response to these challenges, the present invention proposes an edge and semantic collaboration module (ESSM) to promote the lowest-level edge features and the highest-level semantic features to perform edge semantic collaborative enhancement on the current coding features. Specifically, it extracts edge details from the lowest-level coding features and the current-level coding features through the detail refinement block (DRB), uses the highest-level coding features and the current-level coding features to guide semantic positioning through the semantic-guided positioning block (SGL), and aggregates the outputs of the detail refinement block (DRB) and the semantic-guided positioning block (SGL) through the collaborative aggregation block (SAB) to enhance target representation.

[0070] S4. Decoding operations are performed on collaborative features of different levels, and in the decoding process of each level, the decoding features of the higher level and the collaborative features of each higher level before the current level are used to guide the decoding operation of the collaborative features of the current level, so as to retain the semantic information of the high-level features, and finally obtain the small target image in the target infrared image.

[0071] The embodiment of the present invention is designed as follows Figure 4 During the decoding process, semantic information is gradually diluted as it is transferred from deep layers to shallow layers in the multi-level feature fusion module (MLFF). This can lead to missing target structures or over-prediction of foreground structures. The proposed multi-level feature fusion module (MLFF) promotes interaction between feature layers through a multi-level guidance mechanism, fusing features from different encoder granularities to improve segmentation completeness and accuracy.

[0072] The embodiment of the present invention fully utilizes the edge detail information of low-level features and the semantic information of high-level features, so that key information of different levels can be retained, has high generalization and robustness, and effectively segments small infrared targets.

[0073] The following is a detailed description of each feature processing module of the present invention. Figure 3 In the embodiment of the present invention, the multi-scale edge-aware attention feature extraction is performed on the lowest-level coding feature and the highest-level coding feature respectively, and the target coding feature is extracted based on the preset multi-scale edge-aware attention module (MSEA). The target coding feature can be the lowest-level coding feature, the highest-level coding feature, or the intermediate processing feature of other levels, which specifically includes the following steps:

[0074] S21. Performing dilated convolution operations of different scales on the target coding features to be processed to obtain dilated convolution features of different scales, and performing cross-layer feature fusion on the dilated convolution features of different scales to obtain multi-scale context features; wherein the target coding features include the lowest-level coding features or the highest-level coding features;

[0075] Small infrared targets vary in size and variety, and it is often difficult for the network to capture clear edge information. Therefore, the embodiment of the present invention enhances the network's ability to capture contextual features of different scales through feature extraction methods. The present invention extracts rich multi-scale contextual information through densely connected multiple dilated convolutions of different scales (3×3 dilated convolutions with a dilation rate of 2N+1, where N is the number of dilated convolutions). The gradually increasing receptive field captures features from local edges to global structures, while the dense connection enhances the integrity and diversity of the context through cross-layer feature fusion, ultimately generating multi-scale contextual features, which provides a robust feature basis for subsequent processing.

[0076] Furthermore, in an embodiment of the present invention, cross-layer feature fusion of dilated convolution features of different scales is performed specifically by fusing the dilated convolution features of different scales through element-by-element addition, that is, after performing the dilated convolution operation on the dilated convolution features of the previous scale, the dilated convolution features of all previous scales are fused through element-by-element addition to obtain the dilated convolution features of the current scale.

[0077] S22, performing edge-focused embedded attention feature extraction on the multi-scale contextual features to extract an edge-enhanced attention map of the multi-scale contextual features, and using the edge-enhanced attention map to locally enhance features of a target area in the multi-scale contextual features to obtain an edge-focused embedded feature map;

[0078] In an embodiment of the present invention, edge-focused embedded attention features are extracted from multi-scale contextual features through an edge-focused embedded attention module (EFIA), and the dynamically focused edge information is effectively integrated into the spatial attention map through explicit edge supervision to guide the network to learn clear and accurate target boundary features.

[0079] S23, performing a convolution operation on the edge-focused embedded feature map and performing feature fusion with the dilated convolution features of different scales to obtain a multi-scale edge-aware attention feature of the target coding feature.

[0080] In an embodiment of the present invention, in order to further enhance the feature learning capability and effectively utilize multi-level information, the features from the input and the dilated convolution features of different scales and the edge-focused embedded feature maps after two 3×3 convolutions are fused into the final features.

[0081] Furthermore, in step S21, the edge-focused embedded attention feature extraction of the multi-scale context features in the embodiment of the present invention specifically includes:

[0082] S211. Performing channel feature weighted optimization on the multi-scale context features through a preset first-channel attention mechanism to increase the weights of channels that are effective for extracting small objects, thereby obtaining channel-optimized context features.

[0083] This step can be expressed as:

[0084]

[0085] In the above formula, represents channel optimized context features, CA represents channel attention, represents multi-scale context features, and ⊙ represents channel-by-channel multiplication.

[0086] This embodiment of the present invention calculates the channel weight vector through the first-channel attention mechanism to express the importance of different channels for target region feature extraction. It then recalibrates the original multi-scale contextual features through channel feature weighted optimization, suppressing unimportant or noisy channels while enhancing channels containing useful information (especially those related to edges and structures). This performs a preliminary "purification" and "focusing" of the input features, highlighting features that are more valuable for subsequent tasks (edge ​​extraction and spatial attention).

[0087] S212: performing edge prediction on the channel optimization context feature based on a preset target edge loss to obtain an edge prediction map, and normalizing the edge prediction map to obtain an edge weight map. This step can be expressed as:

[0088]

[0089] In the above formula, Represents the edge prediction map, conv 3×3 Represents a 3×3 convolution operation, conv 1×31 After obtaining the edge prediction map, the specific embodiment of the present invention activates the edge prediction map through the sigmoid function to obtain the edge weight map

[0090] The embodiment of the present invention uses a preset target edge loss to force the network to predict the precise boundary position of the target from the channel-optimized features, obtains the probability that each pixel value represents "belonging to the target edge", and converts the boundary information from an implicit feature to an explicit representation, thereby preventing the boundary features from being submerged in the complex background. At the same time, an edge weight map is obtained by activating and processing the edge prediction map, so that pixels predicted as edges are high-weight areas and non-edge areas are low-weight areas. In view of the dynamic changes in edge positions of different images / targets, the embodiment of the present invention can adaptively highlight key boundary areas with the weight map.

[0091] S213, performing spatial position prediction on the channel optimization context feature based on a preset first spatial attention mechanism to obtain a spatial attention map;

[0092] In the embodiment of the present invention, the spatial attention map is expressed as The spatial attention map is a matrix of the same size as the input feature map (such as H×W). The weight value (0-1) of each pixel position represents the relative importance of the feature at that position. The high-weight area (close to 1) is the position that the network needs to focus on, and the low-weight area (close to 0) is the suppressible background or irrelevant area.

[0093] S214, adding the edge weight map and the spatial attention map element by element to obtain an edge enhanced attention map;

[0094] In the embodiment of the present invention, the edge weight map provides local high-precision boundary positioning, and the spatial attention map provides global semantic perception. After element-by-element feature fusion, the attention can have both global target positioning capability and local boundary sharpening capability.

[0095] S215. Multiply the edge-enhanced attention map and the channel-optimized context feature element-by-element to obtain an edge-focused embedding feature map.

[0096] In the embodiment of the present invention, the edge enhanced attention map is expressed as The edge-enhanced attention map and the channel-optimized context feature are multiplied element-by-element to obtain an edge-focused embedding feature map, which can be expressed as:

[0097]

[0098] In the above formula, represents the edge-focused embedding feature map, In the embodiment of the present invention, the enhanced attention map is used to guide the extraction channel to optimize the features of the target area in the context feature, so as to take into account both edge information and spatial information.

[0099] Further, refer to Figure 4 In step S3, the present invention uses an edge and semantic collaboration module (ESSM) to process the coding features of each intermediate level. The edge and semantic collaboration feature extraction of the intermediate level coding features specifically includes the following steps:

[0100] S31, performing multi-scale edge-aware attention feature extraction on the lowest-level coding feature to extract edge detail information of the lowest-level coding feature to obtain a target multi-scale edge-aware attention feature, and performing feature fusion on the target multi-scale edge-aware attention feature and the coding feature of the target level to obtain a target edge detail feature;

[0101] In the embodiment of the present invention, the lowest level coding is used for edge guidance for each intermediate level coding feature, that is, Figure 3 The multi-scale edge-aware attention module (MSEA) shown in FIG. 3 extracts the lowest-level coding feature F1. It should be noted that, since the resolutions of features at different feature processing levels are different, when the multi-scale edge-aware attention module (MSEA) is used to extract the lowest-level coding feature F1 at each feature processing level, the lowest-level coding feature F1 needs to be downsampled from 32×H×W to 32×(H / 2 t-1 )×(W / 2 t-1 ) to adapt to the resolution of the current feature processing level.

[0102] S32, performing dual-channel attention adjustment on the high-level coding features and the coding features of the target level to extract the channel attention information of the target area and obtain a target channel attention map; performing feature fusion on the high-level coding features and the coding features of the target level based on the target channel attention map to obtain a target semantic guidance feature;

[0103] The embodiment of the present invention performs dual-channel attention adjustment on high-level coding features and target-level coding features based on the Dual-Feature Channel Attention (DFCA) module, and its specific implementation method will be described in detail in subsequent embodiments.

[0104] S33. Feature fusion is performed on the target edge detail features and target semantic guidance features through a cross-attention mechanism to obtain edge and semantic collaborative features of the encoding features of the target level.

[0105] Furthermore, in the embodiment of the present invention, step S31 is performed by a detail refinement block (DRB) module, referring to Figure 4 In an embodiment of the present invention, the target multi-scale edge-perceived attention feature and the target level coding feature are fused to obtain the target edge detail feature, including: purifying the target level coding feature through a gating mechanism to retain the edge detail feature to obtain the target gating feature; performing a convolution operation on the target multi-scale edge-perceived attention feature and the target gating feature after channel splicing to obtain a first edge branch feature; performing a convolution operation on the target multi-scale edge-perceived attention feature and the target gating feature after element-by-element addition to obtain a second edge branch feature; performing a convolution operation on the target multi-scale edge-perceived attention feature and the target gating feature after element-by-element multiplication to obtain a third edge branch feature; performing a residual connection on the first edge branch feature, the second edge branch feature, the third edge branch feature and the target level coding feature to obtain the target edge detail feature.

[0106] In an embodiment of the present invention, the target level coding feature is purified by a gating mechanism to retain edge detail features, specifically comprising: performing nonlinear activation on the target level coding feature by a Sigmoid function, and then multiplying the result element-wise with the coding feature of the current channel to obtain the target gated feature. It is specifically expressed as:

[0107]

[0108] In the above formula, is the target gating feature, is the encoding feature of each channel of the target level.

[0109] The embodiment of the present invention obtains a gated map that can characterize edge information through a gating mechanism, and a weight matrix of the spatial dimension of the gating map. The larger the value, the more important the information. Then, by multiplying the element by element with the coding feature of the current channel, the edge information of the coding feature of the current channel can be retained, and the feature matrix can be made sparser, thereby achieving the purpose of purification.

[0110] This embodiment of the present invention performs channel concatenation, element-by-element addition, and element-by-element multiplication on each branch. Each branch's output passes through two 3x3 convolutional layers to ensure full integration of edge information and salient features. Finally, the three branches are subjected to residual fusion. This three-branch feature fusion approach helps refine detailed features while reducing information loss.

[0111] Furthermore, in the embodiment of the present invention, step S32 is performed by the semantic guidance positioning block (SGL), referring to Figure 4 In an embodiment of the present invention, dual-channel attention adjustment is performed on the high-level coding features and the coding features of the target level to extract the channel attention information of the target area, and the target channel attention map is obtained, including: performing global average pooling processing on the high-level coding features and the coding features of the target level respectively to obtain a high-level channel description vector and a target channel description vector; splicing the high-level channel description vector and the target channel description vector to obtain a spliced ​​pooling vector; performing a full connection operation on both sides of the spliced ​​pooling vector to extract channel interaction information through a first-layer full connection operation, and obtaining a target channel attention map after restoring the dimension through a second-layer full connection operation.

[0112] It can be specifically expressed as:

[0113]

[0114] Where, represents the target channel attention map, GAP is the global average pooling, FC1 is the first fully connected layer, FC2 is the second fully connected layer, Concat represents channel-by-channel concatenation, F t is the encoding feature of each channel of the target level.

[0115] Furthermore, in an embodiment of the present invention, the feature fusion of the high-level coding features and the coding features of the target level based on the target channel attention map includes: channel-by-channel multiplication of the target channel attention map and the coding features of the target level to obtain channel attention features; feature extraction of the high-level coding features based on a preset second spatial attention mechanism to obtain a high-level spatial attention map; element-by-element multiplication of the channel attention features and the high-level spatial attention map to obtain a first semantic branch feature; element-by-element multiplication of the obtained high-level spatial attention map and the coding features of the target level to obtain a second semantic branch feature; residual connection of the coding features of the target level, the first semantic branch features and the second semantic branch features to obtain a target semantic guided feature.

[0116] In this embodiment of the present invention, the target channel attention map and the encoding features of the target level are multiplied channel by channel to obtain the channel attention feature representation:

[0117]

[0118] In the above formula, represents the channel attention feature, F t represents the encoding feature of the tth level, represents the target channel attention map, and ⊙ represents channel-by-channel multiplication.

[0119] The feature fusion of each feature can be expressed as:

[0120]

[0121] In the above formula, represents the target semantic guided feature, UP represents upsampling, SA represents spatial attention, Represents element-wise multiplication.

[0122] The semantically guided positioning block (SGL) proposed in the embodiment of the present invention can use the semantic information of the highest-level coding features to guide the coding features of the target level in the channel and spatial dimensions to retain details and enhance the attention of the target area through the attention mechanism. Among them, the Dual-Feature Channel Attention (DFCA) module, under the guidance of semantic clues, shows higher sensitivity to the characteristics of the information channel than the traditional CA. Finally, the spatial attention map of the highest-order coding features is used to modulate the highest-order coding features and the coding features of the target level, selectively enhance the target-related spatial position, and fuse the features through residual connections to improve positioning accuracy.

[0123] Furthermore, the embodiment of the present invention uses a collaborative aggregation block (SAB) to perform feature fusion on the target edge detail features and target semantic guidance features through a cross-attention mechanism. The output of the detail refinement block (DRB) is rich in edge details and provides high-resolution local features; the output of the semantic guidance localization block (SGL) focuses on semantic information and captures global context and regional consistency. In order to collaboratively aggregate these two types of information, the embodiment of the present invention proposes a collaborative aggregation block (SAB) to achieve a better balance between details and semantics.

[0124] Reference Figure 4 The embodiment of the present invention uses a cross-attention mechanism to fuse the target edge detail features and the target semantic guidance features, including: performing channel projection on the target edge detail features through 1x1 convolution to obtain a query tensor; performing channel projection on the target semantic guidance features through 1x1 convolution to obtain a key tensor and a value tensor; performing channel-level matrix multiplication on the query tensor and the key tensor to calculate the spatial position similarity of the query tensor and the key tensor on each channel to obtain a spatial position similarity matrix; performing feature weighted fusion on the value tensor based on the spatial position similarity matrix to highlight important spatial position features, and performing residual connection with the target semantic guidance features to obtain edge and semantic collaborative features.

[0125] Specifically, the calculation process of query tensor, key tensor and value tensor is expressed as follows:

[0126]

[0127] In the above formula, Q represents the query tensor, K represents the key tensor, and V represents the value tensor, which are all C×H×W matrices. represents the target semantic guidance feature, Represents target edge detail features.

[0128] Furthermore, the channel-level matrix multiplication calculation of the query tensor and the key tensor is expressed as follows:

[0129] A=Softmax(Q#K T ), (9)

[0130] In the above formula, A represents the spatial position similarity matrix. Unlike traditional position self-attention (generating an attention map of (H×W)×(H×W)) and channel self-attention (generating an attention map of C×C), SAB focuses on modeling channel-level spatial dependencies. Specifically, it calculates the spatial position similarity of Q and K on each channel c through matrix multiplication. This is equivalent to calculating the product of the transpose (W×H) of its spatial dimension (H×W) and itself (H×W) for each channel c to generate a (H×W)×(H×W) matrix. Since this process is performed independently on the channel dimension, a three-dimensional attention map of dimension C×H×H is eventually generated (when H=W, it can also be expressed as C×H×W; strictly speaking, each channel corresponds to an H×H spatial attention map, and if H=W, the spatial dimension is H×W).

[0131] Furthermore, the value tensor is weightedly fused based on the spatial position similarity matrix to highlight important spatial position features, and is residually connected with the target semantic guidance features to express it as:

[0132]

[0133] In the above formula, γ is a weight parameter used to control the weight ratio of the fused features to the original input features. A residual connection is then made with the target semantic guidance features to ensure effective information transfer, allowing the network to gradually learn the importance of the fused features and stabilize the training process.

[0134] Unlike traditional positional self-attention (such as generating an (H×W)×(H×W) attention map) and channel self-attention (generating a C×C attention map), SAB focuses on channel-level spatial relationships to weigh the contributions of different feature sources, thereby achieving synergistic fusion. It calculates the spatial similarity of (H×W)×(W×H) on each channel through matrix multiplication, and generates a total attention map of C×H×H. In the case of H=W, it is C×H×W. This design avoids explicit reshaping and is implemented only through transpose and matrix multiplication, greatly reducing computational complexity and memory usage, while efficiently fusing multi-source features.

[0135] The edge and semantic collaboration module (ESSM) provided by the embodiment of the present invention explores and enhances the target edge information through the detail refinement block (DRB), reshapes the semantic information through the semantic guidance positioning block (SGL), and then dynamically models the edge and semantically important information in the collaborative aggregation block (SAB) to improve the model's ability to express significant target areas.

[0136] Furthermore, in step S3, during the decoding process, semantic information is easily diluted when it is transferred from deep layers to shallow layers, resulting in missing target structures or over-prediction of foreground. To address this problem, the embodiment of the present invention proposes a multi-level feature fusion module (MLFF), such as Figure 5 As shown in the figure, the multi-level feature fusion module (MLFF) promotes feature level interaction through a multi-level guidance mechanism, fuses features from different granularities of the encoder, and improves segmentation completeness and accuracy.

[0137] Further, refer to Figure 5 In an embodiment of the present invention, the decoding operation of the collaborative features of the previous layer and the high-level collaborative features before the current layer are used to guide the decoding operation of the collaborative features of the current layer during the decoding operation of each layer, including: upsampling the high-level collaborative features before the current layer to adjust to the spatial resolution of the previous layer, and then performing element-by-element multiplication to obtain the high-level guiding features; element-by-element multiplication of the collaborative features of the current layer and the high-level guiding features to obtain the current reference features; element-by-element multiplication of the decoding features of the previous layer of the current layer and the high-level guiding features to obtain the upper-level reference features; performing a convolution operation on the current reference features, and then performing feature fusion with the collaborative features of the current layer by element-by-element addition to obtain the first branch decoding features; performing a convolution operation on the upper-level reference features, and then performing feature fusion with the decoding features of the previous layer by element-by-element addition to obtain the second branch decoding features; performing a convolution operation on the high-level guiding features, and performing a splicing operation with the first branch decoding features and the second branch decoding features to obtain the pre-decoding features; performing a decoding operation on the pre-decoding features to obtain the decoding features of the current layer.

[0138] It should be noted that when the current level is the highest level, the decoding operation of each level described in the embodiment of the present invention utilizes the decoding features of the previous level and the high-level collaborative features before the current level to guide the decoding operation of the collaborative features of the current level, and also includes: if the current level is the highest level, the highest-level collaborative features are used as the highest-level decoding features; after splicing the highest-level collaborative features with the second-highest-level collaborative features, the decoding operation is performed to obtain the second-highest-level decoding features.

[0139] In a specific embodiment of the present invention, for a five-layer network, MLFF is applied to the last three stages of the encoder, and each MLFF receives the collaborative features of the current layer.t , the decoding feature M of the previous level t+1 , and all advanced collaborative features (e.g. O5, O4, O3).

[0140] Specifically, each high-level collaborative feature is adjusted to O by upsampling t The spatial resolution is then multiplied pixel by pixel to fuse higher-level collaborative features. Then it is injected into O t and M t+1 In the residual structure, the 3x3 convolution is updated to the first branch decoding feature and the second branch decoding features Afterwards, with and And the fusion features of the intermediate 3x3 convolution processing Splicing to generate pre-decoding features The formula is

[0141]

[0142] In the above formula, Represents the pre-decoding features of the current level, indicating The first branch decodes the features, represents the second branch decoding feature, Represents the features obtained after the high-level guidance features perform convolution operations.

[0143] By using this multi-level semantic feature guidance approach, MLFF effectively avoids the dilution of semantic information, ensuring that semantic information remains consistent and gradually strengthened during the decoding process. Furthermore, the level-by-level fusion and residual connection mechanisms ensure the stability of feature transfer and reduce the interference of complex backgrounds on the segmentation results. The pre-decoded features are decoded to obtain the decoded features of the current level. This effectively bridges the semantic gaps between features at different levels, ensuring high resolution and high quality of the segmentation results.

[0144] In this embodiment of the present invention, a new edge semantic collaboration network (ESSNet) is proposed to address the problems of small target size, blurred edges, and complex background interference in infrared small target detection, significantly improving the performance of infrared small target detection. ESSNet achieves deep collaboration between edge and semantic information through ESSM, MSEA, and MLFF. ESSM achieves multi-level feature interaction through remote modulation, MSEA optimizes the attention mechanism through explicit edge supervision to improve boundary detection accuracy, and MLFF alleviates semantic degradation through a hierarchical guidance mechanism to ensure the integrity of the target structure.

[0145] An embodiment of the present invention provides an infrared small target detection method based on an edge and semantic collaborative network. After obtaining coding features at different levels, the method uses multi-scale edge-aware attention feature extraction to extract the lowest-level coding features and the highest-level coding features through explicit edge supervision to integrate the edge information after dynamic focus into the spatial attention map to guide the extraction of target area features, thereby obtaining the lowest-level collaborative features and the highest-level collaborative features. The edge and semantic collaborative feature extraction respectively uses the lowest-level coding features to guide the extraction of edge detail features of intermediate-level coding features and uses the highest-level coding features to guide the extraction of target area features of intermediate-level coding features, thereby obtaining intermediate collaborative features. During the decoding operation of each level, the decoding features of the previous level and the collaborative features before the current level are used to guide the decoding operation of the collaborative features of the current level, thereby preserving the semantic information of the high-level features and the boundary information of the low-level features, thereby finally obtaining a small target image in the target infrared image. The present invention fully utilizes the edge detail information of the low-level features and the semantic information of the high-level features, thereby preserving the key information of different levels, having high generalization and robustness, and effectively segmenting infrared small targets.

[0146] Those skilled in the art will understand that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art in the art to which the present invention pertains. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with those in the context of the prior art and, unless specifically defined, will not be interpreted in an idealized or overly formal sense.

[0147] For simplicity of description, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because certain steps can be performed in other orders or simultaneously according to the embodiments of the present invention. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0148] Furthermore, those skilled in the art will appreciate that although some embodiments herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, any of the claimed embodiments may be used in any combination.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for infrared small target detection based on edge and semantic collaborative network, characterized in that: The method comprises: Performing multi-level encoding on the target infrared image to extract encoding features of different levels of the target area in the target infrared image; Multi-scale edge-aware attention feature extraction is performed on the lowest-level coding features and the highest-level coding features, respectively, so as to integrate the edge information after dynamic focus into the spatial attention map through explicit edge supervision to guide the extraction of features of the target area, and obtain the lowest-level collaborative features and the highest-level collaborative features respectively; Edge and semantic collaborative features are extracted from the intermediate-level coding features respectively, so as to use the lowest-level coding features to guide the extraction of edge detail features of the intermediate-level coding features, and use the highest-level coding features to guide the extraction of target area features of the intermediate-level coding features, and obtain the intermediate-level collaborative features of each intermediate-level coding feature through weighted feature fusion; The collaborative features of different levels are decoded, and in the decoding process of each level, the decoding features of the higher level and the collaborative features of each higher level before the current level are used to guide the decoding operation of the collaborative features of the current level, so as to retain the semantic information of the high-level features and finally obtain the small target image in the target infrared image.

2. The method according to claim 1, characterized in that The multi-scale edge-aware attention feature extraction is performed on the lowest-level coding features and the highest-level coding features, respectively, including: Performing dilated convolution operations of different scales on the target coding features to be processed to obtain dilated convolution features of different scales, and performing cross-layer feature fusion on the dilated convolution features of different scales to obtain multi-scale context features; wherein the target coding features include the lowest-level coding features or the highest-level coding features; Performing edge-focused embedded attention feature extraction on the multi-scale contextual features to extract an edge-enhanced attention map of the multi-scale contextual features, and locally enhancing features of a target area in the multi-scale contextual features using the edge-enhanced attention map to obtain an edge-focused embedded feature map; The edge-focused embedded feature map is convolved and then fused with dilated convolution features of different scales to obtain a multi-scale edge-aware attention feature of the target coding feature.

3. The method according to claim 2, characterized in that The performing edge-focused embedding attention feature extraction on the multi-scale context features includes: The channel feature weighted optimization of multi-scale context features is performed through the preset first-channel attention mechanism to increase the weight of channels that are effective for small object extraction and obtain channel-optimized context features; Performing edge prediction on the channel optimization context feature based on a preset target edge loss to obtain an edge prediction map, and normalizing the edge prediction map to obtain an edge weight map; Performing spatial position prediction on the channel-optimized context features based on a preset first spatial attention mechanism to obtain a spatial attention map; Adding the edge weight map and the spatial attention map element by element to obtain an edge enhanced attention map; The edge-enhanced attention map and the channel-optimized context feature are element-wise multiplied to obtain an edge-focused embedding feature map.

4. The method according to any one of claims 1 to 3, characterized in that The extracting edge and semantic collaborative features from the intermediate-level coding features respectively includes: Perform multi-scale edge-aware attention feature extraction on the lowest-level coding feature to extract edge detail information of the lowest-level coding feature to obtain the target multi-scale edge-aware attention feature; perform feature fusion on the target multi-scale edge-aware attention feature and the coding feature of the target level to obtain the target edge detail feature; Perform dual-channel attention adjustment on the high-level coding features and the target-level coding features to extract the channel attention information of the target area and obtain the target channel attention map; perform feature fusion on the high-level coding features and the target-level coding features based on the target channel attention map to obtain the target semantic guidance features; The target edge detail features and target semantic guidance features are fused through a cross-attention mechanism to obtain the edge and semantic collaborative features of the encoding features at the target level.

5. The method according to claim 4, characterized in that The feature fusion of the target multi-scale edge perception attention features and the target level encoding features to obtain the target edge detail features includes: The encoding features of the target level are purified through a gating mechanism to retain edge detail features and obtain target gating features; The target multi-scale edge-aware attention feature and the target gated feature are channel-concatenated and then convolved to obtain the first edge branch feature. The target multi-scale edge-aware attention feature and the target gated feature are element-wise added and then convolved to obtain the second edge branch feature; The target multi-scale edge-aware attention feature and the target gated feature are element-wise multiplied and then convolved to obtain the third edge branch feature. The first edge branch feature, the second edge branch feature, the third edge branch feature and the encoding feature of the target level are residually connected to obtain the target edge detail feature.

6. The method according to claim 4, characterized in that The dual-channel attention adjustment is performed on the high-level coding features and the target-level coding features to extract the channel attention information of the target area and obtain the target channel attention map, including: Performing global average pooling processing on the high-level coding features and the target-level coding features respectively to obtain a high-level channel description vector and a target channel description vector; splicing the high-level channel description vector and the target channel description vector to obtain a spliced ​​pooling vector; A two-layer fully connected operation is performed on the concatenated pooled vector to extract channel interaction information through the first layer of fully connected operation, and the target channel attention map is obtained after the dimension is restored through the second layer of fully connected operation.

7. The method according to claim 6, characterized in that The feature fusion of high-level coding features and target-level coding features based on the target channel attention map includes: Multiply the target channel attention map and the encoding features of the target layer channel by channel to obtain the channel attention features; Based on the preset second spatial attention mechanism, feature extraction is performed on the high-level encoding features to obtain a high-level spatial attention map; Multiply the channel attention feature and the high-level spatial attention map element-wise to obtain the first semantic branch feature; Multiply the high-level spatial attention map and the encoding features of the target layer element by element to obtain the second semantic branch features; The target level encoding features, the first semantic branch features and the second semantic branch features are residually connected to obtain the target semantic guidance features.

8. The method according to claim 4, characterized in that The feature fusion of the target edge detail features and the target semantic guidance features through the cross attention mechanism includes: Perform channel projection on the target edge detail features through 1x1 convolution to obtain the query tensor; The target semantic guide features are channel-projected through 1x1 convolution to obtain key tensors and value tensors; Perform channel-level matrix multiplication on the query tensor and the key tensor to calculate the spatial position similarity of the query tensor and the key tensor on each channel to obtain a spatial position similarity matrix; The value tensor is weightedly fused based on the spatial position similarity matrix to highlight important spatial position features, and is residually connected with the target semantic guidance features to obtain edge and semantic collaborative features.

9. The method according to claim 1, characterized in that The process of the decoding operation at each level using the decoding features of the previous level and the high-level collaborative features before the current level to guide the decoding operation of the collaborative features of the current level includes: After upsampling the high-level collaborative features before the current level to the spatial resolution of the previous level, they are multiplied element by element to obtain high-level guidance features. The collaborative features of the current level and the high-level guidance features are multiplied element by element to obtain the current reference features; the decoding features of the previous level of the current level and the high-level guidance features are multiplied element by element to obtain the upper-level reference features; After performing the convolution operation on the current reference feature, it is fused with the collaborative feature of the current level by element-by-element addition to obtain the first branch decoding feature; After performing the convolution operation on the upper-level reference feature, it is fused with the decoding feature of the previous layer by element-by-element addition to obtain the second-branch decoding feature; Perform a convolution operation on the high-level guidance feature and concatenate it with the first-branch decoding feature and the second-branch decoding feature to obtain a pre-decoding feature; Perform a decoding operation on the pre-decoding features to obtain the decoding features of the current level.

10. The method according to claim 9, characterized in that The method of using the decoding features of the previous level and the high-level collaborative features before the current level to guide the decoding operation of the collaborative features of the current level during the decoding operation of each level further includes: If the current level is the highest level, the highest level collaborative feature is used as the highest level decoding feature; After concatenating the highest-level collaborative features with the second-highest-level collaborative features, a decoding operation is performed to obtain the second-highest-level decoding features.

Images