Dynamic context sensing pyramid network for infrared weak and small target detection and infrared weak and small target detection method

Abstract

The invention relates to a dynamic context sensing pyramid network for infrared weak and small target detection and an infrared weak and small target detection method. Firstly, a dynamic context filtering module is used for adaptively capturing different size information of an input image to improve the feature representation capability of a target; secondly, a dynamic pyramid context module which is represented by multi-scale features generated by a plurality of dynamic context filtering modules is constructed, and the module can dynamically acquire multi-scale context information, so that more semantic information of the infrared small target is reserved in a deep network. And finally, realizing exchange of feature information of the same layer and establishment of association between different layers through a designed random attention fusion module, and ensuring fusion of deep semantic information and shallow spatial position information, thereby improving detection performance. The method can adapt to various complex backgrounds, reduces the false alarm rate and false alarm rate in actual detection, and has better target detection capability, background suppression capability and stronger robustness.

Description

Technical Field

[0001] The present invention relates to the field of infrared image small target detection, and in particular to a dynamic context-aware pyramid network for infrared small target detection and an infrared small target detection method. Background Art

[0002] Infrared small target detection is one of the key technologies of infrared search and tracking, and plays an important role in system warning, military guidance technology, and space target detection. However, due to the long imaging distance, the target size is usually a point with several pixels, without shape and texture information. Under the influence of complex background clutter, the target will be submerged in the background, resulting in a large number of missed detections.

[0003] Traditional target detection algorithms are sensitive to noise, and the detection effect is unstable when the target size varies greatly. At the same time, in actual image sequences, the grayscale distribution of the target does not always satisfy the two-dimensional Gaussian distribution. It is easily affected by the imaging distance, environment, and target type. The size of the target varies between 2*2 and 10*10, and the grayscale peak of the target is relatively discrete, making it difficult to detect accurately. In recent years, researchers have proposed many detection networks for small visible light targets. However, directly applying these networks to infrared small target detection tasks will not be able to accurately detect them, because the way to detect bounding boxes in visible light images is very different from that of infrared small targets. However, with the increase in the number of publicly available infrared small target data sets in recent years, infrared small target detection technology has developed from traditional manual feature extraction algorithms to machine learning, deep learning, and other directions. At the same time, pixel-level segmentation methods based on convolutional neural networks are widely used in infrared small target detection tasks and have achieved good detection results. However, the loss of small target information still exists in the deep layer, resulting in poor robustness to complex background changes, variable small targets, and dim situations. Summary of the invention

[0004] The present invention aims to solve the technical problems in the prior art and provides a dynamic context-aware pyramid network for infrared dim small target detection and an infrared dim small target detection method.

[0005] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0006] A dynamic context-aware pyramid network for infrared dim small target detection, comprising: a backbone network, a dynamic pyramid context module and a random attention fusion module;

[0007] The backbone network is used to extract features from the input infrared image;

[0008] The dynamic pyramid context module is used to adaptively capture the different scale information of the extracted feature map, enhance its context information and high-level semantic information; and to adaptively subdivide the semantic information to generate multi-scale context information;

[0009] The random attention fusion module is used to effectively fuse semantically rich deep features and spatially fine shallow features to obtain more accurate spatial and position information of small infrared targets.

[0010] In the above technical solution, the backbone network is the ResNet-18 network.

[0011] In the above technical solution, the dynamic pyramid context module is composed of four dynamic context filter modules; the dynamic context filter module adaptively obtains a specific scale representation related to the input feature map.

[0012] An infrared dim small target detection method using the above-mentioned dynamic context-aware pyramid network comprises the following steps:

[0013] Step 1) Given an infrared image as input, the backbone network is used to extract features of the input infrared image to obtain a feature map X;

[0014] Step 2) For the feature map X extracted by the backbone network, it is divided into two branches and processed simultaneously; the feature maps obtained by the two branches are fused through deep convolution operations to obtain a feature map S of a specific scale k ;

[0015] Step 3) Use 1×1 convolution operation to k Processing is performed to fuse the channel information of the two branch feature maps to obtain a specific scale feature map S output by the dynamic context filter module;

[0016] Step 4) Parallel execution of multiple dynamic context filter modules with different k-value kernel filters, after 1×1 convolution processing, the dimensionality reduction result is obtained

[0017] Step 5) Connect the generated feature map S with the feature map extracted by the backbone network, process the aggregated channel information through 1×1 convolution, and obtain the output result M of the dynamic pyramid context module;

[0018] Step 6) The semantically rich deep features and spatially fine shallow features are effectively fused through the random attention fusion module to achieve the exchange of channel and spatial information in the same layer features.

[0019] In the above technical solution, step 2) specifically includes:

[0020] Step 2-1) Perform 1×1 convolution on the input feature map to reduce the number of channels and simplify the feature map to obtain the simplified feature map F k (X′);

[0021] In the above technical solution, step 2) specifically includes:

[0022] Step 2-2) Generate a filter G with a kernel size of k by performing adaptive average pooling and 1×1 convolution operations on the input feature map in sequence. k (X′).

[0023] In the above technical solution, step 6) specifically includes:

[0024] First, the high-level and low-level features of the input are processed by random attention blocks respectively to obtain X d ′ and X l ′;

[0025] Then, the top-down global channel modulation module and the bottom-up local point attention module are executed in parallel to obtain the feature map X d ″ and X l ″:

[0026] X d =L(X d ′)=L(SA(X d ))

[0027] X l =G(X l ′)=G(SA(X l ))

[0028] Among them, L(·), SA(·) and G(·) represent global channel modulation, random attention modulation and local point attention modulation respectively;

[0029] Then, X d ′ and X l ′ is added and restored to the size of C×H×W, and then combined with the processed feature map X d ″ and X l ″Perform pixel-by-pixel dot product to obtain the final output fusion feature

[0030]

[0031] in, Represents element-wise multiplication.

[0032] The present invention has the following beneficial effects:

[0033] The present invention discloses a dynamic context-aware pyramid network for infrared dim small target detection. The network uses U-Net as the host network and ResNet-18 as the backbone network to achieve end-to-end detection of infrared dim small targets. A dynamic pyramid context module is constructed after the deepest feature layer, which can dynamically obtain multi-scale context information using context-aware filters with different convolution kernel sizes, thereby reducing the loss of detail information between different sub-regions. Then, combining the characteristics of random attention modulation and bidirectional asymmetric fusion, a random attention fusion module is proposed to enhance the representation ability of target features and suppress conflicting information and noise, thereby solving the problems of small target detail loss, inter-layer information redundancy, and insufficient feature fusion.

[0034] The infrared dim small target detection method of the present invention has better target detection capability, background suppression capability and stronger robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0036] Figure 1 It is a schematic diagram of the overall architecture of the dynamic context-aware pyramid network for infrared dim small target detection of the present invention;

[0037] In the figure: SAF-random attention fusion module; DPC-dynamic pyramid context module; red line-upsampling operation; blue line-downsampling operation.

[0038] Figure 2 It is a schematic diagram of a dynamic context filtering module of a dynamic context-aware pyramid network for infrared dim small target detection of the present invention;

[0039] Figure 3 It is a schematic diagram of the architecture of a dynamic pyramid context module of a dynamic context-aware pyramid network for infrared dim small target detection of the present invention;

[0040] Figure 4 A schematic diagram of random attention blocks of a dynamic context-aware pyramid network for infrared dim small target detection according to the present invention;

[0041] Figure 5 It is a schematic diagram of a random attention fusion module of a dynamic context-aware pyramid network for infrared dim small target detection of the present invention;

[0042] In the figure: SA - random attention block; GlobalAvgpool2D - global average pooling operation. DETAILED DESCRIPTION

[0043] The dynamic context-aware pyramid network for infrared dim small target detection and the infrared dim small target detection method of the present invention involve English and English abbreviations, and the Chinese meanings of the words are specifically as follows:

[0044] DPC: Dynamic Pyramid Context Module;

[0045] SAF: Stochastic Attention Fusion Module;

[0046] UP-sample: upsampling;

[0047] Conv-1: convolutional layer 1;

[0048] Stage-1: feature layer 1 after downsampling;

[0049] Stage-2: feature layer 2 after downsampling;

[0050] DCM: Dynamic Context Filter Module;

[0051] Fuse: fusion;

[0052] Split: split;

[0053] Aggregate: merge;

[0054] SA: random attention block;

[0055] Conv+BN: convolution and data batch normalization;

[0056] Sigmoid: S-shaped activation function;

[0057] RELU: rectified linear unit;

[0058] GlobalAvgpool2D: 2D global average pooling;

[0059] Fully Connected: fully connected;

[0060] Point-wise Conv: point-by-point convolution.

[0061] The inventive concept of the present invention is:

[0062] Since infrared targets have few pixels and small size, multi-scale representation is an effective method for extracting features of small infrared targets. Dilated convolution is used to expand the receptive field to construct multi-scale representation, but this convolution operation introduces a large amount of computation and easily causes information loss in the local neighborhood. To solve this problem, the present invention proposes a dynamic context-aware pyramid network.

[0063] The dynamic context-aware pyramid network consists of a backbone network, a dynamic pyramid context module, and a random attention fusion module. Given an infrared image as input, the dynamic context-aware pyramid first uses the ResNet-18 network as the backbone network to extract features from the input infrared image and obtain the feature map X. Then, the dynamic context filter module adaptively captures the different scale information of the extracted feature map to enhance its context information and high-level semantic information. Subsequently, the dynamic pyramid context module is used to adaptively subdivide the semantic information to generate multi-scale context information. Further, the random attention fusion module is used to effectively fuse the semantically rich deep features and the spatially fine shallow features to obtain more accurate spatial and position information of the infrared small target. The final output binary image is the infrared small target detection result.

[0064] The present invention uses a dynamic context filter module to adaptively enhance the context information of multi-level features through the proposed dynamic pyramid context module, and uses a designed random attention fusion module to suppress redundant information and conflicting information and fuse fine detail information and high-level semantic information. The dynamic context-aware pyramid network proposed in the present invention can adapt to various complex backgrounds, reduce the missed detection rate and false alarm rate in actual detection, and has better target detection ability, background suppression ability and stronger robustness.

[0065] The present invention is described in detail below with reference to the accompanying drawings.

[0066] like Figure 1 As shown in FIG. 1 , the dynamic context-aware pyramid network proposed in the present invention includes three parts: a backbone network, a dynamic pyramid context module, and a random attention fusion module. The dynamic pyramid context module is composed of four dynamic context filter modules. The dynamic context filter module adaptively obtains a specific scale representation related to the input feature map, such as Figure 2 As shown in the figure. The red dashed box is called the context-aware filter. It is composed of the input feature map The context information is adaptively generated, where W, H, and C represent the width, height, and number of channels of the feature map, respectively.

[0067] Compared with traditional filters, these filters contain rich contextual information and high-level semantics, and can better adapt to the input feature map, capture different scale information of the input image, and are more adaptive and flexible. Therefore, they are applied to the representation of different feature scales for multi-scale learning.

[0068] The infrared dim small target detection method of the dynamic context-aware pyramid network of the present invention comprises the following steps:

[0069] Step 1) Given an infrared image as input, the backbone network is used to extract features of the input infrared image to obtain a feature map X;

[0070] Step 2) For the feature map X extracted by the backbone network, it is divided into two branches and processed simultaneously; the feature maps obtained by the two branches are fused through deep convolution operations to obtain a feature map S of a specific scale k ;

[0071] Step 3) Use 1×1 convolution operation to k Processing is performed to fuse the channel information of the two branch feature maps to obtain a specific scale feature map S output by the dynamic context filter module;

[0072] Step 4) Parallel execution of multiple dynamic context filter modules with different k-value kernel filters, after 1×1 convolution processing, the dimensionality reduction result is obtained

[0073] Step 5) Connect the generated feature map S with the feature map extracted by the backbone network, process the aggregated channel information through 1×1 convolution, and obtain the output result M of the dynamic pyramid context module;

[0074] Step 6) The semantically rich deep features and spatially fine shallow features are effectively fused through the random attention fusion module to achieve the exchange of channel and spatial information in the same layer features.

[0075] Wherein, step 2) comprises:

[0076] Step 2-1) Perform 1×1 convolution on the input feature map to reduce the number of channels and simplify the feature map to obtain the simplified feature map F k (X′);

[0077] Step 2-2) Generate a filter G with a kernel size of k by performing adaptive average pooling and 1×1 convolution operations on the input feature map in sequence. k (X′).

[0078] Among them, the backbone network is used to implement step 1), the dynamic pyramid context module is used to implement steps 2)-5), and the random attention fusion module is used to implement step 6).

[0079] The functions and effects of the dynamic pyramid context module are described in detail below.

[0080] First, corresponding to step 2), for the feature map extracted by the backbone network It is divided into two branches for processing: branch 1 and branch 2. Branch 1 corresponds to step 2-1), and branch 2 corresponds to step 2-2).

[0081] Specifically:

[0082] Branch 1: Perform 1×1 convolution on the input feature map to reduce the number of channels and simplify the feature map to obtain a simplified feature map. Among them, C′ (C′≤C) represents the number of channels of the simplified feature map, and k represents the convolution kernel size of the context-aware filter.

[0083] Branch 2: Generate a filter with a kernel size of k by performing adaptive average pooling and 1×1 convolution operations on the input feature map in sequence

[0084] Then, corresponding to step 3), a deep convolution operation is performed to fuse the feature maps of the upper and lower branches to obtain a feature map of a specific scale:

[0085]

[0086] Among them, the symbol Represents depthwise convolution.

[0087] Finally, for S k Perform a 1×1 convolution operation to fuse the channel information of the upper and lower branch feature maps to obtain a specific scale feature map output by the dynamic context filter module Therefore, a dynamic context filtering module that applies multiple context-aware filters with different kernel sizes generates multi-scale feature representations.

[0088] The goal of the dynamic pyramid context module is to aggregate the multi-scale feature representations generated by multiple dynamic context filter modules. Its architecture is as follows Figure 3 As shown. First, the input image X is feature extracted through the backbone network convolution operation to obtain the feature map X′. Then, corresponding to step 4), a dynamic context filter module with multiple kernel filters with different k values ​​is executed in parallel, and after 1×1 convolution processing, the dimensionality reduction result is obtained. Where K is the kernel size of the context-aware filter.

[0089] Then, corresponding to step 5), the feature maps [S i ] is connected with the feature map X′ extracted by the backbone network. Finally, the aggregated channel information is processed by 1×1 convolution to obtain the output result M of the dynamic pyramid context module. Therefore, the dynamic context filter modules of different scales form a context pyramid module, which can dynamically obtain multi-scale context information.

[0090] Deep neural networks can better extract high-level semantic information of small targets. However, small targets in infrared images only occupy a few pixels and lack intrinsic information. The targets may be lost in deep networks. Therefore, how to prevent the reduction or even loss of fine detail information of targets in deep networks is an important issue. At the same time, features of different scales have semantic differences that cannot be ignored. Directly fusing features of different scales will bring a lot of redundant and conflicting information, reducing the ability of multi-scale expression. In order to solve these problems, the present invention proposes a random attention fusion module, such as Figure 5 As shown in Figure 1, the module consists of a random attention block and a bidirectional asymmetric fusion block. The random attention block effectively combines the spatial attention mechanism and the channel attention mechanism to prevent tiny target features from being submerged in conflicting information while highlighting the most obvious feature areas of small infrared targets. The bidirectional asymmetric fusion block contains a bottom-up local attention modulation branch and a top-down global attention modulation branch to fuse low-level spatial detail information and high-level semantic information.

[0091] Assume that the input feature map is Where w, h, c represent the height, width and number of channels of the feature map respectively. The random attention block divides the input x into g groups in the channel dimension: x = [x1, x2, ···, x g ], For each set of sub-features x k , different importance coefficients are generated by the spatial and channel attention modules, so that it can gradually learn specific semantic features during the training process. Feature x k Will be split into two branches along the channel dimension: like Figure 4 The upper branch is used to learn channel attention features: In order to be as lightweight as possible, the sub-features x are firstly pooled using global average pooling k1 Embed global information to get channel statistics Then execute the scale transformation function in sequence And sigmoid function. Finally, with each group of sub-features x k1 Perform element-by-element multiplication to generate the output of the channel branch The calculation process for generating channel attention features is as follows:

[0092]

[0093] x′ k1 =σ(F c (l))·x k1 =σ(W1l+b1)·x k1 (3)

[0094] Among them, σ(·) represents the sigmoid function, and are the parameters for scaling and moving respectively.

[0095] The lower branch is used to learn the spatial attention feature: the spatial attention focuses more on the important factor "where", which complements the channel attention that focuses more on "what". In the implementation process, the group norm is first used to transform the input sub-feature x k2 Then, through the scale transformation function To enhance x k2 Finally, multiply the split sub-feature x element by element. k2 Get the output of the spatial branch The process can be described as follows:

[0096] x′ k2 =σ(F c (GN(x k2 )))·x k2 =σ(W2·GN(x k2 )+b2)·x k2 (4)

[0097] Among them, W2 and b2 are shapes Parameters.

[0098] Then, connect the upper and lower branches to generate the number of channels and input sub-features x k The same feature map x′ k , as shown below:

[0099]

[0100] Next, all the sub-features x′ are processed in parallel. k Aggregation is performed to obtain a feature representation of the same size as the input feature x. Finally, the cross-group information flow in the channel dimension is achieved through the "channel shuffle" operator.

[0101] The bidirectional asymmetric fusion block can ensure that the deep semantic information of small targets is extracted while retaining the fine detail information of small targets. It is divided into a bottom-up point local channel attention mechanism for processing low-level detail information and a top-down global channel attention mechanism for processing high-level semantic information. Assume that the low-level feature is X l and high-level features are X d , X d After convolution with X l The block has the same size. The calculation process of the point attention mechanism and the global channel attention of the block is shown in formula (6) and formula (7):

[0102]

[0103] Among them, s, and d represent the Sigmoid function, Batch Normalization function, and ReLU function respectively. PWConv represents point-wise convolution. d Channel statistics obtained by global average pooling: This block can fully integrate low-level features and high-level features, but the feature information within the layer is not fully utilized, and the detailed information of the target is easily submerged in the background noise.

[0104] In order to solve the problems of small target detail loss, inter-layer information redundancy, and insufficient feature fusion, the random attention block is introduced into the bidirectional asymmetric fusion block to perform random attention modulation on the low-level details and high-level semantic information of the small target, and realize the exchange of channel and spatial information in the same-layer features, thereby ensuring that the fine details of the small target are not submerged by background noise. Finally, the feature information of different layers is fused across layers through the local point attention mechanism and the global channel attention mechanism to ensure the fusion of deep semantic information and shallow spatial position information.

[0105] like Figure 5 As shown, after upsampling, the given high-level feature X d and low-level features X l Corresponding to step 6), first, the high-level and low-level features of the input are processed by random attention blocks respectively to obtain X d ′ and X l Then, the top-down global channel modulation module and the bottom-up local point attention module are executed in parallel to obtain the feature map X d ″ and X l ″, the calculation process is shown in formula (8) and formula (9) respectively:

[0106] X d =L(X d ′)=L(SA(X d )) (8)

[0107] X l =G(X l ′)=G(SA(X l )) (9)

[0108] Among them, L(·), SA(·) and G(·) represent global channel modulation, random attention modulation and local point attention modulation respectively. Then, X d ′ and X l ′ is added and restored to the size of C×H×W, and then combined with the processed feature map X d ″ and X l ″Perform pixel-by-pixel dot product to obtain the final output fusion feature

[0109]

[0110] in, Represents element-wise multiplication.

[0111] At the same time, different feature layers are upsampled to the same size, and the features of different network layers are fused together through the connection operation to generate a robust feature map, which can improve the feature representation of small targets. The final output binary image is the infrared small target detection result.

[0112] The present invention discloses a dynamic context-aware pyramid network for infrared dim small target detection. The network uses U-Net as the host network and ResNet-18 as the backbone network to achieve end-to-end detection of infrared dim small targets. A dynamic pyramid context module is constructed after the deepest feature layer, which can dynamically obtain multi-scale context information using context-aware filters with different convolution kernel sizes, thereby reducing the loss of detail information between different sub-regions. Then, combining the characteristics of random attention modulation and bidirectional asymmetric fusion, a random attention fusion module is proposed to enhance the representation ability of target features and suppress conflicting information and noise, thereby solving the problems of small target detail loss, inter-layer information redundancy, and insufficient feature fusion.

[0113] The infrared dim small target detection method of the present invention has better target detection capability, background suppression capability and stronger robustness.

[0114] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A dynamic context-aware pyramid network for infrared small target detection, characterized in that: include: Backbone network, dynamic pyramid context module and random attention fusion module; The backbone network is used to extract features from the input infrared image; The dynamic pyramid context module is used to adaptively capture the different scale information of the extracted feature map, enhance its context information and high-level semantic information; and to adaptively subdivide the semantic information to generate multi-scale context information; The random attention fusion module is used to effectively fuse semantically rich deep features and spatially fine shallow features to obtain more accurate spatial and position information of small infrared targets.

2. The dynamic context-aware pyramid network for infrared small target detection according to claim 1, characterized in that: The backbone network is the ResNet-18 network.

3. The dynamic context-aware pyramid network for infrared small target detection according to claim 1, characterized in that: The dynamic pyramid context module consists of four dynamic context filter modules; the dynamic context filter module adaptively obtains a specific scale representation related to the input feature map.

4. An infrared dim small target detection method using the dynamic context-aware pyramid network of claim 1, characterized in that: The following steps are involved: Step 1) Given an infrared image as input, the backbone network is used to extract features of the input infrared image to obtain a feature map X; Step 2) For the feature map X extracted by the backbone network, it is divided into two branches and processed simultaneously; the feature maps obtained by the two branches are fused through deep convolution operations to obtain a feature map S of a specific scale k ; Step 3) Use 1×1 convolution operation to k Processing is performed to fuse the channel information of the two branch feature maps to obtain a specific scale feature map S output by the dynamic context filter module; Step 4) Parallel execution of multiple dynamic context filter modules with different k-value kernel filters, after 1×1 convolution processing, the dimensionality reduction result is obtained Step 5) Connect the generated feature map S with the feature map extracted by the backbone network, process the aggregated channel information through 1×1 convolution, and obtain the output result M of the dynamic pyramid context module; Step 6) The semantically rich deep features and spatially fine shallow features are effectively fused through the random attention fusion module to achieve the exchange of channel and spatial information in the same layer features.

5. The infrared small target detection method according to claim 4, characterized in that: Step 2) specifically includes: Step 2-1) Perform 1×1 convolution on the input feature map to reduce the number of channels and simplify the feature map to obtain the simplified feature map F k (X′).

6. The infrared small target detection method according to claim 4, characterized in that: Step 2) specifically includes: Step 2-2) Generate a filter G with a kernel size of k by performing adaptive average pooling and 1×1 convolution operations on the input feature map in sequence. k (X′).

7. The infrared small target detection method according to claim 4, characterized in that: Step 6) specifically includes: First, the high-level and low-level features of the input are processed by random attention blocks respectively to obtain X d ′ and X l ′; Then, the top-down global channel modulation module and the bottom-up local point attention module are executed in parallel to obtain the feature map X d ″ and X l ″: X d ″=L(X d ′)=L(SA(X d )) X l ″=G(X l ′)=G(SA(X l )) Among them, L(·), SA() and G() represent global channel modulation, random attention modulation and local point attention modulation respectively; Then, X d ′ and X l ′ is added and restored to the size of C×H×W, and then combined with the processed feature map X d ″ and X l ″Perform pixel-by-pixel dot product to obtain the final output fusion feature in, Represents element-wise multiplication.

Images