Infrared target detection method based on multi-attention shape adaptation network

Abstract

The invention discloses an infrared target detection method based on a multi-attention shape adaptive network, comprising: obtaining an infrared image to be detected; inputting the infrared image to be detected into a pre-trained multi-attention shape adaptive network, and obtaining a binary segmentation map corresponding to the infrared image to be detected output by the network; the multi-attention shape adaptive network comprises: a densely nested U-shaped feature extraction network and an output layer based on spatial feature fusion, a bidirectional shape adaptive extraction module and a multi-channel shape enhancement module connected in sequence are added to the lateral connection of the densely nested U-shaped feature extraction network, the densely nested U-shaped feature extraction network is used to spatially fuse visual features of different semantic levels, the extraction module is used to adaptively extract the shape information of the infrared target, and the enhancement module is used to enhance the shape information extracted by the extraction module. The invention can fuse visual features of different semantic levels, reduce the semantic gap in the feature fusion stage, and improve the accuracy of infrared target detection.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an infrared target detection method based on a multi-attention shape adaptive network. Background Art

[0002] Since infrared cameras can provide temperature information in addition to visible light information, they can be used to capture infrared images for target detection at night when there is insufficient light, the ambient light is overexposed, the target is blocked by trees, or the contrast between the target and the environment is low. For this reason, infrared cameras play an irreplaceable role in the field of target detection.

[0003] Infrared target detection for infrared images is a key issue in the fields of reconnaissance, remote sensing, early warning, emergency rescue, etc. Traditional infrared target detection methods mainly include image processing methods and methods based on artificially designed features. Image processing methods can be further divided into morphology-based methods, threshold segmentation-based methods, and frequency domain-based methods. However, when image processing methods are used for infrared target detection in infrared images, it is often difficult to determine the optimal threshold for image segmentation. When the background conditions are complex, the segmentation threshold needs to be manually adjusted to find the infrared target area. Methods based on artificially designed features usually include two stages: feature extraction and pattern classification. The features include local binary features (LBP), histogram of oriented gradients (HOG), and some other grayscale statistical features. However, since artificially designed features are sensitive to changes in target shape and different background conditions, when these methods are applied to different scenes, they often need to manually adjust parameters or even redesign the algorithm, which lacks adaptability and robustness.

[0004] In recent years, with the continuous development of deep learning technology, convolutional neural networks have also been used for infrared target detection. At present, many infrared target detection methods based on convolutional neural networks have been proposed to obtain accurate infrared target shape and positioning information.

[0005] In order to improve the detection accuracy, the existing infrared target detection methods based on convolutional neural networks have designed different feature extraction structures, feature enhancement structures and feature fusion structures. In terms of feature extraction, the feature pyramid structure is usually used. The feature pyramid structure is used to extract features from different semantic levels. The main types include: top-down feature pyramid structure, encoder-decoder feature pyramid structure and spatial feature pyramid structure. The top-down feature pyramid structure fuses image features of different semantic levels from top to bottom, and restores the image to its original resolution through upsampling operation to obtain the segmentation result. The feature pyramid structure of the encoder-decoder has two symmetrical branches. The feature maps of the downsampling module and the corresponding upsampling module are fused through the skip connection structure, and multi-level semantic features are introduced to obtain the segmentation result. The spatial feature pyramid structure usually contains multiple pooling branches, which are used to generate multi-scale features and fuse them to obtain the segmentation result. In terms of feature fusion, in order to retain more representative features, long skip connection feature fusion structure and dense skip connection feature fusion structure are designed. The features of infrared images and visible light images are aggregated through the long skip connection feature fusion structure to complete multi-modal feature fusion and improve the accuracy of infrared target detection. The features extracted by encoders, decoders and multimodal fusion modules at each level are fused through the short jump feature fusion structure to improve the accuracy of infrared target detection. In terms of feature enhancement, attention modules and different types of convolution kernels are usually used. The attention module learns to assign different weights to different features through network training to enhance effective information, thereby improving the efficiency of feature extraction. Convolution kernels with special functions can also enhance different types of features. For example, deformable convolution is applied to infrared target tracking, and a learnable offset is added to the square convolution sampling area to improve the network's ability to extract features of targets of different sizes and shapes; for example, dilated convolution is used to expand the network's receptive field to enhance the image's spatial features.

[0006] However, due to the diverse size and morphology of infrared targets, the existing feature pyramid structure is designed to aggregate features at different semantic levels, and does not specifically consider how to reduce the semantic gap between features, resulting in insufficient shape fitting of infrared targets and inaccurate detail segmentation. At the same time, in the downsampling stage of the feature pyramid module, the features of tiny infrared targets are easily lost, resulting in missed detection. In addition, the existing attention module and convolution kernel structure are more suitable for enhancing the features of natural objects. When applied to infrared target feature enhancement, they need to be redesigned based on the characteristics of weak texture and large deformation of infrared targets. Summary of the invention

[0007] In order to solve some or all of the technical problems existing in the above-mentioned prior art, the present invention provides an infrared target detection method based on a multi-attention shape adaptive network.

[0008] The technical solution of the present invention is as follows:

[0009] A method for infrared target detection based on a multi-attention shape adaptation network is provided, the method comprising:

[0010] Acquire the infrared image to be detected;

[0011] Inputting the infrared image to be detected into a pre-trained multi-attention shape adaptation network to obtain a binary segmentation map corresponding to the infrared image to be detected output by the multi-attention shape adaptation network;

[0012] The multi-attention shape adaptation network includes: a densely nested U-shaped feature extraction network based on spatial feature fusion and an output layer, a bidirectional shape adaptation extraction module and a multi-channel shape enhancement module connected in sequence are added to the lateral connection of the densely nested U-shaped feature extraction network based on spatial feature fusion, the input of the densely nested U-shaped feature extraction network based on spatial feature fusion is used to receive infrared images, and the output is connected to the output layer, the densely nested U-shaped feature extraction network based on spatial feature fusion is used to spatially fuse visual features of different semantic levels to obtain highly representative infrared target features, the bidirectional shape adaptation extraction module is used to adaptively extract shape information of infrared targets, the multi-channel shape enhancement module is used to enhance the shape information extracted by the bidirectional shape adaptation extraction module, and the output layer is used to receive and process the output of the densely nested U-shaped feature extraction network based on spatial feature fusion to obtain and output a binary segmentation map.

[0013] In some optional embodiments, the densely nested U-shaped feature extraction network based on spatial feature fusion includes: an encoder branch and a decoder branch, the encoder branch includes one, the decoder branch includes four symmetrical branches, the encoder branch is shared by four symmetrical decoder branches, the encoder branch and the decoder branch are also connected by dense jump connections, and a bidirectional shape adaptive extraction module and a multi-channel shape enhancement module connected in sequence are added to the lateral connection of each decoder branch, and the output of each decoder branch is respectively connected to the output layer.

[0014] In some optional embodiments, the encoder branch includes five feature layers sequentially connected from top to bottom through the downsampling layer, and the decoder branch includes a first decoder branch, a second decoder branch, a third decoder branch and a fourth decoder branch;

[0015] The first decoder branch includes a first feature fusion layer, a first bidirectional shape adaptive extraction module and a first multi-channel shape enhancement module, the second feature layer of the encoder branch is connected to the first feature fusion layer through an upsampling layer, the first feature layer of the encoder branch is connected to the first feature fusion layer, the output of the first feature fusion layer is horizontally connected to the first bidirectional shape adaptive extraction module, the first feature layer of the encoder branch is jump-connected to the first bidirectional shape adaptive extraction module, the first bidirectional shape adaptive extraction module and the first multi-channel shape enhancement module are connected in sequence, the output of the first multi-channel shape enhancement module is connected to the output layer, and the first feature fusion layer is used to splice and fuse the input features;

[0016] The second decoder branch includes a second feature fusion layer, a third feature fusion layer, a second bidirectional shape adaptive extraction module, a second multi-channel shape enhancement module, a third bidirectional shape adaptive extraction module and a third multi-channel shape enhancement module. The third feature layer of the encoder branch is connected to the second feature fusion layer through an upsampling layer, the second feature layer of the encoder branch is connected to the second feature fusion layer, the first feature fusion layer is connected to the second feature fusion layer through a downsampling layer, the output of the second feature fusion layer is horizontally connected to the second bidirectional shape adaptive extraction module, the second feature layer of the encoder branch is also jump-connected to the second bidirectional shape adaptive extraction module, the second bidirectional shape adaptive extraction module and the second multi-channel shape enhancement module are connected. The output of the second multi-channel shape enhancement module is connected to the third feature fusion layer through the upsampling layer, the output of the first multi-channel shape enhancement module is also connected to the third feature fusion layer, the first feature layer of the encoder branch is also jump-connected to the third feature fusion layer, the output of the third feature fusion layer is horizontally connected to the third bidirectional shape adaptive extraction module, the first feature layer of the encoder branch is also jump-connected to the third bidirectional shape adaptive extraction module, the third bidirectional shape adaptive extraction module and the third multi-channel shape enhancement module are connected in sequence, the output of the third multi-channel shape enhancement module is connected to the output layer, and the second feature fusion layer and the third feature fusion layer are both used to splice and fuse the input features;

[0017] The third decoder branch includes a fourth feature fusion layer, a fifth feature fusion layer, a sixth feature fusion layer, a fourth bidirectional shape adaptive extraction module, a fourth multi-channel shape enhancement module, a fifth bidirectional shape adaptive extraction module, a fifth multi-channel shape enhancement module, a sixth bidirectional shape adaptive extraction module and a sixth multi-channel shape enhancement module. The fourth feature layer of the encoder branch is connected to the fourth feature fusion layer through an upsampling layer, the third feature layer of the encoder branch is connected to the fourth feature fusion layer, the second feature fusion layer is connected to the fourth feature fusion layer through a downsampling layer, the output of the fourth feature fusion layer is horizontally connected to the fourth bidirectional shape adaptive extraction module, the third feature layer of the encoder branch is also jump-connected to the fourth bidirectional shape adaptive extraction module, the fourth bidirectional shape adaptive extraction module and the fourth multi-channel shape enhancement module are connected in sequence, the output of the fourth multi-channel shape enhancement module is connected to the fifth feature fusion layer through an upsampling layer, the output of the second multi-channel shape enhancement module is also connected to the fifth feature fusion layer, the second feature layer of the encoder branch is also jump-connected to the fifth feature fusion layer, the third feature fusion layer The downsampling layer is connected to the fifth feature fusion layer, the output of the fifth feature fusion layer is horizontally connected to the fifth bidirectional shape adaptive extraction module, the second feature layer of the encoder branch is also jump-connected to the fifth bidirectional shape adaptive extraction module, the fifth bidirectional shape adaptive extraction module and the fifth multi-channel shape enhancement module are connected in sequence, the output of the fifth multi-channel shape enhancement module is connected to the sixth feature fusion layer through the upsampling layer, the output of the third multi-channel shape enhancement module is also connected to the sixth feature fusion layer, the first feature layer of the encoder branch is also jump-connected to the sixth feature fusion layer, the output of the first multi-channel shape enhancement module is also jump-connected to the sixth feature fusion layer, the output of the sixth feature fusion layer is horizontally connected to the sixth bidirectional shape adaptive extraction module, the first feature layer of the encoder branch is also jump-connected to the sixth bidirectional shape adaptive extraction module, the sixth bidirectional shape adaptive extraction module and the sixth multi-channel shape enhancement module are connected in sequence, the output of the sixth multi-channel shape enhancement module is connected to the output layer, the fourth feature fusion layer, the fifth feature fusion layer and the sixth feature fusion layer are all used to splice and fuse the input features;

[0018] The fourth decoder branch includes a seventh feature fusion layer, an eighth feature fusion layer, a ninth feature fusion layer, a tenth feature fusion layer, a seventh bidirectional shape adaptive extraction module, a seventh multi-channel shape enhancement module, an eighth bidirectional shape adaptive extraction module, an eighth multi-channel shape enhancement module, a ninth bidirectional shape adaptive extraction module, a ninth multi-channel shape enhancement module, a tenth bidirectional shape adaptive extraction module and a tenth multi-channel shape enhancement module, the fifth feature layer of the encoder branch is connected to the seventh feature fusion layer through an upsampling layer, the fourth feature layer of the encoder branch is connected to the seventh feature fusion layer, the fourth feature fusion layer is connected to the seventh feature fusion layer through a downsampling layer, and the output of the seventh feature fusion layer is laterally connected to the seventh bidirectional shape adaptive extraction module. The fourth feature layer of the encoder branch is also jump-connected to the seventh bidirectional shape adaptive extraction module, the seventh bidirectional shape adaptive extraction module and the seventh multi-channel shape enhancement module are connected in sequence, the output of the seventh multi-channel shape enhancement module is connected to the eighth feature fusion layer through the upsampling layer, the output of the fourth multi-channel shape enhancement module is also connected to the eighth feature fusion layer, the third feature layer of the encoder branch is also jump-connected to the eighth feature fusion layer, the fifth feature fusion layer is connected to the eighth feature fusion layer through the downsampling layer, the output of the eighth feature fusion layer is laterally connected to the eighth bidirectional shape adaptive extraction module, the third feature layer of the encoder branch is also jump-connected to the eighth bidirectional shape adaptive extraction module, the eighth bidirectional shape adaptive extraction module The output of the eighth multi-channel shape enhancement module is connected to the ninth feature fusion layer through the upsampling layer, the output of the fifth multi-channel shape enhancement module is also connected to the ninth feature fusion layer, the second feature layer of the encoder branch is also jump-connected to the ninth feature fusion layer, the output of the second multi-channel shape enhancement module is also jump-connected to the ninth feature fusion layer, the sixth feature fusion layer is connected to the ninth feature fusion layer through the downsampling layer, the output of the ninth feature fusion layer is laterally connected to the ninth bidirectional shape adaptive extraction module, the second feature layer of the encoder branch is also jump-connected to the ninth bidirectional shape adaptive extraction module, the ninth bidirectional shape adaptive extraction module and the ninth multi-channel shape enhancement module are connected in sequence, and the ninth multi-channel The output of the channel shape enhancement module is connected to the tenth feature fusion layer through the upsampling layer, the output of the sixth multi-channel shape enhancement module is also connected to the tenth feature fusion layer, the first feature layer of the encoder branch is also jump-connected to the tenth feature fusion layer, the output of the first multi-channel shape enhancement module is also jump-connected to the tenth feature fusion layer, the output of the third multi-channel shape enhancement module is also jump-connected to the tenth feature fusion layer, the output of the tenth feature fusion layer is laterally connected to the tenth bidirectional shape adaptive extraction module, the first feature layer of the encoder branch is also jump-connected to the tenth bidirectional shape adaptive extraction module, the tenth bidirectional shape adaptive extraction module and the tenth multi-channel shape enhancement module are connected in sequence, and the output of the tenth multi-channel shape enhancement module is connected to the output layer.The seventh feature fusion layer, the eighth feature fusion layer, the ninth feature fusion layer and the tenth feature fusion layer are all used to concatenate and fuse the input features.

[0019] In some optional embodiments, the bidirectional shape adaptive extraction module includes: a horizontal deformable convolution layer, a vertical deformable convolution layer, a first Sigmoid activation function layer, a second Sigmoid activation function layer, a first feature fusion unit, a second feature fusion unit, a first splicing unit, a second splicing unit and a third splicing unit;

[0020] The input end of the horizontal deformable convolution layer is used to connect with the feature layer of the encoder branch corresponding to the bidirectional shape adaptive extraction module, the output end of the horizontal deformable convolution layer is connected with the input end of the first Sigmoid activation function layer, the output end of the first Sigmoid activation function layer is connected with the input end of the first feature fusion unit, the input end of the first feature fusion unit is also used to connect with the feature fusion layer of the decoder branch corresponding to the bidirectional shape adaptive extraction module, the output end of the first feature fusion unit is connected with the input end of the first splicing unit, the input end of the first splicing unit is also used to connect with the feature layer of the encoder branch corresponding to the bidirectional shape adaptive extraction module, the output end of the first splicing unit is connected with the input end of the third splicing unit, the input end of the vertical deformable convolution layer Used to be connected to the feature layer of the encoder branch corresponding to the bidirectional shape adaptive extraction module, the output end of the longitudinal deformable convolution layer is connected to the input end of the second Sigmoid activation function layer, the output end of the second Sigmoid activation function layer is connected to the input end of the second feature fusion unit, the input end of the second feature fusion unit is also used to be connected to the feature fusion layer of the decoder branch corresponding to the bidirectional shape adaptive extraction module, the output end of the second feature fusion unit is connected to the input end of the second splicing unit, the input end of the second splicing unit is also used to be connected to the feature layer of the encoder branch corresponding to the bidirectional shape adaptive extraction module, the output end of the second splicing unit is connected to the input end of the third splicing unit, and the output of the third splicing unit is the output of the bidirectional shape adaptive extraction module;

[0021] The horizontal deformable convolution layer performs horizontal deformable convolution on the input features, the vertical deformable convolution layer performs vertical deformable convolution on the input features, the first Sigmoid activation function layer uses the Sigmoid activation function to perform nonlinear transformation on the input features, the second Sigmoid activation function layer uses the Sigmoid activation function to perform nonlinear transformation on the input features, the first feature fusion unit is used to multiply and fuse two input features, the second feature fusion unit is used to multiply and fuse two input features, the first splicing unit is used to splice two input features, the second splicing unit is used to splice two input features, and the third splicing unit is used to splice two input features.

[0022] In some optional embodiments, the multi-channel shape enhancement module includes: a channel attention module, a shape enhancement module, a third feature fusion unit and a fourth feature fusion unit;

[0023] The input end of the channel attention module is used to be connected to the output end of the bidirectional shape adaptive extraction module, the output end of the channel attention module is connected to the input end of the third feature fusion unit, the input end of the third feature fusion unit is also used to be connected to the output end of the bidirectional shape adaptive extraction module, the output end of the third feature fusion unit is connected to the input end of the shape enhancement module, the output end of the shape enhancement module is connected to the input end of the fourth feature fusion unit, the input end of the fourth feature fusion unit is also connected to the output end of the third feature fusion unit, and the output of the fourth feature fusion unit is the output of the multi-channel shape enhancement module;

[0024] The channel attention module is used to assign different weight values ​​to each channel of the target feature. The shape enhancement module further enhances the shape information of the target feature by constructing a shape-adaptive weight map. The third feature fusion unit is used to multiply and fuse two input features. The fourth feature fusion unit is used to multiply and fuse two input features.

[0025] In some optional implementations, the output of the channel attention module is expressed as:

[0026]

[0027] Att C =S(MLP(P max (O BSAA ))+MLP(P avg (O BSAA )));

[0028] Among them, O C Represents the output of the channel attention module, Att C represents a one-dimensional attention weight map, represents the element-by-element matrix multiplication operation, O BSAA represents the output of the bidirectional shape adaptive extraction module, S(·) represents the Sigmoid activation function, MLP(·) represents a multi-layer perceptron, which contains a hidden layer and is used to output weight values, P max (·) represents the maximum pooling operation, P avg (·) represents the average pooling operation.

[0029] In some optional embodiments, the output of the shape enhancement module is expressed as:

[0030]

[0031] Att S =S(ShapeFilter k×k [P max (O C ),P avg (O C )]);

[0032] Among them, O S represents the output of the shape enhancement module, Att S Represents a two-dimensional shape enhancement weight map, ShapeFilter k×k represents a k×k deformable convolution, and [·] represents a feature concatenation operation.

[0033] In some optional embodiments, the multi-attention shape adaptation network is pre-trained by a multi-scale multi-branch deep supervised training method.

[0034] In some optional implementations, the multi-scale multi-branch deep supervised training method includes the following steps:

[0035] Step 21, obtaining a training data set, the training data including an infrared image and a binary segmentation map of an infrared target corresponding to the infrared image;

[0036] Step 22, inputting the infrared images in all the training data into the multi-attention shape adaptive network in batches, obtaining a first predicted infrared target binary segmentation map corresponding to each infrared image output by the output layer of the multi-attention shape adaptive network, and performing upsampling operation on the outputs at different network depth nodes of the multi-attention shape adaptive network to scale them to the same size, and then performing splicing and convolution processing to obtain a second predicted infrared target binary segmentation map corresponding to each infrared image;

[0037] Step 23, calculating the target detection loss function according to the infrared target binary segmentation map in the training data, and the first predicted infrared target binary segmentation map and the second predicted infrared target binary segmentation map obtained by the multi-attention shape adaptation network;

[0038] Step 24, determine whether the preset training stop condition is reached. If so, use the current multi-attention shape adaptation network as the multi-attention shape adaptation network that has completed the training. If not, use the target detection loss function to update the parameters of the multi-attention shape adaptation network and return to step 21.

[0039] In some optional embodiments, the method further comprises:

[0040] When pre-training the multi-attention shape adaptation network, pruning is performed on the trained multi-attention shape adaptation network.

[0041] The main advantages of the technical solution of the present invention are as follows:

[0042] The infrared target detection method based on multi-attention shape adaptive network of the present invention adopts a densely nested U-shaped feature extraction network as a feature extraction backbone network, which can fuse visual features of different semantic levels, reduce the reduction of infrared target features in the downsampling process, and obtain highly representative features. At the same time, in the feature extraction and fusion process, a bidirectional shape adaptive extraction module is set to adaptively extract the shape information of the infrared target, which can reduce the shape difference between adjacent features and thus bridge the semantic gap between the features to be fused. By setting a multi-channel shape enhancement module to further enhance the infrared target shape information, the semantic gap in the feature fusion stage can be further reduced, thereby obtaining a more robust feature fusion result, improving the accuracy of infrared target segmentation, and significantly improving the detection accuracy of infrared targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the embodiments of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0044] Figure 1 A flowchart of an infrared target detection method based on a multi-attention shape adaptive network provided by an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of the structure of a multi-attention shape adaptation network provided by an embodiment of the present invention;

[0046] Figure 3 for Figure 2 The structural diagram of the densely nested U-shaped feature extraction network based on spatial feature fusion is shown in;

[0047] Figure 4 for Figure 2 The structural diagram of the bidirectional shape adaptive extraction module shown in;

[0048] Figure 5 for Figure 2 A schematic diagram of the structure of the multi-channel shape enhancement module shown in;

[0049] Figure 6 A schematic diagram corresponding to the multi-scale multi-branch deep supervision training method provided in an embodiment of the present invention;

[0050] Figure 7A schematic diagram of the structure of a multi-attention shape adaptation network after pruning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0052] The technical solution provided by the embodiments of the present invention is described in detail below with reference to the accompanying drawings.

[0053] refer to Figure 1-2 The embodiment of the present invention provides an infrared target detection method based on a multi-attention shape adaptive network, the method comprising the following steps:

[0054] Step 1, obtaining an infrared image to be detected;

[0055] Step 2: Input the infrared image to be detected into the pre-trained multi-attention shape adaptation network to obtain a binary segmentation map corresponding to the infrared image to be detected output by the multi-attention shape adaptation network.

[0056] In an embodiment of the present invention, a multi-attention shape adaptation network includes: a densely nested U-shaped feature extraction network based on spatial feature fusion and an output layer, a bidirectional shape adaptation extraction module and a multi-channel shape enhancement module connected in sequence are added to the lateral connection of the densely nested U-shaped feature extraction network based on spatial feature fusion, the input of the densely nested U-shaped feature extraction network based on spatial feature fusion is used to receive infrared images, and the output is connected to the output layer, the densely nested U-shaped feature extraction network based on spatial feature fusion is used to spatially fuse visual features of different semantic levels to obtain highly representative infrared target features, the bidirectional shape adaptation extraction module is used to adaptively extract shape information of infrared targets, the multi-channel shape enhancement module is used to enhance the shape information extracted by the bidirectional shape adaptation extraction module, and the output layer is used to receive and process the output of the densely nested U-shaped feature extraction network based on spatial feature fusion to obtain and output a binary segmentation map.

[0057] The infrared target detection method based on multi-attention shape adaptive network provided by the embodiment of the present invention adopts a densely nested U-shaped feature extraction network as the feature extraction backbone network, which can fuse visual features of different semantic levels, reduce the reduction of infrared target features in the downsampling process, and obtain highly representative features. At the same time, in the feature extraction and fusion process, a bidirectional shape adaptive extraction module is set to adaptively extract the shape information of the infrared target, which can reduce the shape difference between adjacent features and thus bridge the semantic gap between the features to be fused. By setting a multi-channel shape enhancement module to further enhance the shape information of the infrared target, the semantic gap in the feature fusion stage can be further reduced, thereby obtaining a more robust feature fusion result, improving the accuracy of infrared target segmentation, and significantly improving the detection accuracy of infrared targets.

[0058] refer to Figure 2-Figure 3 Furthermore, in an embodiment of the present invention, a densely nested U-shaped feature extraction network based on spatial feature fusion includes: an encoder branch and a decoder branch, the encoder branch includes one, the decoder branch includes four symmetrical branches, the encoder branch is shared by four symmetrical decoder branches, the encoder branch and the decoder branch are also connected by dense jump connections, and a bidirectional shape adaptive extraction module and a multi-channel shape enhancement module connected in sequence are added to the lateral connection of each decoder branch, and the output of each decoder branch is respectively connected to the output layer.

[0059] Based on the above structure, the densely nested U-shaped feature extraction network based on spatial feature fusion provided by the embodiment of the present invention performs shape enhancement on the extracted features through a bidirectional shape adaptive extraction module and a multi-channel shape enhancement module during feature extraction, which can bridge the semantic gap between features of different levels, and then fuse features of different semantic levels in a dense skip connection manner to obtain target features with higher representation.

[0060] Further, refer to Figure 2-Figure 3 In the densely nested U-shaped feature extraction network based on spatial feature fusion provided by an embodiment of the present invention, the encoder branch includes five feature layers sequentially connected from top to bottom through a downsampling layer;

[0061] The first decoder branch includes a first feature fusion layer, a first bidirectional shape adaptive extraction module and a first multi-channel shape enhancement module, the second feature layer of the encoder branch is connected to the first feature fusion layer through an upsampling layer, the first feature layer of the encoder branch is connected to the first feature fusion layer, the output of the first feature fusion layer is horizontally connected to the first bidirectional shape adaptive extraction module, the first feature layer of the encoder branch is jump-connected to the first bidirectional shape adaptive extraction module, the first bidirectional shape adaptive extraction module and the first multi-channel shape enhancement module are connected in sequence, the output of the first multi-channel shape enhancement module is connected to the output layer, and the first feature fusion layer is used to concatenate and fuse the input features;

[0062] The second decoder branch includes a second feature fusion layer, a third feature fusion layer, a second bidirectional shape adaptive extraction module, a second multi-channel shape enhancement module, a third bidirectional shape adaptive extraction module and a third multi-channel shape enhancement module, the third feature layer of the encoder branch is connected to the second feature fusion layer through an upsampling layer, the second feature layer of the encoder branch is connected to the second feature fusion layer, the first feature fusion layer is connected to the second feature fusion layer through a downsampling layer, the output of the second feature fusion layer is horizontally connected to the second bidirectional shape adaptive extraction module, the second feature layer of the encoder branch is also jump-connected to the second bidirectional shape adaptive extraction module, the second bidirectional shape adaptive extraction module and the second multi-channel shape enhancement module The enhancement modules are connected in sequence, the output of the second multi-channel shape enhancement module is connected to the third feature fusion layer through the upsampling layer, the output of the first multi-channel shape enhancement module is also connected to the third feature fusion layer, the first feature layer of the encoder branch is also jump-connected to the third feature fusion layer, the output of the third feature fusion layer is horizontally connected to the third bidirectional shape adaptive extraction module, the first feature layer of the encoder branch is also jump-connected to the third bidirectional shape adaptive extraction module, the third bidirectional shape adaptive extraction module and the third multi-channel shape enhancement module are connected in sequence, the output of the third multi-channel shape enhancement module is connected to the output layer, and the second feature fusion layer and the third feature fusion layer are both used to splice and fuse the input features;

[0063] The third decoder branch includes a fourth feature fusion layer, a fifth feature fusion layer, a sixth feature fusion layer, a fourth bidirectional shape adaptive extraction module, a fourth multi-channel shape enhancement module, a fifth bidirectional shape adaptive extraction module, a fifth multi-channel shape enhancement module, a sixth bidirectional shape adaptive extraction module and a sixth multi-channel shape enhancement module. The fourth feature layer of the encoder branch is connected to the fourth feature fusion layer through an upsampling layer, the third feature layer of the encoder branch is connected to the fourth feature fusion layer, the second feature fusion layer is connected to the fourth feature fusion layer through a downsampling layer, the output of the fourth feature fusion layer is horizontally connected to the fourth bidirectional shape adaptive extraction module, the third feature layer of the encoder branch is also jump-connected to the fourth bidirectional shape adaptive extraction module, the fourth bidirectional shape adaptive extraction module and the fourth multi-channel shape enhancement module are connected in sequence, the output of the fourth multi-channel shape enhancement module is connected to the fifth feature fusion layer through an upsampling layer, the output of the second multi-channel shape enhancement module is also connected to the fifth feature fusion layer, the second feature layer of the encoder branch is also jump-connected to the fifth feature fusion layer, and the third feature fusion layer is connected to the fourth bidirectional shape adaptive extraction module through a downsampling layer. The fourth feature fusion layer is connected to the fifth feature fusion layer through the downsampling layer, the output of the fifth feature fusion layer is horizontally connected to the fifth bidirectional shape adaptive extraction module, the second feature layer of the encoder branch is also jump-connected to the fifth bidirectional shape adaptive extraction module, the fifth bidirectional shape adaptive extraction module and the fifth multi-channel shape enhancement module are connected in sequence, the output of the fifth multi-channel shape enhancement module is connected to the sixth feature fusion layer through the upsampling layer, the output of the third multi-channel shape enhancement module is also connected to the sixth feature fusion layer, the first feature layer of the encoder branch is also jump-connected to the sixth feature fusion layer, the output of the first multi-channel shape enhancement module is also jump-connected to the sixth feature fusion layer, the output of the sixth feature fusion layer is horizontally connected to the sixth bidirectional shape adaptive extraction module, the first feature layer of the encoder branch is also jump-connected to the sixth bidirectional shape adaptive extraction module, the sixth bidirectional shape adaptive extraction module and the sixth multi-channel shape enhancement module are connected in sequence, the output of the sixth multi-channel shape enhancement module is connected to the output layer, the fourth feature fusion layer, the fifth feature fusion layer and the sixth feature fusion layer are all used to splice and fuse the input features;

[0064] The fourth decoder branch includes a seventh feature fusion layer, an eighth feature fusion layer, a ninth feature fusion layer, a tenth feature fusion layer, a seventh bidirectional shape adaptive extraction module, a seventh multi-channel shape enhancement module, an eighth bidirectional shape adaptive extraction module, an eighth multi-channel shape enhancement module, a ninth bidirectional shape adaptive extraction module, a ninth multi-channel shape enhancement module, a tenth bidirectional shape adaptive extraction module and a tenth multi-channel shape enhancement module, the fifth feature layer of the encoder branch is connected to the seventh feature fusion layer through an upsampling layer, the fourth feature fusion layer is connected to the seventh feature fusion layer through a downsampling layer, and the output of the seventh feature fusion layer is laterally connected to the seventh bidirectional shape adaptive extraction module. The fourth feature layer of the encoder branch is also jump-connected to the seventh bidirectional shape adaptive extraction module, the seventh bidirectional shape adaptive extraction module and the seventh multi-channel shape enhancement module are connected in sequence, the output of the seventh multi-channel shape enhancement module is connected to the eighth feature fusion layer through the upsampling layer, the output of the fourth multi-channel shape enhancement module is also connected to the eighth feature fusion layer, the third feature layer of the encoder branch is also jump-connected to the eighth feature fusion layer, the fifth feature fusion layer is connected to the eighth feature fusion layer through the downsampling layer, the output of the eighth feature fusion layer is laterally connected to the eighth bidirectional shape adaptive extraction module, the third feature layer of the encoder branch is also jump-connected to the eighth bidirectional shape adaptive extraction module, the eighth bidirectional shape adaptive extraction module and The eighth multi-channel shape enhancement module is connected in sequence, the output of the eighth multi-channel shape enhancement module is connected to the ninth feature fusion layer through the upsampling layer, the output of the fifth multi-channel shape enhancement module is also connected to the ninth feature fusion layer, the second feature layer of the encoder branch is also jump-connected to the ninth feature fusion layer, the output of the second multi-channel shape enhancement module is also jump-connected to the ninth feature fusion layer, the sixth feature fusion layer is connected to the ninth feature fusion layer through the downsampling layer, the output of the ninth feature fusion layer is laterally connected to the ninth bidirectional shape adaptive extraction module, the second feature layer of the encoder branch is also jump-connected to the ninth bidirectional shape adaptive extraction module, the ninth bidirectional shape adaptive extraction module and the ninth multi-channel shape enhancement module are connected in sequence, and the ninth multi-channel The output of the channel shape enhancement module is connected to the tenth feature fusion layer through the upsampling layer, the output of the sixth multi-channel shape enhancement module is also connected to the tenth feature fusion layer, the first feature layer of the encoder branch is also jump-connected to the tenth feature fusion layer, the output of the first multi-channel shape enhancement module is also jump-connected to the tenth feature fusion layer, the output of the third multi-channel shape enhancement module is also jump-connected to the tenth feature fusion layer, the output of the tenth feature fusion layer is laterally connected to the tenth bidirectional shape adaptive extraction module, the first feature layer of the encoder branch is also jump-connected to the tenth bidirectional shape adaptive extraction module, the tenth bidirectional shape adaptive extraction module and the tenth multi-channel shape enhancement module are connected in sequence, and the output of the tenth multi-channel shape enhancement module is connected to the output layer.The seventh feature fusion layer, the eighth feature fusion layer, the ninth feature fusion layer and the tenth feature fusion layer are all used to concatenate and fuse the input features.

[0065] Based on the above structure, the densely nested U-shaped feature extraction network based on spatial feature fusion provided by the embodiment of the present invention can reduce the semantic gap in the feature fusion stage when performing feature extraction, fully integrate features of different semantic levels, and reduce the reduction of infrared target features in the downsampling process, so that the network feature extraction capability is optimized, and highly representative features can be obtained, thereby improving the detection accuracy of infrared targets.

[0066] It should be noted that in the attached Figure 2-3 In the figure, f1n1 represents the first feature layer, f2n1 represents the second feature layer, f3n1 represents the third feature layer, f4n1 represents the fourth feature layer, f5n1 represents the fifth feature layer, f1n2 represents the first feature fusion layer, f2n2 represents the second feature fusion layer, f1n3 represents the third feature fusion layer, f3n2 represents the fourth feature fusion layer, f2n3 represents the fifth feature fusion layer, f1n4 represents the sixth feature fusion layer, f4n2 represents the seventh feature fusion layer, f3n3 represents the eighth feature fusion layer, f2n4 represents the ninth feature fusion layer, f1n5 represents the tenth feature fusion layer, BSAA represents the bidirectional shape adaptive extraction module, and CSE represents the multi-channel shape enhancement module.

[0067] refer to Figure 2 and Figure 4 , Further, in an embodiment of the present invention, the bidirectional shape adaptive extraction module includes: a horizontal deformable convolution layer, a vertical deformable convolution layer, a first Sigmoid activation function layer, a second Sigmoid activation function layer, a first feature fusion unit, a second feature fusion unit, a first splicing unit, a second splicing unit and a third splicing unit;

[0068] The input end of the horizontal deformable convolution layer is used to connect with the feature layer of the encoder branch corresponding to the bidirectional shape adaptive extraction module, the output end of the horizontal deformable convolution layer is connected with the input end of the first Sigmoid activation function layer, the output end of the first Sigmoid activation function layer is connected with the input end of the first feature fusion unit, the input end of the first feature fusion unit is also used to connect with the feature fusion layer of the decoder branch corresponding to the bidirectional shape adaptive extraction module, the output end of the first feature fusion unit is connected with the input end of the first splicing unit, the input end of the first splicing unit is also used to connect with the feature layer of the encoder branch corresponding to the bidirectional shape adaptive extraction module, the output end of the first splicing unit is connected with the input end of the third splicing unit, the input end of the vertical deformable convolution layer Used to be connected to the feature layer of the encoder branch corresponding to the bidirectional shape adaptive extraction module, the output end of the longitudinal deformable convolution layer is connected to the input end of the second Sigmoid activation function layer, the output end of the second Sigmoid activation function layer is connected to the input end of the second feature fusion unit, the input end of the second feature fusion unit is also used to be connected to the feature fusion layer of the decoder branch corresponding to the bidirectional shape adaptive extraction module, the output end of the second feature fusion unit is connected to the input end of the second splicing unit, the input end of the second splicing unit is also used to be connected to the feature layer of the encoder branch corresponding to the bidirectional shape adaptive extraction module, the output end of the second splicing unit is connected to the input end of the third splicing unit, and the output of the third splicing unit is the output of the bidirectional shape adaptive extraction module;

[0069] The horizontal deformable convolution layer performs horizontal deformable convolution on the input features, the vertical deformable convolution layer performs vertical deformable convolution on the input features, the first Sigmoid activation function layer uses the Sigmoid activation function to perform nonlinear transformation on the input features, the second Sigmoid activation function layer uses the Sigmoid activation function to perform nonlinear transformation on the input features, the first feature fusion unit is used to multiply and fuse two input features, the second feature fusion unit is used to multiply and fuse two input features, the first splicing unit is used to splice two input features, the second splicing unit is used to splice two input features, and the third splicing unit is used to splice two input features.

[0070] Based on the above structure, when performing feature extraction, the bidirectional shape adaptive extraction module provided by the embodiment of the present invention first uses horizontal deformable convolution and vertical deformable convolution in the horizontal and vertical directions respectively to extract infrared target shape features at a lower semantic level, the infrared target shape features at the lower semantic level are the features transmitted by the feature layer jump of the encoder branch connected to the bidirectional shape adaptive extraction module, and then the two extracted shape features are fused with the spatial mixed features respectively to obtain horizontal shape features and vertical shape features, the spatial mixed features are the features transmitted by the feature fusion layer of the decoder branch connected to the bidirectional shape adaptive extraction module, and finally the horizontal shape features and the vertical shape features are spliced ​​to obtain and output the target features with enhanced shape information.

[0071] Since infrared targets have large shape variations and weak texture information, inaccurate detail segmentation is prone to occur. In an embodiment of the present invention, by utilizing the above-mentioned bidirectional shape adaptive extraction module, the shape information of the infrared target can be adaptively extracted, the shape difference between adjacent features can be reduced, and the semantic gap between the features to be fused can be compensated, thereby improving the accuracy of infrared target segmentation.

[0072] In the embodiment of the present invention, the horizontal deformable convolution layer performs 1×k horizontal deformable convolution on the input features, and the vertical deformable convolution layer performs k×1 vertical deformable convolution on the input features, where k represents the horizontal and vertical sizes of the convolution kernel.

[0073] It should be noted that in the attached Figure 4 In , Sigmoid represents the Sigmoid activation function layer. represents the feature fusion unit, Represents a splicing unit.

[0074] Furthermore, in an embodiment of the present invention, the multi-channel shape enhancement module is an attention mechanism for further enhancing the shape information of the features extracted by the bidirectional shape adaptive extraction module.

[0075] refer to Figure 2 and Figure 5 ,In an embodiment of the present invention, the multi-channel shape enhancement module includes: a channel attention module, a shape enhancement module, a third feature fusion unit and a fourth feature fusion unit;

[0076] The input end of the channel attention module is used to be connected to the output end of the bidirectional shape adaptive extraction module, the output end of the channel attention module is connected to the input end of the third feature fusion unit, the input end of the third feature fusion unit is also used to be connected to the output end of the bidirectional shape adaptive extraction module, the output end of the third feature fusion unit is connected to the input end of the shape enhancement module, the output end of the shape enhancement module is connected to the input end of the fourth feature fusion unit, the input end of the fourth feature fusion unit is also connected to the output end of the third feature fusion unit, and the output of the fourth feature fusion unit is the output of the multi-channel shape enhancement module;

[0077] The channel attention module is used to assign different weight values ​​to each channel of the target feature. The shape enhancement module further enhances the shape information of the target feature by constructing a shape-adaptive weight map. The third feature fusion unit is used to multiply and fuse two input features. The fourth feature fusion unit is used to multiply and fuse two input features.

[0078] In an embodiment of the present invention, the channel attention module in the multi-channel shape enhancement module learns to assign different weight values ​​to each channel of the feature, guiding the network to focus on the feature channel with the infrared target. The shape enhancement module in the multi-channel shape enhancement module further enhances the shape information of the infrared target by constructing a shape adaptive weight map, thereby further enhancing the shape information of the features extracted by the bidirectional shape adaptive extraction module and improving the detection accuracy of the infrared target.

[0079] It should be noted that in the attached Figure 5 middle, Represents the feature fusion unit.

[0080] Furthermore, in an embodiment of the present invention, the output of the channel attention module can be expressed as:

[0081]

[0082] Att C =S(MLP(P max (O BSAA ))+MLP(P avg (O BSAA )));

[0083] Among them, O C Represents the output of the channel attention module, Att C represents a one-dimensional attention weight map, R represents a real number set, C1 represents the number of channels of the feature map extracted by the bidirectional shape adaptive extraction module, represents the element-by-element matrix multiplication operation, O BSAArepresents the output of the bidirectional shape adaptive extraction module, S(·) represents the Sigmoid activation function, MLP(·) represents a multi-layer perceptron, which contains one hidden layer and is used to output c weight values, c represents the number of channels of the multi-layer perceptron output tensor, and P max (·) represents the maximum pooling operation, P avg (·) represents the average pooling operation.

[0084] Furthermore, in the embodiment of the present invention, the output of the shape enhancement module can be expressed as:

[0085]

[0086] Att S =S(ShapeFilter k×k [P max (O C ),P avg (O C )]);

[0087] Among them, O S represents the output of the shape enhancement module, Att S represents a two-dimensional shape enhancement weight map, R represents a real number set, C2 represents the number of channels of the output feature map of the channel attention module, H represents the length size of the output feature map of the channel attention module, W represents the width size of the output feature map of the channel attention module, represents the element-by-element matrix multiplication operation, S(·) represents the Sigmoid activation function, ShapeFilter k×k represents a k×k deformable convolution, [·] represents the feature concatenation operation, P max (·) represents the maximum pooling operation, P avg (·) represents the average pooling operation.

[0088] refer to Figure 2 ,Further, in the embodiment of the present invention, the output layer includes: a concatenation unit and a 1×1 convolution layer;

[0089] The input end of the splicing unit is respectively connected to the outputs of multiple decoder branches of the densely nested U-shaped feature extraction network based on spatial feature fusion, and the output end of the splicing unit is connected to the input end of the 1×1 convolution layer, and the output of the 1×1 convolution layer is the output of the output layer;

[0090] The concatenation unit is used to concatenate multiple input features, and the 1×1 convolution layer is used to perform 1×1 convolution on the input features.

[0091] In an embodiment of the present invention, the output layer obtains and outputs a binary segmentation map by concatenating and convolving features output by multiple decoder branches of a densely nested U-shaped feature extraction network based on spatial feature fusion.

[0092] It should be noted that in the attached Figure 2 In , 1×1Conv represents a 1×1 convolutional layer. Represents a splicing unit.

[0093] Furthermore, in an embodiment of the present invention, the multi-attention shape adaptation network is pre-trained by a multi-scale multi-branch deep supervision training method.

[0094] refer to Figure 6 In an embodiment of the present invention, pre-training by multi-scale multi-branch deep supervised training specifically includes the following steps:

[0095] Step 21, obtaining a training data set, the training data including an infrared image and a binary segmentation map of an infrared target corresponding to the infrared image;

[0096] In the embodiment of the present invention, when acquiring training data, the infrared image may use currently existing image data.

[0097] Step 22, inputting the infrared images in all the training data into the multi-attention shape adaptive network in batches, obtaining a first predicted infrared target binary segmentation map corresponding to each infrared image output by the output layer of the multi-attention shape adaptive network, and performing upsampling operation on the outputs at different network depth nodes of the multi-attention shape adaptive network to scale them to the same size, and then performing splicing and convolution processing to obtain a second predicted infrared target binary segmentation map corresponding to each infrared image;

[0098] Step 23, calculating the target detection loss function according to the infrared target binary segmentation map in the training data, and the first predicted infrared target binary segmentation map and the second predicted infrared target binary segmentation map obtained by the multi-attention shape adaptation network;

[0099] Step 24, determine whether the preset training stop condition is reached. If so, use the current multi-attention shape adaptation network as the multi-attention shape adaptation network that has completed the training. If not, use the target detection loss function to update the parameters of the multi-attention shape adaptation network and return to step 21.

[0100] In an embodiment of the present invention, the parameters of the multi-attention shape adaptation network are initialization parameters. During the training process of the model, the parameters of the model are continuously updated and learned.

[0101] In the embodiment of the present invention, the training stop condition is specifically set according to the actual situation, for example, the number of training iterations reaches a set number of iterations or the optimization index reaches a set threshold. Among them, the target detection loss function can be used as the optimization index.

[0102] In the embodiment of the present invention, the target detection loss function is specifically set according to the actual situation, and the existing conventional loss function for infrared target detection can be adopted.

[0103] In an embodiment of the present invention, the stochastic gradient descent method is used to train and update the parameters of the multi-attention shape adaptation network.

[0104] In an embodiment of the present invention, by adopting a multi-scale multi-branch deep supervised training method to pre-train the multi-attention shape adaptive network to update the parameters of the multi-attention shape adaptive network, the training efficiency can be improved, overfitting can be prevented, and the detection accuracy of the trained multi-attention shape adaptive network can be ensured.

[0105] It should be noted that in the attached Figure 6 In the above, up-sampling refers to the up-sampling operation.

[0106] refer to Figure 7 Furthermore, in an optional implementation of an embodiment of the present invention, when pre-training the multi-attention shape adaptation network, the trained multi-attention shape adaptation network can also be pruned to simplify the structure and parameters of the multi-attention shape adaptation network.

[0107] In the embodiments of the present invention, the multi-attention shape adaptation network can be pruned according to the actual infrared image detection task requirements to simplify the structure and parameters of the multi-attention shape adaptation network, thereby improving the detection efficiency and reducing the detection time while ensuring the detection accuracy.

[0108] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An infrared target detection method based on multi-attention shape adaptation network, characterized in that: The method comprises: Acquire the infrared image to be detected; Inputting the infrared image to be detected into a pre-trained multi-attention shape adaptation network to obtain a binary segmentation map corresponding to the infrared image to be detected output by the multi-attention shape adaptation network; The multi-attention shape adaptation network includes: a densely nested U-shaped feature extraction network based on spatial feature fusion and an output layer, a bidirectional shape adaptation extraction module and a multi-channel shape enhancement module connected in sequence are added to the lateral connection of the densely nested U-shaped feature extraction network based on spatial feature fusion, the input of the densely nested U-shaped feature extraction network based on spatial feature fusion is used to receive infrared images, and the output is connected to the output layer, the densely nested U-shaped feature extraction network based on spatial feature fusion is used to spatially fuse visual features of different semantic levels to obtain highly representative infrared target features, the bidirectional shape adaptation extraction module is used to adaptively extract shape information of infrared targets, the multi-channel shape enhancement module is used to enhance the shape information extracted by the bidirectional shape adaptation extraction module, and the output layer is used to receive and process the output of the densely nested U-shaped feature extraction network based on spatial feature fusion to obtain and output a binary segmentation map.

2. The infrared target detection method based on multi-attention shape adaptive network according to claim 1 is characterized in that: The densely nested U-shaped feature extraction network based on spatial feature fusion includes: an encoder branch and a decoder branch, the encoder branch includes one, the decoder branch includes four symmetrical branches, the encoder branch is shared by four symmetrical decoder branches, the encoder branch and the decoder branch are also connected by dense jump connections, a bidirectional shape adaptive extraction module and a multi-channel shape enhancement module connected in sequence are added to the lateral connection of each decoder branch, and the output of each decoder branch is respectively connected to the output layer.

3. The infrared target detection method based on multi-attention shape adaptive network according to claim 2 is characterized in that: The encoder branch includes five feature layers sequentially connected from top to bottom through the downsampling layer, and the decoder branch includes a first decoder branch, a second decoder branch, a third decoder branch and a fourth decoder branch; The first decoder branch includes a first feature fusion layer, a first bidirectional shape adaptive extraction module and a first multi-channel shape enhancement module, the second feature layer of the encoder branch is connected to the first feature fusion layer through an upsampling layer, the first feature layer of the encoder branch is connected to the first feature fusion layer, the output of the first feature fusion layer is horizontally connected to the first bidirectional shape adaptive extraction module, the first feature layer of the encoder branch is jump-connected to the first bidirectional shape adaptive extraction module, the first bidirectional shape adaptive extraction module and the first multi-channel shape enhancement module are connected in sequence, the output of the first multi-channel shape enhancement module is connected to the output layer, and the first feature fusion layer is used to splice and fuse the input features; The second decoder branch includes a second feature fusion layer, a third feature fusion layer, a second bidirectional shape adaptive extraction module, a second multi-channel shape enhancement module, a third bidirectional shape adaptive extraction module and a third multi-channel shape enhancement module. The third feature layer of the encoder branch is connected to the second feature fusion layer through an upsampling layer, the second feature layer of the encoder branch is connected to the second feature fusion layer, the first feature fusion layer is connected to the second feature fusion layer through a downsampling layer, the output of the second feature fusion layer is horizontally connected to the second bidirectional shape adaptive extraction module, the second feature layer of the encoder branch is also jump-connected to the second bidirectional shape adaptive extraction module, the second bidirectional shape adaptive extraction module and the second multi-channel shape enhancement module are connected. The output of the second multi-channel shape enhancement module is connected to the third feature fusion layer through the upsampling layer, the output of the first multi-channel shape enhancement module is also connected to the third feature fusion layer, the first feature layer of the encoder branch is also jump-connected to the third feature fusion layer, the output of the third feature fusion layer is horizontally connected to the third bidirectional shape adaptive extraction module, the first feature layer of the encoder branch is also jump-connected to the third bidirectional shape adaptive extraction module, the third bidirectional shape adaptive extraction module and the third multi-channel shape enhancement module are connected in sequence, the output of the third multi-channel shape enhancement module is connected to the output layer, and the second feature fusion layer and the third feature fusion layer are both used to splice and fuse the input features; The third decoder branch includes a fourth feature fusion layer, a fifth feature fusion layer, a sixth feature fusion layer, a fourth bidirectional shape adaptive extraction module, a fourth multi-channel shape enhancement module, a fifth bidirectional shape adaptive extraction module, a fifth multi-channel shape enhancement module, a sixth bidirectional shape adaptive extraction module and a sixth multi-channel shape enhancement module. The fourth feature layer of the encoder branch is connected to the fourth feature fusion layer through an upsampling layer, the third feature layer of the encoder branch is connected to the fourth feature fusion layer, the second feature fusion layer is connected to the fourth feature fusion layer through a downsampling layer, the output of the fourth feature fusion layer is horizontally connected to the fourth bidirectional shape adaptive extraction module, the third feature layer of the encoder branch is also jump-connected to the fourth bidirectional shape adaptive extraction module, the fourth bidirectional shape adaptive extraction module and the fourth multi-channel shape enhancement module are connected in sequence, the output of the fourth multi-channel shape enhancement module is connected to the fifth feature fusion layer through an upsampling layer, the output of the second multi-channel shape enhancement module is also connected to the fifth feature fusion layer, the second feature layer of the encoder branch is also jump-connected to the fifth feature fusion layer, the third feature fusion layer The downsampling layer is connected to the fifth feature fusion layer, the output of the fifth feature fusion layer is horizontally connected to the fifth bidirectional shape adaptive extraction module, the second feature layer of the encoder branch is also jump-connected to the fifth bidirectional shape adaptive extraction module, the fifth bidirectional shape adaptive extraction module and the fifth multi-channel shape enhancement module are connected in sequence, the output of the fifth multi-channel shape enhancement module is connected to the sixth feature fusion layer through the upsampling layer, the output of the third multi-channel shape enhancement module is also connected to the sixth feature fusion layer, the first feature layer of the encoder branch is also jump-connected to the sixth feature fusion layer, the output of the first multi-channel shape enhancement module is also jump-connected to the sixth feature fusion layer, the output of the sixth feature fusion layer is horizontally connected to the sixth bidirectional shape adaptive extraction module, the first feature layer of the encoder branch is also jump-connected to the sixth bidirectional shape adaptive extraction module, the sixth bidirectional shape adaptive extraction module and the sixth multi-channel shape enhancement module are connected in sequence, the output of the sixth multi-channel shape enhancement module is connected to the output layer, the fourth feature fusion layer, the fifth feature fusion layer and the sixth feature fusion layer are all used to splice and fuse the input features; The fourth decoder branch includes a seventh feature fusion layer, an eighth feature fusion layer, a ninth feature fusion layer, a tenth feature fusion layer, a seventh bidirectional shape adaptive extraction module, a seventh multi-channel shape enhancement module, an eighth bidirectional shape adaptive extraction module, an eighth multi-channel shape enhancement module, a ninth bidirectional shape adaptive extraction module, a ninth multi-channel shape enhancement module, a tenth bidirectional shape adaptive extraction module and a tenth multi-channel shape enhancement module, the fifth feature layer of the encoder branch is connected to the seventh feature fusion layer through an upsampling layer, the fourth feature layer of the encoder branch is connected to the seventh feature fusion layer, the fourth feature fusion layer is connected to the seventh feature fusion layer through a downsampling layer, and the output of the seventh feature fusion layer is laterally connected to the seventh bidirectional shape adaptive extraction module. The fourth feature layer of the encoder branch is also jump-connected to the seventh bidirectional shape adaptive extraction module, the seventh bidirectional shape adaptive extraction module and the seventh multi-channel shape enhancement module are connected in sequence, the output of the seventh multi-channel shape enhancement module is connected to the eighth feature fusion layer through the upsampling layer, the output of the fourth multi-channel shape enhancement module is also connected to the eighth feature fusion layer, the third feature layer of the encoder branch is also jump-connected to the eighth feature fusion layer, the fifth feature fusion layer is connected to the eighth feature fusion layer through the downsampling layer, the output of the eighth feature fusion layer is laterally connected to the eighth bidirectional shape adaptive extraction module, the third feature layer of the encoder branch is also jump-connected to the eighth bidirectional shape adaptive extraction module, the eighth bidirectional shape adaptive extraction module The output of the eighth multi-channel shape enhancement module is connected to the ninth feature fusion layer through the upsampling layer, the output of the fifth multi-channel shape enhancement module is also connected to the ninth feature fusion layer, the second feature layer of the encoder branch is also jump-connected to the ninth feature fusion layer, the output of the second multi-channel shape enhancement module is also jump-connected to the ninth feature fusion layer, the sixth feature fusion layer is connected to the ninth feature fusion layer through the downsampling layer, the output of the ninth feature fusion layer is laterally connected to the ninth bidirectional shape adaptive extraction module, the second feature layer of the encoder branch is also jump-connected to the ninth bidirectional shape adaptive extraction module, the ninth bidirectional shape adaptive extraction module and the ninth multi-channel shape enhancement module are connected in sequence, and the ninth multi-channel The output of the channel shape enhancement module is connected to the tenth feature fusion layer through the upsampling layer, the output of the sixth multi-channel shape enhancement module is also connected to the tenth feature fusion layer, the first feature layer of the encoder branch is also jump-connected to the tenth feature fusion layer, the output of the first multi-channel shape enhancement module is also jump-connected to the tenth feature fusion layer, the output of the third multi-channel shape enhancement module is also jump-connected to the tenth feature fusion layer, the output of the tenth feature fusion layer is laterally connected to the tenth bidirectional shape adaptive extraction module, the first feature layer of the encoder branch is also jump-connected to the tenth bidirectional shape adaptive extraction module, the tenth bidirectional shape adaptive extraction module and the tenth multi-channel shape enhancement module are connected in sequence, and the output of the tenth multi-channel shape enhancement module is connected to the output layer.The seventh feature fusion layer, the eighth feature fusion layer, the ninth feature fusion layer and the tenth feature fusion layer are all used to concatenate and fuse the input features.

4. The infrared target detection method based on multi-attention shape adaptive network according to claim 2 or 3, characterized in that: The bidirectional shape adaptive extraction module includes: a horizontal deformable convolution layer, a vertical deformable convolution layer, a first Sigmoid activation function layer, a second Sigmoid activation function layer, a first feature fusion unit, a second feature fusion unit, a first splicing unit, a second splicing unit and a third splicing unit; The input end of the horizontal deformable convolution layer is used to connect with the feature layer of the encoder branch corresponding to the bidirectional shape adaptive extraction module, the output end of the horizontal deformable convolution layer is connected with the input end of the first Sigmoid activation function layer, the output end of the first Sigmoid activation function layer is connected with the input end of the first feature fusion unit, the input end of the first feature fusion unit is also used to connect with the feature fusion layer of the decoder branch corresponding to the bidirectional shape adaptive extraction module, the output end of the first feature fusion unit is connected with the input end of the first splicing unit, the input end of the first splicing unit is also used to connect with the feature layer of the encoder branch corresponding to the bidirectional shape adaptive extraction module, the output end of the first splicing unit is connected with the input end of the third splicing unit, the input end of the vertical deformable convolution layer Used to be connected to the feature layer of the encoder branch corresponding to the bidirectional shape adaptive extraction module, the output end of the longitudinal deformable convolution layer is connected to the input end of the second Sigmoid activation function layer, the output end of the second Sigmoid activation function layer is connected to the input end of the second feature fusion unit, the input end of the second feature fusion unit is also used to be connected to the feature fusion layer of the decoder branch corresponding to the bidirectional shape adaptive extraction module, the output end of the second feature fusion unit is connected to the input end of the second splicing unit, the input end of the second splicing unit is also used to be connected to the feature layer of the encoder branch corresponding to the bidirectional shape adaptive extraction module, the output end of the second splicing unit is connected to the input end of the third splicing unit, and the output of the third splicing unit is the output of the bidirectional shape adaptive extraction module; The horizontal deformable convolution layer performs horizontal deformable convolution on the input features, the vertical deformable convolution layer performs vertical deformable convolution on the input features, the first Sigmoid activation function layer uses the Sigmoid activation function to perform nonlinear transformation on the input features, the second Sigmoid activation function layer uses the Sigmoid activation function to perform nonlinear transformation on the input features, the first feature fusion unit is used to multiply and fuse two input features, the second feature fusion unit is used to multiply and fuse two input features, the first splicing unit is used to splice two input features, the second splicing unit is used to splice two input features, and the third splicing unit is used to splice two input features.

5. The infrared target detection method based on multi-attention shape adaptive network according to claim 4 is characterized in that: The multi-channel shape enhancement module includes: a channel attention module, a shape enhancement module, a third feature fusion unit and a fourth feature fusion unit; The input end of the channel attention module is used to be connected to the output end of the bidirectional shape adaptive extraction module, the output end of the channel attention module is connected to the input end of the third feature fusion unit, the input end of the third feature fusion unit is also used to be connected to the output end of the bidirectional shape adaptive extraction module, the output end of the third feature fusion unit is connected to the input end of the shape enhancement module, the output end of the shape enhancement module is connected to the input end of the fourth feature fusion unit, the input end of the fourth feature fusion unit is also connected to the output end of the third feature fusion unit, and the output of the fourth feature fusion unit is the output of the multi-channel shape enhancement module; The channel attention module is used to assign different weight values ​​to each channel of the target feature. The shape enhancement module further enhances the shape information of the target feature by constructing a shape-adaptive weight map. The third feature fusion unit is used to multiply and fuse the two input features. The fourth feature fusion unit is used to multiply and fuse the two input features.

6. The infrared target detection method based on multi-attention shape adaptive network according to claim 5 is characterized in that: The output of the channel attention module is expressed as: To C =S(MLP(P max (O BSAA ))+MLP(P avg (O BSAA ))); Among them, O C Represents the output of the channel attention module, Att C represents a one-dimensional attention weight map, represents the element-by-element matrix multiplication operation, O BSAA represents the output of the bidirectional shape adaptive extraction module, S(·) represents the Sigmoid activation function, MLP(·) represents a multi-layer perceptron, which contains a hidden layer and is used to output weight values, P max (·) represents the maximum pooling operation, P avg (·) represents the average pooling operation.

7. The infrared target detection method based on multi-attention shape adaptive network according to claim 6 is characterized in that: The output of the shape enhancement module is expressed as: To S =S(ShapeFilter k×k [P max (O C ),P avg (O C )]); Among them, O S represents the output of the shape enhancement module, Att S Represents a two-dimensional shape enhancement weight map, ShapeFilter k×k represents a k×k deformable convolution, and [·] represents a feature concatenation operation.

8. The infrared target detection method based on multi-attention shape adaptive network according to claim 2 or 3, characterized in that: The multi-attention shape adaptation network is pre-trained by a multi-scale multi-branch deep supervised training method.

9. The infrared target detection method based on multi-attention shape adaptive network according to claim 8 is characterized in that: The multi-scale multi-branch deep supervision training method comprises the following steps: Step 21, obtaining a training data set, the training data including an infrared image and a binary segmentation map of an infrared target corresponding to the infrared image; Step 22, inputting the infrared images in all the training data into the multi-attention shape adaptive network in batches, obtaining a first predicted infrared target binary segmentation map corresponding to each infrared image output by the output layer of the multi-attention shape adaptive network, and performing upsampling operation on the outputs at different network depth nodes of the multi-attention shape adaptive network to scale them to the same size, and then performing splicing and convolution processing to obtain a second predicted infrared target binary segmentation map corresponding to each infrared image; Step 23, calculating the target detection loss function according to the infrared target binary segmentation map in the training data, and the first predicted infrared target binary segmentation map and the second predicted infrared target binary segmentation map obtained by the multi-attention shape adaptation network; Step 24, determine whether the preset training stop condition is reached. If so, use the current multi-attention shape adaptation network as the multi-attention shape adaptation network that has completed the training. If not, use the target detection loss function to update the parameters of the multi-attention shape adaptation network and return to step 21.

10. The infrared target detection method based on multi-attention shape adaptive network according to claim 9 is characterized in that: The method further comprises: When pre-training the multi-attention shape adaptation network, pruning is performed on the trained multi-attention shape adaptation network.

Images