A sandglass visual transformer dual-path encoding and decoding network camouflage target detection method

Abstract

The application discloses a kind of Hourglass visual Transformer dual-path encoding and decoding network camouflage target detection methods, first, Hourglass visual Transformer (Hourglass-VIT) is used to extract low-level features and high-level features of multi-scale simultaneously;Second, the feature fusion of top-down and bottom-up is carried out to multi-scale feature map using dual-path feature pyramid decoder;Then, high-low interactive enhancement module is used to mine low-level and high-level feature correlation, and obtain multi-scale correlation enhanced feature map;Finally, the prediction map of camouflage target is generated by simultaneously inputting multi-scale correlation enhanced feature into network detection head, and the detection of camouflage target in image is realized.The method of the application can significantly improve the distinguishability and recognition ability of camouflage target in multi-class complex scene, especially for small target, target boundary detection effect is good.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for detecting camouflaged targets using an hourglass vision Transformer dual-channel encoder-decoder network. Background Technology

[0002] Camouflage is a key survival skill evolved by organisms in nature. By changing color, texture, and other characteristics, organisms can blend into their environment, thereby deceiving or hiding themselves. Camouflage target detection aims to identify objects that are highly similar to their surroundings and has been widely used in military target detection, medical image segmentation, and industrial defect detection.

[0003] In recent years, with the development of deep learning technology and the release of large-scale camouflage datasets, research on camouflage target analysis has received strong technical and data support. However, CNN-based methods (such as DGNet and SINet) are limited by the inherent limitation of CNNs in extracting features from local receptive fields, making it difficult to capture long-range dependencies of features and global cues of camouflaged objects. This often results in predicting only incomplete object regions, especially for multiple objects, large objects, and occlusion scenarios. In contrast, the Visual Transformer (ViT) effectively builds long-range dependency models using self-attention operations, demonstrating good performance. However, because ViT segments images into multiple blocks and processes them sequentially, the output resolution is relatively low, providing only a single resolution. Furthermore, its attention mechanism mainly focuses on global information, with insufficient perception of local details, leading to poor segmentation performance for small targets and subtle target boundaries (such as FSPNet and UGTR). Summary of the Invention

[0004] The purpose of this invention is to provide a method for detecting camouflaged targets using an hourglass vision Transformer dual-channel encoder-decoder network.

[0005] To achieve the above objectives, the present invention is implemented according to the following technical solution:

[0006] This invention includes the following steps:

[0007] S1: Extracting multi-scale feature maps: The input image is processed by the encoder to extract multi-scale feature maps, ranging from low-level features with graphic characteristics to high-level features with semantic characteristics, capturing spatial information from coarse to fine.

[0008] S2: Feature fusion: Low-level features with rich details and high-level features with semantics from different scales of the encoder are fused from the middle of the backbone network in a top-down or bottom-up manner through different paths of the decoder.

[0009] S3: Multi-scale fusion feature association enhancement: The fused features further explore and utilize the complex relationship between low-level local features and high-level cue features, and fuse the structural details and overall semantic information of the camouflaged target to obtain a multi-scale association enhancement feature map of the camouflaged target;

[0010] S4: Generate a prediction map of the camouflaged target using multi-scale correlation enhancement feature map: Simultaneously input the multi-scale correlation enhancement feature map into the network detection head module to generate a prediction map of segmented camouflaged targets, thereby realizing the detection of camouflaged targets in the image.

[0011] Preferably, the encoder in step S1 is an Hourglass-VIT encoder; the decoder in step S2 is a dual-path feature pyramid decoder; and the multi-scale fusion feature association enhancement in step S3 adopts a high-low interaction enhancement module.

[0012] The beneficial effects of this invention are:

[0013] This invention presents a method for detecting camouflaged targets using an hourglass-visual Transformer dual-channel encoder-decoder network. Compared to existing technologies, this invention employs an Hourglass-VIT encoder to output multi-level, multi-scale feature maps, gradually recovering spatial dimensionality information lost during downsampling in forward propagation. Furthermore, a dual-channel feature pyramid decoder is designed to progressively aggregate low-level features rich in detail and high-level features with semantic meaning, avoiding noise and dilution of weak feature cues introduced by direct feature fusion. To more effectively mine and utilize the complex relationship between low-level local features and high-level cue features, this invention designs a high-low feature interaction enhancement module that maintains equal attention to both high-level and low-level features, further mining and fusing the structural details and overall semantic information of the camouflaged target, thereby effectively identifying the camouflaged object. The method of this invention significantly improves the discernibility and recognition ability of camouflaged targets in various complex scenes, especially demonstrating excellent detection performance for small targets and target boundaries. Attached Figure Description

[0014] Figure 1 This is a flowchart of the camouflaged target detection method using hourglass vision Transformer dual-path feature pyramid decoding according to the present invention;

[0015] Figure 2 This is a schematic diagram of the hourglass vision Transformer dual-path feature pyramid decoding network structure;

[0016] Figure 3 This is a schematic diagram of the dual-path feature pyramid decoder group fusion module;

[0017] Figure 4 This is a schematic diagram of the high-low feature interaction enhancement module;

[0018] Figure 5 These are sample images for detecting camouflaged targets in multiple scenarios. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The illustrative embodiments and descriptions herein are used to explain the present invention, but are not intended to limit the present invention.

[0020] This invention provides a method for detecting camouflaged targets using hourglass vision Transformer dual-path feature pyramid decoding, the flowchart of which is shown below. Figure 1 As shown, the deep learning model framework is as follows: Figure 2 As shown. The specific implementation method is as follows:

[0021] S1: The Hourglass-VIT encoder is used to extract multi-scale feature maps. The input image is processed by the Hourglass-VIT encoder to extract multi-scale feature maps, ranging from low-level features with graphical properties to high-level features with semantic properties, capturing spatial information from coarse to fine. The Hourglass-VIT encoder model framework is as follows: Figure 2 As shown, it specifically includes the following sub-steps:

[0022] S1.1: The input image scale is uniformly scaled to h×w×3. In this example, both h and w are 384.

[0023] S1.2: Given an image I∈R HxWxC Use convolution in the patch embedding layer to divide it into HW / k 2 Patch the feature map to reduce its spatial resolution, or use pixel-shuffle to restore the feature map resolution, and then perform linear projection to obtain a one-dimensional labeled embedding sequence T. o ∈R HW / k2×C1 (downsampling) or T o ∈R HWk2×C2 (Upsampling), where k is the scaling scale, C1 and C2 are the number of channels, and H and W represent the height and width of the image, respectively.

[0024] S1.3: To preserve the original image spatial location information, additional learnable locations are embedded in E. p

[0025] Add to the token to form a new token T p =T0+E pThen, all tokens are input into a two-layer Transformer encoder, where each layer contains a multi-head self-attention module (MSA) and a multilayer perceptron (MLP) structure. Both the MAS and MLP contain a residual structure and layer normalization (LN). This process can be described as follows:

[0026] T M =MSA(LN(T) p ))+T p

[0027] T = MLP(LN(T) M ))+T M

[0028] S1.4: The token after passing through the Transformer encoder is reshaped into a feature map X of size H / K×W / K×C1 or HK×WK×C2.

[0029] S1.5: Repeat steps S1.1-S1.4 six times to obtain Hourglass-VIT structures with different scales, consisting of 12 stacked Transformer blocks. The feature map obtained from each Transformer block can be considered as {x1,x2,x3,x4,x5,x6,x7,x8,x9,x10,x11,x12}. {x1,x2,x3,x4,x5,x6} and

[0030] {x7, x8, x9, x10, x11, x12} are inputs into the low-level or high-level decoding paths of the decoder, respectively. Furthermore, residual connections are used to introduce features passed from the corresponding scale coding layer to preserve local and detailed information, compensate for the information loss from upsampling, and enhance the model's reconstruction capability, such as... Figure 2 As shown by the red dashed line.

[0031] S2: Feature fusion is performed using a dual-path feature pyramid decoder. Low-level features rich in detail and high-level features with semantic meaning from different scales of the encoder are fused from the middle of the backbone network via different paths in the decoder, either top-down or bottom-up. The dual-path feature pyramid decoder is as follows: Figure 2 As shown;

[0032] S2.1: The feature maps {x1,x2,x3,x4,x5,x6} and {x7,x8,x9,x10,x11,x12} obtained from the Transformer blocks of each layer of Hourglass-VIT are respectively input into the low-level or high-level decoding path of the decoder.

[0033] S2.2: The decoder, based on the Group Fusion Module (GFM), simultaneously establishes top-down and bottom-up feature flows starting from the middle of the backbone, achieving dual-path feature fusion. Taking bottom-up feature fusion as an example, the decoder accepts the feature map from the current layer of Hourglass-VIT and the output of the previous GFM layer (if present) as input to the current GFM layer. After integration according to certain rules, the GFM guides the high-resolution features and uses the output as input to the next GFM layer. Finally, after three stages of GFM processing, two outputs of half the original image size are obtained. The GFM process can be represented as:

[0034] F o1 =UP(CBR(Concat(CBR(Concat(F5,F6)),F7)))

[0035] F o2 =UP(CBR(Concat(CBR(Concat(F5,F6)),F8)))

[0036] Where Concat indicates concatenation by channel, CBR represents convolution, batch normalization, and ReLU activation function, and UP indicates upsampling by 2x. Initially, F5, F6, F7, and F8 represent the feature maps {x5, x6, x7, x8} of the same size in the Hourglass-VIT layer, respectively. Alternatively, F5 and F6 represent the output of the previous GFM layer, and F7 and F8 represent the feature maps of the current Hourglass-VIT layer.

[0037] S2.3: The top-down and bottom-up processes are consistent, starting from small-scale features in the middle of the network and gradually merging large-scale, low-level features rich in detail downwards.

[0038] S2.4: The 12 feature vectors output by the Hourglass-VIT backbone after feature fusion are mapped to the fused high-level feature F. h1 F h2 and low-level features F I1 F l2 High-level and low-level features are fused separately to obtain the final fused features, which are then input into the feature interaction enhancement module.

[0039] F h =CBR 3×3 (CBR 1×1 (Concat(F h1 ,F h2 )))

[0040] F l =CBR 3×3 (CBR 1×1(Concat(F I1 ,F l2 )))

[0041] Wherein, Concat indicates concatenation by channel; CBR 1x1 This indicates a convolution with output channel C2 being half the size of input channel C1, a kernel size of 1, and batch normalization and ReLU activation function; Conv 3x3 F represents a convolution with a kernel size of 3, batch normalization, and ReLU activation function; h F l This represents the final output of the dual-path feature shrinking decoder.

[0042] S3: Multi-scale fusion feature association enhancement is performed using a high-low interaction enhancement module. The fused features are further processed by the feature interaction enhancement module to mine and utilize the complex relationships between low-level local features and high-level cue features, fusing structural details and overall semantic information of the camouflaged target to obtain a multi-scale association enhanced feature map of the camouflaged target. The high-low feature interaction enhancement module is as follows: Figure 4 As shown;

[0043] S3.1: To inject detailed appearance features into high-level semantic features, the high-level feature Fh is processed through two linear mapping functions with trainable weights. and The dimensionality-reduced feature sequences Q and K are obtained to reduce computational cost. Low-level features Fl are processed through a windowed multi-head self-attention module (W-MSA) to obtain a local attention vector A, enhancing the model's focus on local features. This process can be represented as:

[0044] A = W - MSA(F) l )

[0045] S3.2: Apply the softmax function to A to generate a weight map and calculate its Hadamard product with K to increase the weight of edge pixels, and then perform adaptive average pooling. Obtain the feature map V. This operation is... Figure 4 F(.) can be represented as:

[0046]

[0047] S3.3: Apply matrix multiplication to K and V to explore the correlation between them, and use the softmax operation to generate a relevant attention map W.

[0048]

[0049] S3.4: After obtaining the relevant attention map W, it is multiplied with the feature map Q. The resulting feature map is then input into the graph convolutional layer GCN(-) to learn high-level semantic relationships between regions (sets of pixels with similar features). To reconstruct the graph domain features into the original structural features, the inner product between GCN(-) and G is calculated and passed through a linear mapping function. Map back to two-dimensional image features with the same dimensions as the original features, and then combine with feature F. h Combining these, we obtain the final output of FIEM, and the above operations can be performed accordingly. Figure 4 Upper and middle parts:

[0050]

[0051] S3.5: Similarly, in order to inject contextual information into shallow features, this invention addresses F h and F I Repeat the above steps to obtain F2. Note that F... h This operation does not go through the W-MSA module. Figure 4 The lower half. Finally, F1 and F2 are further merged to obtain the final output:

[0052] F o =Conv 3x3 (Conv 1×1 (cat(F1,F2)))

[0053] Where cat represents concatenation of channel dimensions, Conv 1x1 This indicates a convolution with a kernel size of 1, adjusted to change the number of channels, such that F... o F1, F2, and Conv all have the same number of channels. 3x3 This represents a convolution with a kernel size of 3.

[0054] S4: Generate a prediction map of the camouflaged target using multi-scale correlation enhancement features. Simultaneously, input the multi-scale correlation enhancement features into the network detection head module to generate a segmented prediction map of the camouflaged target, thus achieving camouflaged target detection in the image. For example... Figure 2 and Figure 4 As shown;

[0055] S4.1: The shallow features with injected semantic information, the deep features with injected boundary information, and the fused features obtained from the high-low feature interaction enhancement module are fed into the Sigmoid segmentation head to generate a prediction map of the camouflaged target.

[0056] Out1 = Sigmoid(Conv) 1×1 Up 2× (CBR(F o )))

[0057] Out2 = Sigmoid(Conv) 1×1 Up 2× (CBR(F1)))

[0058] Out3 = Sigmoid(Conv) 1×1 Up 2× (CBR(F2)))

[0059] Where CBR represents convolution, batch normalization, and ReLU activation function operations; Up2x represents 2x upsampling; Conv 1x1 This represents a convolution with input channel C, output channel 1, and kernel size 1.

[0060] S4.2: The network loss function consists of three parts:

[0061]

[0062] Where G represents the truth graph; L BCE This represents the binary cross-entropy loss; λ1, λ2, λ3 represent the weighted binary cross-entropy loss; λ1, λ2, λ3 represent the loss balancing weights, which are set to 0.25, 0.25, 1.0 in this example.

[0063] The technical solutions of the present invention are not limited to the specific embodiments described above. Any technical modifications made in accordance with the technical solutions of the present invention fall within the protection scope of the present invention.

Claims

1. A method for detecting camouflaged targets using an hourglass-shaped visual Transformer dual-channel encoder-decoder network, characterized in that, Includes the following steps: S1: Extracting multi-scale feature maps: The input image is processed by an encoder to extract multi-scale feature maps, ranging from low-level features with graphical characteristics to high-level features with semantic characteristics, capturing spatial information from coarse to fine; the encoder is an Hourglass-VIT encoder; the decoder in step S2 is a dual-path feature pyramid decoder; the multi-scale fusion feature association enhancement in step S3 adopts a high-low interaction enhancement module; S1.1: The input image scale is uniformly scaled to h×w×3; S1.2: Given an image I ∈ Use convolution in the patch embedding layer to divide it into HW / k 2 One approach is to use patches to reduce the feature map spatial resolution, or to use pixel-shuffle to restore the feature map resolution, followed by linear projection to obtain a one-dimensional labeled embedding sequence. or Where k is the scaling factor, , H represents the number of channels, and H and W represent the height and width of the image, respectively. S1.3: Embed additional learnable locations in E p Add to the token to form a new token T p = T0+ E p Then, all tokens are input into a two-layer Transformer encoder, where each layer contains a multi-head self-attention module and a multilayer perceptron structure; both MAS and MLP contain a residual structure and layer normalization, and this process is described as follows: S1.4: The token after passing through the Transformer encoder is reshaped into a feature map X of size H / K ×W / K ×C1 or HK ×WK×C2; S1.5: Repeat steps S1.1-S1.4 six times to obtain Hourglass-VIT structures with different scales, consisting of 12 stacked Transformer blocks. The feature maps obtained from each Transformer block are considered as {x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12}. {x1, x2, x3, x4, x5, x6} and {x7, x8, x9, x10, x11, x12} are input into the low-level and high-level decoding paths of the decoder, respectively. S2: Feature fusion: Low-level features with rich details and high-level features with semantics from different scales of the encoder are fused from the middle of the backbone network in a top-down or bottom-up manner through different paths of the decoder. S3: Multi-scale fusion feature association enhancement: The fused features further explore and utilize the complex relationship between low-level local features and high-level cue features, and fuse the structural details and overall semantic information of the camouflaged target to obtain a multi-scale association enhancement feature map of the camouflaged target; S4: Generate a prediction map of the camouflaged target using multi-scale correlation enhancement feature map: Simultaneously input the multi-scale correlation enhancement feature map into the network detection head module to generate a prediction map of segmented camouflaged targets, thereby realizing the detection of camouflaged targets in the image.

2. The method for detecting camouflaged targets using an hourglass vision Transformer dual-channel encoder-decoder network according to claim 1, characterized in that: Step S2 specifically involves: S2.1: The feature maps {x1, x2, x3, x4, x5, x6} and {x7, x8, x9, x10, x11, x12} obtained from the Transformer blocks of each layer of Hourglass-VIT are respectively input into the low-level or high-level decoding path of the decoder; S2.2: The decoder accepts the feature map from the current layer of Hourglass-VIT and the output of the previous GFM layer as input to the current GFM layer. After integration according to certain rules, the GFM guides the high-resolution features and uses the output as input to the next GFM layer. Finally, after three stages of GFM processing, two outputs of half the size of the original image are obtained. The GFM process is represented as follows: Where Concat means concatenation by channel, CBR means convolution, batch normalization and ReLU activation function, and UP means upsampling by 2x; initially, F5, F6, F7, F8 represent the feature maps {x5, x6, x7, x8} of the same size in the middle of Hourglass-VIT respectively; or, F5, F6 represent the output of the previous GFM layer, and F7, F8 represent the feature maps of the current Hourglass-VIT layer; S2.3: The top-down and bottom-up processes are consistent, starting from small-scale features in the middle of the network and gradually merging large-scale, low-level features rich in detail downwards; S2.4: The 12 feature vectors output by the Hourglass-VIT backbone after feature fusion are mapped to the fused high-level feature F. h1 F h2 and low-level features F I1 F l2 High-level features and low-level features are fused separately to obtain the final fused features, which are then input into the feature interaction enhancement module. Wherein, Concat indicates concatenation by channel; CBR 1x1 This indicates a convolution with output channel C2 being half the size of input channel C1, a kernel size of 1, and batch normalization and ReLU activation function; Conv 3x3 F represents a convolution with a kernel size of 3, batch normalization, and ReLU activation function; h F l This represents the final output of the dual-path feature shrinking decoder.

3. The method for detecting camouflaged targets using an hourglass vision Transformer dual-channel encoder-decoder network according to claim 2, characterized in that: Step S3 specifically involves: S3.1: The high-level feature Fh is passed through two linear mapping functions with trainable weights. and The dimensionality-reduced feature sequences Q and K are obtained. The low-level features Fl are passed through a window multi-head self-attention module to obtain the local attention vector A, which strengthens the model's attention to local features. This process is represented as: S3.2: Generate a weight map for A using the softmax function and calculate its Hadamard product with K, then perform adaptive average pooling. The feature map V is obtained; it is represented as: S3.3: Apply matrix multiplication to K and V to explore the correlation between them, and use the softmax operation to generate a relevant attention map W. S3.4: After obtaining the relevant attention map W, multiply it with the feature map Q, and then input the resulting feature map into a convolutional layer GCN(-) to learn high-level semantic relationships between regions; calculate the inner product between GCN(-) and G and pass it through a linear mapping function. Map back to two-dimensional image features with the same dimensions as the original features, and then combine with feature F. h Combining these, we obtain the final output of FIEM: S3.5: For F h and F I Perform the above operations by swapping to obtain F2, F h Without going through the W-MSA module, F1 and F2 are further fused to obtain the final output: Where cat represents concatenation of channel dimensions, Conv 1x1 This indicates a convolution with a kernel size of 1, adjusted to change the number of channels, such that F... o F1, F2, and Conv all have the same number of channels. 3x3 This represents a convolution with a kernel size of 3.

4. The method for detecting camouflaged targets using an hourglass vision Transformer dual-channel encoder-decoder network according to claim 3, characterized in that: Step S4 specifically involves: S4.1: The shallow features with injected semantic information, the deep features with injected boundary information, and the fused features obtained from the high-low feature interaction enhancement module are fed into the Sigmoid segmentation head to generate a prediction map of the camouflaged target. Where CBR represents convolution, batch normalization, and ReLU activation function operations; Up2x represents 2x upsampling; Conv 1x1 This represents a convolution with C input channels, 1 output channel, and a kernel size of 1. S4.2: The network loss function consists of three parts: Where G represents the truth graph; L BCE This represents the binary cross-entropy loss; This represents the weighted binary cross-entropy loss; , , This represents the loss balance weight.

Images