Small target information extraction network based on Manhattan self-attention and information fusion
By introducing Manhattan self-attention and information fusion technology into the small-target information extraction network, the problem of low accuracy and recall of small-target detection in complex environments is solved, and efficient extraction of small-target information is achieved.
Patent Information
- Application Number
- CN202411775563.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-05
- Publication Date
- 2025-03-14
AI Technical Summary
In complex environments, the accuracy and recall rate of small target detection are low, and it is difficult for the prior art to effectively extract small target information.
A small-objective information extraction network based on Manhattan's self-attention and information fusion is designed. Through the Manhattan's self-attention layer and multi-layer information refinement module, the possibility of small targets being ignored is reduced and the interaction between small targets and surrounding targets is enhanced.
It effectively improves the detection accuracy and recall rate of small targets, ensuring the extraction effect of small target information in scenarios where large, medium and small targets exist simultaneously.
Smart Images
Figure CN119625498A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of deep learning and computer vision, and in particular to a small target information extraction network based on Manhattan self-attention and information fusion, which is improved based on a Deformable-DETR-based network. Background Art
[0002] In the field of object detection, small object detection has always been one of the most important challenges for SOTA (State-Of-The-Art) object detection models. In a complex environment with large, medium and small objects, small objects are easily identified as background, which leads to a decrease in the accuracy and recall of small object detection. Object detection researchers and practitioners have adopted many methods to try to solve the shortcomings of insufficient feature information of small objects in deep networks.
[0003] In 2017, TYLin et al. proposed the FPN (Feature Pyramid Network) network based on Faster R-CNN. Through top-down feature transfer and splicing structure, the multi-level features were utilized. The FPN network was combined with the Faster R-CNN network, which greatly improved the accuracy of Faster R-CNN. FPN uses multi-layer feature maps to reduce the information loss of small targets between multi-layer networks and enhance the detection ability of small targets. Due to the excellent multi-scale detection ability of FPN, it is widely used in YOLO and RCNN to enhance the detection ability of small targets. PAN (Path Aggregation Network) is a method for dealing with multi-scale problems proposed by Liu et al. in 2018. Unlike FPN, PAN is constructed by upsampling from low-resolution feature maps and downsampling from high-resolution feature maps, connecting them to form a path. In this process, the information of each layer of feature maps will be fused with the feature maps of the upper and lower adjacent layers, but unlike FPN, PAN will add and cascade the results of the fusion of feature maps at different levels. This can avoid the loss of information in the summing process, while retaining more detailed information, thereby improving detection accuracy.
[0004] Initially, DETR only used one layer of feature maps due to the huge self-attention overhead. Subsequently, Zhu et al. borrowed the idea of deformable convolutional networks and used a deformable attention module to replace the Transformer attention module in DETR to reduce the amount of calculation and improve the detection speed. To address the problem of poor small target detection, Deformable-DETR introduces multi-scale feature maps into the deformable attention module, performs deformable attention on feature maps of different scales, and adds the attention calculation results of each layer to obtain multi-scale attention, thereby improving the detection effect of small targets.
[0005] The small target information extraction network based on Manhattan self-attention and information fusion proposed in the present invention can effectively improve the accuracy and recall rate of small targets in scenarios where large, medium and small targets exist at the same time. Summary of the invention
[0006] The purpose of the present invention is to design a network for extracting small target information in complex environments. The decay matrix imposed by Manhattan self-attention can prevent the model from paying too much attention to the prominent targets in the image under the condition of self-attention long-distance modeling, resulting in ignoring the information of small targets. And considering that small targets still need to interact with surrounding targets to obtain overall information, the surrounding target detection decay coefficient is designed to strengthen the interaction between the retained small target information and other surrounding targets.
[0007] The present invention provides a small target information extraction network based on Manhattan self-attention and information fusion, which mainly includes a multi-layer feature aggregation module for aggregating multi-layer feature maps; an encoder based on deformable attention for extracting overall image features; an encoder based on Manhattan self-attention for extracting features of small and medium-sized targets; and a multi-layer information refinement module for performing information fusion and strengthening feature retention of small targets in the overall feature map.
[0008] The present invention is mainly divided into the following steps:
[0009] 1. The multi-layer feature aggregation module aggregates the top three feature maps output by the backbone network into feature map F m ;
[0010] 2. Feature map F output by the backbone network d After being encoded by a deformable attention-based encoder, the output
[0011] 3. Feature map F m After being encoded by the Manhattan self-attention encoder, the output feature map
[0012] 4. Multi-layer information refinement module, and Aggregation is performed to obtain a feature map that completes information fusion.
[0013] In the above steps, the feature maps output by the backbone network will be sent to the encoder based on deformable attention and the multi-layer feature aggregation module respectively. The multi-layer feature aggregation module will aggregate the top three feature maps, and the aggregated feature maps will be used as the input of the encoder based on Manhattan self-attention. The multi-layer information refinement module will obtain the outputs of the encoder based on deformable attention and the encoder based on Manhattan self-attention, perform information refinement operations, and complete information fusion.
[0014] In this implementation, the backbone network uses the ResNet50 backbone network, and the output feature maps of different layers of the backbone network are selected for the subsequent network to continue to extract object features. In this example, ResNet50 is used, and the four feature maps output by ResNet50 are used, which are marked from high to low as C1, C2, C3, and C4. C1, C2, C3, and C4 are flattened according to the length and width dimensions, and Concat is used to obtain F d Feature map.
[0015] Among them, C1, C2, and C3 are used as inputs of the multi-layer feature aggregation module. In the multi-layer feature aggregation module, C1 is upsampled to twice the length and width of the original C1 through a ConvTranspose2d with kernel_size=3, stride=2, and cropped or padded to the length and width of C2 to obtain C1 ′ C3 is downsampled to half the length and width of the original C3 through a Conv with kernel_size=3 and stride=2, and C3 is obtained. ′ . Set C1 ′ , C2, C3 ′ Flatten according to the length and width dimensions, add them up to get F m Feature map.
[0016] The encoder based on deformable attention includes deformable attention and feedforward layers. The image features output by the above backbone network will be stretched, concat, and input into the encoder based on deformable attention. Finally, the encoder based on deformable attention will output the overall features of the image. The formula is as follows:
[0017]
[0018] Among them, z q represents the query feature, p q represents the query position, x represents the input feature map, M represents the number of attention heads, which is 8 in this example, and K represents the number of sampling points. m A represents the weight matrix. mqk is the sampling point weight, Δpmqk is the relative offset, get p q The sample position after offset.
[0019] The Manhattan self-attention based encoder has a Manhattan self-attention layer and a feed-forward network. The input of the Manhattan self-attention based encoder is labeled F m In the Manhattan self-attention layer, F m It will pass through three different projection layers to obtain Q, K, and V. Q, K, and V will be input into the Manhattan self-attention layer for Manhattan self-attention calculation. The formula of the Manhattan self-attention layer is as follows:
[0020]
[0021] Where Q, K, V are the input F of the Manhattan self-attention encoder m Obtained through the projection layer. Is the decay matrix representing the i-th layer, and the specific generation method is as follows.
[0022] By F m The network generates indices from zero to width or height. The two-dimensional coordinate network is generated through these two indices, and the coordinate difference between pixels, namely Manhattan distance, is calculated to generate a two-dimensional matrix representing each pair of pixels. After multiplying the two-dimensional matrix by the decay rate, D is obtained. 2d ; Based on the original Manhattan self-attention, the decay coefficient of surrounding target detection is added. The specific method is that when Manhattan self-attention is applied multiple times, the distance decay mask will reduce the decay effect after each layer of Manhattan self-attention. The specific implementation is to process the decay matrix as follows in the i-th layer of Manhattan self-attention based encoder:
[0023]
[0024] Where D 2d represents the generated decay matrix, layer_decay is the surrounding object detection decay coefficient, and i represents the current encoder layer index. The generated MHSA (F m ) will be with F m After a residual connection, it is input into the feedforward layer and finally obtained
[0025] The multi-layer information refinement module has a Conv2d with kernel_size=3, stride=2 and a ConvTranspose2d with kernel_size=3, stride=2 for up and down sampling respectively. and Will Through Conv2d and ConvTranspose2d respectively, we get The model will Perform Concat to obtain features Will Divide and obtain and in Indicates the three special reconnaissance maps at the top level. Indicates the additional feature maps except the top three feature maps. and Concat along the number of channels, then project back to the previous number of channels through FFN to obtain the feature map F f .Will and F f Perform Concat to obtain the fused feature map.
[0026] Due to the adoption of the above technical solution, the present invention has the following advantages:
[0027] 1. Manhattan self-attention is used to extract information from images, which avoids small targets being treated as backgrounds and improves the accuracy and recall of small targets.
[0028] 2. At the same time, different decay rates are applied to different heads in Manhattan self-attention, ensuring that the accuracy and recall of salient objects in the image are not affected. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:
[0030] Figure 1 It is a schematic diagram of a small target information extraction network based on Manhattan self-attention and information fusion of the present invention.
[0031] Figure 2 Schematic diagram of the Manhattan self-attention based encoder of the present invention.
[0032] Figure 3 It is a schematic diagram of the multi-layer feature aggregation module of the present invention.
[0033] Figure 4 It is a schematic diagram of the multi-layer information refinement module of the present invention.
[0034] Figure 5 It is a network schematic diagram of a specific implementation scheme of a small target information extraction network based on Manhattan self-attention and information fusion of the present invention.
[0035] Figure 6 It is a decaying visualization graph of Manhattan self-attention of the present invention. Specific implementation plan
[0036] The present invention is further described in detail below in conjunction with specific implementations, which are intended to explain the present invention rather than to limit it.
[0037] The present invention proposes a small target extraction network based on Manhattan self-attention and information fusion, such as Figure 1 As shown, it specifically includes an encoder based on deformable attention, an encoder based on Manhattan self-attention, a multi-layer feature aggregation module, and a multi-layer information refinement module.
[0038] Specific implementation method:
[0039] Obtain a dataset of complex environments, which requires objects of different sizes. The MS COCO dataset is used in the specific implementation method.
[0040] like Figure 5 A detection network based on a small target information extraction network based on Manhattan self-attention and information fusion is constructed. The constructed detection network includes a backbone network, an encoder based on deformable attention, and Figure 2 The constructed encoder based on Manhattan self-attention, multi-layer feature aggregation module, multi-layer information refinement module, and decoder based on deformable attention are as follows:
[0041] Step 1: Build the backbone network ResNet50, input a 3-channel image, extract image features through the backbone network, and obtain 4 feature maps output by the backbone network, labeled C1, C2, C3, and C4 from high to low. Flatten C1, C2, C3, and C4 according to the length and width dimensions, and concat to obtain the feature map F d .
[0042] Step 2: If Figure 3 Construct a multi-layer feature aggregation module. Use the backbone network outputs C1, C2, and C3 as the input of the multi-layer feature aggregation module. In the multi-layer feature aggregation module, C1 is upsampled to twice the length and width of the original C1 through a ConvTranspose2d with kernel_size=3, stride=2. If the feature map after sampling is different from the size of C2, the model will crop the feature map to the same length and width as C2 to obtain C1. ′ C3 is downsampled to half the length and width of C3 through a Conv with kernel_size=3 and stride=2 to obtain C′3. Add C′1, C2, and C′3, flatten them according to the length and width dimensions, and concat to obtain the feature map F m .
[0043] Step 3: Construct a deformable attention-based encoder including deformable attention and feedforward layers. The image features output by the above backbone network will be stretched, concat, and input into the deformable attention-based encoder. Input feature F d First, after deformable attention, the overall feature extraction is performed, and then a FFN is passed through, and this step is repeated 6 times to obtain
[0044] Step 4: Figure 2 As shown, an encoder based on Manhattan self-attention is constructed. The encoder based on Manhattan self-attention has a Manhattan self-attention layer and a feedforward network. The input feature of the encoder based on Manhattan self-attention is marked as F m In the Manhattan self-attention layer, F m It will pass through three different projection layers to obtain Q, K, and V. Q, K, and V will be input into the Manhattan self-attention layer for Manhattan self-attention calculation. The formula of the Manhattan self-attention layer is as follows:
[0045]
[0046] Where Q, K, V are the input F of the Manhattan self-attention encoder m Obtained through the projection layer. Is the decay matrix representing the i-th layer, and the specific generation method is as follows:
[0047] By F m The network generates indices from zero to width or height. The two-dimensional coordinate network is generated through these two indices, and the coordinate difference between pixels, namely Manhattan distance, is calculated to generate a two-dimensional matrix representing each pair of pixels. After multiplying the two-dimensional matrix by the decay rate, D is obtained. 2d ; Based on the original Manhattan self-attention, the decay coefficient of surrounding target detection is added. The specific method is that when Manhattan self-attention is applied multiple times, the distance decay mask will reduce the decay effect after each layer of Manhattan self-attention. The specific implementation is to process the decay matrix as follows in the i-th layer of Manhattan self-attention based encoder:
[0048]
[0049] Where D 2d represents the generated decay matrix, layer_decay is the surrounding target detection decay coefficient, which is set to 1 in the specific implementation scheme, and i represents the current encoder number. The generated MHSA (F m ) will be with F m After a residual connection, the input is fed into the feedforward layer, and this step is repeated 6 times to finally obtain Manhattan self-attention visualization Figure 6 shown.
[0050] Step 5: Figure 4 As shown in Figure 2, a multi-layer information refinement module is constructed. and Will Through Conv2d and ConvTranspose2d respectively, we get The model will Perform Concat to obtain features Will Divide and obtain and in Represents the top three special reconnaissance maps, dimensions and same. Indicates the additional feature maps except the top three feature maps. and Concat along the number of channels, then project back to the previous number of channels through FFN to obtain the feature map F f .Will and F f Perform Concat to obtain the fused feature map.
[0051] Step 6: Build a decoder. The fused feature map is used as the decoder input, and the decoder maintains multiple queries. After a self-attention, the query will query the image features in the feature map output in step 5 through cross attention, and output the prediction of the objects in the image.
Claims
1. A small target information extraction network based on Manhattan self-attention and information fusion, characterized by The constructed small target information extraction network includes: an encoder based on deformable attention, an encoder based on Manhattan self-attention, a multi-layer feature aggregation module, and a multi-layer information refinement module; first, the n-layer feature map output by the backbone network; in the n-layer feature map, the three feature maps output by the last three layers of the backbone network are marked as C1, C2, and C3 from high to low; C1, C2, and C3 are used as the input of the multi-layer feature aggregation module, and the feature map F is output. m ; The original n-layer feature map is Flattened and Concat to obtain the feature map F d ; The feature map F m As the input of the Manhattan self-attention based encoder, the output is marked as a feature map F d The feature map is used as the input of the deformable attention-based encoder, and the output is marked as Will and Input into the multi-layer information refinement module to obtain the feature map that completes the information fusion; The multi-layer feature aggregation module is characterized by having a conv2d with kernel_size=3 and stride=2 and a ConvTranspose2d with kernel_size=3 and stride=2 for upsampling and downsampling respectively; the specific steps are as follows: C3 is downsampled by a Conv2d to obtain C′3, and C1 is upsampled by ConvTranspose2d to obtain C′1; C′1, C2, and C′3 are flattened according to the length and width dimensions, added, and normalized to obtain the feature map F m ; The multi-layer information refinement module is characterized by having a Conv2d with kernel_size=3, stride=2 and a ConvTranspose2d with kernel_size=3, stride=2 for up and down sampling respectively; and Will After the Resize operation, we pass Conv2d and ConvTranspose2d to obtain The model will Perform Flatten and Concat to obtain features Will Divide and obtain and in Indicates the three special reconnaissance maps at the top level. represents the additional feature maps except the top three feature maps; and Concat along the number of channels, then project back to the previous number of channels through FFN to obtain the feature map F f ;Will and F f Concat is performed to obtain the feature map that finally completes the information fusion.
2. The Manhattan self-attention based encoder according to claim 1, characterized in that The decay matrix is added to the self-attention in the encoder to become Manhattan self-attention, and the decay coefficient of surrounding object detection is added on this basis; the specific method is that when Manhattan self-attention is applied multiple times, the distance decay mask will reduce the decay effect after each layer of Manhattan self-attention; the specific implementation is to process the decay matrix as follows in the i-th layer of the Manhattan self-attention based encoder: Where D 2d Represents the generated decay matrix, layer_decay is the surrounding target detection decay coefficient, and i represents the current encoder layer number.