Multi-scale feature fusion method for gradually fusing features from top to bottom

Abstract

The invention discloses a multi-scale feature fusion method for progressive feature fusion from top to bottom, and relates to the technical field of target detection in computer vision. The method comprises the following steps: (1) gradually fusing a multi-scale fusion structure of features from top to bottom; (2) an adaptive space fusion method with an extrusion weight layer; (3) an extrusion weight module used for generating feature weights; according to the invention, a deep learning method based on a prediction model is adopted, targets with similar shapes but not the same type can be effectively detected, and the distinguishing effect of target detection is improved.

Description

Technical Field

[0001] The present invention relates to a multi-scale feature fusion method with top-down progressive feature fusion, belonging to the technical field of object detection in computer vision. Background Art

[0002] Multi-scale feature fusion is a key technology in object detection algorithms in the field of computer vision, aiming to effectively improve the detection performance of the model for objects of different scales. With the increase in object scale and complexity, it is difficult for single-scale feature extraction to fully capture the diverse information of objects. Therefore, multi-scale feature fusion has become an important means to address this challenge.

[0003] First, multi-scale feature fusion is based on extracting information at different levels of an image. Low-level features usually contain detailed information and are suitable for detecting small-sized objects, while high-level features focus more on the global context and are applicable to large-sized objects. By introducing multi-scale features, the algorithm can take into account both local and global information simultaneously, improving the sensitivity to objects of different scales. Second, the methods adopted in multi-scale feature fusion include pyramid structures, multi-branch networks, and feature pyramid networks, etc. The pyramid structure realizes hierarchical fusion of multi-scale information by constructing feature pyramids with different resolutions; the multi-branch network designs multiple parallel network branches, with each branch focusing on feature extraction at different scales; the feature pyramid network realizes cross-level interaction and fusion of features through top-down or bottom-up information transfer. The multi-scale feature fusion technology improves the adaptability of object detection algorithms to objects of diverse scales by integrating information at different scales, providing a more powerful solution for object detection in computer vision tasks.

[0004] Although multi-scale feature fusion has made significant progress in object detection, there are still some potential problems: (1) Information loss and blur: Feature fusion may lead to the mixing and loss of information, especially in the pyramid structure. The features transmitted in the hierarchical structure may be smoothed or blurred, making it difficult for the model to maintain details when dealing with complex scenes or objects with similar features, resulting in misdetection. (2) The trade-off problem of different scales: When fusing multi-scale features, features of different scales may have different contributions to the final result. How to effectively balance the weights of each scale remains a challenge. Summary of the Invention

[0005] In order to solve the problem of loss of detailed information caused by the existing multi-scale feature fusion method in cross-layer information fusion, the present invention proposes a multi-scale feature fusion method with top-down progressive feature fusion, aiming to design a network structure with top-down progressive feature fusion, so that feature information can be fused across layers while reducing the loss of feature information of low-level features, reducing the misdetection rate of similar objects, and improving the detection accuracy of similar objects.

[0006] To achieve the above-mentioned invention object, the present invention adopts the following technical solutions:

[0007] (1) Select image features of different scales from the feature extraction network in stages. Taking the selection of four features as an example, namely {C2, C3, C4, C5};

[0008] (2) First, fuse the features of {C5, C4} through the proposed ASFF-SW to generate features of two scales, namely {C 51 , C 41}; Then, fuse the features of {C 51 , C 41 , C3} through the proposed ASFF-SW to generate features of three scales, namely {C 51 , C 41 , C 31}; Then, fuse the features of {C 51 , C 41 , C 31 , C2} through the proposed ASFF-SW to generate {C 51 , C 41 , C 31 , C 21}; Finally, through continuous learning of convolution and downsampling operation on C 51 , multi-scale features {P2, P3, P4, P5, P6} for predicting object categories and regressing object positions are obtained;

[0009] (3) Finally, use the generated multi-scale features to perform regression prediction of object categories and positions.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0011] 1. The present invention starts fusing from high-level features and gradually incorporates low-level features, reducing the number of operations on low-level features and greatly reducing the information loss of low-level features;

[0012] 2. The present invention improves the weight generation method in the adaptive fusion method, enabling the fusion weights to make full use of context information, enhancing the fusion effect of adaptive fusion, and improving the robustness of the network;

[0013] 3. The present invention retains sufficient detailed information in the image features of each feature scale, achieving the best effect in distinguishing similar objects, effectively improving the effect of similar object detection, and reducing the false detection rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flowchart of the multi-scale feature fusion method for top-down progressive feature fusion of the present invention;

[0015] Figure 2 This is the network structure diagram of the improved adaptive feature fusion method of the present invention. This structure diagram takes the feature fusion of three levels as an example;

[0016] Figure 3 This is the network structure diagram for generating the feature weight layer in the adaptive feature fusion method of the present invention;

[0017] Figure 4 This is the superiority display diagram of the present invention in the detection results. Detailed implementation manners

[0018] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific implementation manners described herein are only used to explain the present invention and are not used to limit the present invention.

[0019] As Figures 1-3 , the multi-scale feature fusion method of progressive feature fusion from top to bottom includes the following steps:

[0020] (1) Select image features of different scales from the feature extraction network in stages. Taking the selection of four features as an example, namely {C2, C3, C4, C5};

[0021] (2) First, fuse {C5, C4} through the proposed ASFF-SW to generate features of two scales {C 51 , C 41}; then fuse {C 51 , C 41 , C3} through the proposed ASFF-SW to generate features of three scales {C 51 , C 41 , C 31}; then fuse {C 51 , C 41 , C 31 , C2} through the proposed ASFF-SW to generate {C 51 , C 41 , C 31 , C 21}; finally, through continuous learning of convolution and downsampling operation on C 51 , multi-scale features {P2, P3, P4, P5, P6} for predicting the object category and regressing the object position are obtained;

[0022] (3) Finally, use the generated multi-scale features to perform regression prediction on the object category and position. The detailed steps are as follows:

[0023] (1) Extract three or four scales of multi-scale features {C2, C3, C4, C5} of the image from the backbone network;

[0024] (2) Then, we first fuse the high-level features C5 and C4 through the improved adaptive feature fusion method to generate two new fused features {C 51 , C 41};

[0025] (3) Then, the improved adaptive feature fusion method is also used to generate the new C 51 , C 41 The C3 features extracted from the backbone network are fused to obtain three new fusion features {C 52 , C 42 , C 32};

[0026] (4) Then, the newly generated C is transformed into 52 , C 42 , C 32 And the C2 features extracted from the backbone network are fused to obtain four new fusion features {C 53 , C 43 , C 33 , C 23};

[0027] (5) The fusion features generated after each fusion continue to perform residual learning. The four fusion features {C 53 , C 43 , C 33 , C 23} After the residual learning unit, the features {P2, P3, P4, P5} are obtained. Subsequently, convolution is used to reduce the dimension of P5 to create P6, resulting in multi-scale features {P2, P3, P4, P5, P6} with a feature span of {4, 8, 16, 32, 64};

[0028] (6) Finally, the generated multi-scale features are used to perform regression prediction of target category and location.

[0029] For the ASFF-SW method, as shown in the attached Figure 2 As shown, the present invention takes three-level feature fusion as an example. ij n ->l represents the feature vector from level n to level l at (i, j). The feature vector of the fusion result is y ij l , which is achieved by the ASFF-SW proposed in this patent, and the feature vector y of the fusion result ij lCan be represented as x ij 1->l , x ij 2->l and x ij 3->l The linear combination of:

[0030] y ij l = w ij 1->l ·x ij 1->l + w ij 2->l ·x ij 2->l + w ij 3->l ·x ij 3->l (1)

[0031] Where w ij 1->l , w ij 2->l and w ij 3->l Are obtained by the squeezing weight module and represent the three levels of adaptive spatial squeezing weights at the l-level scale. And, w ij 1->l , w ij 2->l and w ij 3->l Satisfy the following relationship:

[0032] w ij 1->l + w ij 2->l + w ij 3->l = 1 (2)

[0033] The squeezing weight module is an improved method for adaptive spatial feature fusion proposed in this invention patent, called the squeezing weight module (SWM), aiming to improve the integration of global context information in feature fusion while maintaining computational efficiency. As shown in the appendix Figure 3 Shown, SWM consists of two parts: scale squeezing and channel squeezing. Scale squeezing completely compresses the scale information through two parallel 1x1 convolutions. At the same time, channel squeezing compresses the channel information and scale information from the original features into a 1xHxW weighted layer. This process can be expressed as:

[0034]

[0035] Where Is the input feature map; N pIndicates the number of positions in the feature map (i.e., N p = h × W; i is the index of the query position; j enumerates all possible positions; Ww, Wv, and Wk represent linear transformation matrices (e.g., 1x1 convolution). It should be noted that the reshape operation is not explicitly shown in this formula. After each 1x1 convolution, this invention patent embeds a reshape operation to dynamically adjust the dimension of the feature.

[0036] To verify the effectiveness of the method of this invention, the most authoritative dataset MSCOCO2017 in the field of object detection is used for training and testing. Ten recent deep learning-based feature fusion methods are selected as comparison methods, specifically:

[0037] Method 1: The method proposed by T.-Y. Lin et al., reference "T.-Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, ′S. Belongie, Feature pyramid networks for object detection, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2117–2125."

[0038] Method 2: The method proposed by S. Liu et al., reference "S. Liu, L. Qi, H. Qin, J. Shi, J. Jia, Path aggregation network for instance segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8759–8768."

[0039] Method 3: The method proposed by C. Guo et al., reference "C. Guo, B. Fan, Q. Zhang, S. Xiang, C. Pan, Augfpn: Improving multi-scale feature learning for object detection, in: Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2020, pp. 12595–12604."

[0040] Method 4: The method proposed by D. Zhang et al., reference "D. Zhang, H. Zhang, J. Tang, M. Wang, X. Hua, Q. Sun, Feature pyramid transformer, in: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVIII 16, Springer, 2020, pp. 323–339."

[0041] Method 5: The method proposed by J. Ma et al., reference "J. Ma, B. Chen, Dual refinement feature pyramid networks for object detection, arXiv preprint arXiv:2012.01733 (2020)."

[0042] Method 6: The method proposed by G. Zhao et al., reference "G. Zhao, W. Ge, Y. Yu, Graph fpn: Graph feature pyramid network for object detection, in: Proceedings of the IEEE / CVF international conference on computer vision, 2021, pp. 2763–2772."

[0043] Method 7: The method proposed by S. Huang et al., reference "S. Huang, Z. Lu, R. Cheng, C. He, Fapn: Feature-aligned pyramid network for dense image prediction, in: Proceedings of the IEEE / CVF international conference on computer vision, 2021, pp. 864–873."

[0044] Method 8: The method proposed by J. Xie et al., reference "J. Xie, Y. Pang, J. Nie, J. Cao, J. Han, Latent feature pyramid network for object detection, IEEE Transactions on Multimedia (2022)."

[0045] Method 9: The method proposed by L. Zhu et al., reference "L. Zhu, F. Lee, J. Cai, H. Yu, Q. Chen, An improved feature pyramid network for object detection, Neurocomputing 483 (2022) 127–139."

[0046] Method 10: The method proposed by G. Yang et al., reference "G. Yang, J. Lei, Z. Zhu, S. Cheng, Z. Feng, R. Liang, Afpn: Asymptotic feature pyramid network for object detection (2023). arXiv:2306.15988."

[0047] As shown in Table 1, when the method proposed in the present invention uses mAP as the evaluation index on the COCO2017 val dataset and is compared with other methods, the detection effect of this method has great advantages. As shown Figure 4 below, the detection effect of this method also has great advantages in the ability to distinguish objects with similar shapes.

[0048] Table 1 Comparison of evaluation indexes (mAP) with other methods

[0049]

[0050]

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-scale feature fusion method for top-down progressive feature fusion, characterized in that The steps include the following: (1) Select image features of different scales from the feature extraction network. Taking the selection of four features as an example, namely {C2, C3, C4, C5}; (2) According to the proposed progressive fusion structure of top-down special fusion, first fuse {C5, C4} through the proposed Adaptive Spatial Feature Fusion with Squeeze Weight Module (ASFF-SW) to generate features of two scales {C51, C41}; then fuse {C51, C41, C3} through the proposed ASFF-SW to generate features of three scales {C51, C41, C31}; then fuse {C51, C41, C31, C2} through the proposed ASFF-SW to generate {C51, C41, C31, C21}; finally, through continuous learning of convolution and downsampling operation on C51, multi-scale features {P2, P3, P4, P5, P6} for predicting object categories and regressing object positions are obtained; (3) Finally, use the generated multi-scale features for regression prediction of object categories and positions.

2. The multi-scale feature fusion method for top-down progressive feature fusion according to claim 1, wherein In the step (2), the progressive fusion structure of top-down feature fusion is as follows: First, {C5, C4} are subjected to feature fusion through the proposed ASFF-SW to generate features of two scales {C 51 , C 41}; then, {C 51 , C 41 , C3} are subjected to feature fusion through the proposed ASFF-SW to generate features of three scales {C 51 , C 41 , C 31}; then, {C 51 , C 41 , C 31 , C2} are subjected to feature fusion through the proposed ASFF-SW to generate {C 51 , C 41 , C 31 , C 21}; finally, through continuous learning by convolution and downsampling operation on C 51 , multi-scale features {P2, P3, P4, P5, P6} for predicting object categories and regressing object positions are obtained.

3. A multi-scale feature fusion method with top-down progressive feature fusion according to claim 1, characterized in that The ASFF-SW used in the step (2) improves the generation method of weights in the adaptive spatial fusion; let x ij n-＞l represent the feature vectors from the n-th level to the 1st level at (i, j), and the feature vector of the fusion result is y ij l , which is achieved by the ASFF-SW proposed by us, and the feature vector y ij l of the fusion result can be expressed as a linear combination of x ij 1-＞l , x ij 2-＞l and x ij 3-＞l : y ij l = w ij 1-＞l · x ij 1-＞l + w ij 2-＞l · x ij 2-＞l + w ij 3-＞l · x ij 3-＞l (1) where w ij 1-＞l , w ij 2-＞l and w ij 3-＞l are obtained by the squeezing weight module and represent three levels of adaptive spatial squeezing weights at the l - level scale; and, w ij 1-＞l , w ij 2-＞l and w ij 3-＞l satisfy the following relational expression: w ij 1-＞l +w ij 2-＞l +w ij 3-＞l = 1 (2).

4. A multi-scale feature fusion method with top-down progressive feature fusion according to claim 1, wherein the squeeze weight module (SWM) in the ASFF-SW method used in step (2) is characterized in that: The SWM consists of two parts: scale squeeze and channel squeeze; scale squeeze completely compresses the scale information through two parallel 1x1 convolutions; at the same time, channel squeeze compresses the channel information and scale information from the original features into a 1xHxW weighted layer; this process can be expressed as: wherein is the input feature map; N p represents the number of positions in the feature map (i.e., N p = H×W; i is the index of the query position; j enumerates all possible positions; Ww, Wv, and Wk represent linear transformation matrices (e.g., 1x1 convolution); it should be noted that the reshape operation is not explicitly shown in this formula; after each 1x1 convolution, this invention patent embeds a reshape operation to dynamically adjust the dimension of the feature.

Images