A multi-scale fusion feature extraction method and system for target detection

Abstract

The present invention discloses a multi-scale fusion feature extraction method and system for target detection, wherein the network structure comprises: 1) backbone network feature extraction; 2) feature pyramid reconstruction; 3) feature fusion; 4) reconstruction of bottom-up feature pyramid; 5) adding deep feature enhancement network; 6) prediction; the method of the present invention uses convolution kernels of different sizes to extract features from the backbone network, and reconstructs a feature pyramid network with different receptive fields, so that the detection network can learn the feature representation of instances of the same category at different scales during the training process.

Description

Technical field:

[0001] The present invention belongs to the field of computer science and artificial intelligence, and specifically proposes a multi-scale fusion feature extraction method and system for target detection, which are used to extract multi-scale features in images. Background technology:

[0002] In target detection tasks, since the size of instances is often uncertain, it is difficult for the network to learn a unified feature representation for objects of different scales in the same category through training. This greatly increases the difficulty of target detection and makes multi-scale prediction one of the main challenges of target detection tasks.

[0003] In recent years, with the continuous development of deep learning, researchers have contributed many excellent solutions to multi-scale prediction, among which the feature pyramid structure and its variants have received widespread attention. The starting point of the feature pyramid network is often to promote the fusion of feature maps of different scales and the flow of information by building lateral connections and top-down path enhancement. However, these network structures often ignore the importance of receptive field in network feature expression. Summary of the invention:

[0004] In order to solve the problem that the existing methods ignore the importance of receptive field in network feature expression, the present invention proposes a multi-scale fusion feature extraction method and system for target detection, and its technical solution is as follows:

[0005] A multi-scale fusion feature extraction method for target detection includes the following specific steps:

[0006] Step 1: Take out the intermediate layer feature map [P2, P3, P4, P5] of each block in the backbone network, and then use convolution kernel size 3x3, 1x1 convolution and convolution kernel size 3x3, dilation rate 2 and 3 respectively to extract features from the intermediate layer feature map [P2, P3, P4, P5] of the backbone network, and get the new feature map [Q i2 ,Q i3 ,Q i4 ,Q i5 ], i = 1, 2, 3, 4, indicating that four different convolution kernels are used to extract feature maps with different receptive fields;

[0007] Step 2: Based on the new feature map [Q i2 ,Q i3 ,Q i4 ,Q i5 ] Construct four feature pyramid networks, and the features of the four feature pyramid networks are expressed as [N i2 ,N i3 ,N i4 ,N i5];

[0008] Step 3: [N i2 ,N i3 ,N i4 ,N i5 ] Perform feature fusion on each layer:

[0009]

[0010] Here, the value of n is 4, which means that a new fused feature pyramid is reconstructed on feature maps of four different scales. Here, j = 2, 3, 4, 5, which means that features are fused on different pyramid levels respectively. The final feature map is [M2, M3, M4, M5];

[0011] Step 4: Extract features through convolution operation with a convolution kernel size of 3x3 to construct lateral connections, downsample the feature map M2 and fuse it with the lateral connection feature maps of the corresponding scale to construct a bottom-up feature pyramid [O2, O3, O4, O5];

[0012] Step 5: Add the deep feature enhancement network DFEN to the underlying feature map P2 of the backbone network and the underlying feature map M2 of the fusion feature pyramid network. The structures of the two are exactly the same.

[0013] N i5 '=N i5 +DFEN1(P2)

[0014] O i5 '=O i5 +DFEN2(M2);

[0015] Step 6: Use 3x3 convolution to extract features from the feature pyramid feature map [O2, O3, O4, O5] obtained in step 4.

[0016] Furthermore, in step 2, the construction process of the four feature pyramid networks is as follows: upsampling is performed from the bottom-level features of the backbone network and respectively associated with the feature map [Q 12 ,Q 13 ,Q 14 ,Q 15 ] fusion, build the first top-down feature pyramid network; start upsampling from the bottom layer of the backbone network and respectively combine it with the feature map [Q 22 ,Q 23 ,Q 24 ,Q 25 ] fusion, build a second top-down feature pyramid network; start upsampling from the bottom-level features of the backbone network and combine them with the feature map [Q 32 ,Q 33 ,Q34 ,Q 35 ] fusion, build a third top-down feature pyramid network; start upsampling from the bottom-level features of the backbone network and combine them with the feature map [Q 42 ,Q 43 ,Q 44 ,Q 45 ] are fused to construct the fourth top-down feature pyramid network.

[0017] Accordingly, a multi-scale fusion feature extraction system for target detection includes:

[0018] Backbone network feature extraction module: It is used to extract the intermediate layer feature map [P2, P3, P4, P5] of each block in the backbone network, and then use convolution kernel size 3x3, 1x1 convolution and convolution kernel size 3x3, dilation rate 2 and 3 respectively to extract features from the intermediate layer feature map [P2, P3, P4, P5] of the backbone network, and obtain new feature maps [Q i2 ,Q i3 ,Q i4 ,Q i5 ], i = 1, 2, 3, 4, indicating that four different convolution kernels are used to extract feature maps with different receptive fields;

[0019] Feature pyramid reconstruction module: used to reconstruct the feature map based on the new feature map [Q i2 ,Q i3 ,Q i4 ,Q i5 ] Construct four feature pyramid networks, and the features of the four feature pyramid networks are expressed as [N i2 ,N i3 ,N i4 ,N i5 ];

[0020] Feature fusion module: used to transform [N i2 ,N i3 ,N i4 ,N i5 ] Perform feature fusion on each layer:

[0021]

[0022] Here, the value of n is 4, which means that a new fused feature pyramid is reconstructed on feature maps of four different scales. Here, j = 2, 3, 4, 5, which means that features are fused on different pyramid levels respectively. The final feature map is [M2, M3, M4, M5];

[0023] Reconstruct bottom-up feature pyramid module: It is used to extract features through convolution operations with a convolution kernel size of 3x3 to construct lateral connections, downsample the feature map M2 and fuse it with the lateral connection feature maps of the corresponding scale to construct a bottom-up feature pyramid [O2, O3, O4, O5];

[0024] Add deep feature enhancement network module: It is used to add deep feature enhancement network DFEN to the underlying feature map P2 of the backbone network and the underlying feature map M2 of the fusion feature pyramid network. The structures of the two are exactly the same.

[0025] N i5 '=N i5 +DFEN1(P2)

[0026] O i5 '=O i5 +DFEN2(M2);

[0027] Feature extraction module: used to extract features from the feature pyramid feature map [O2, O3, O4, O5] obtained in step 4 using 3x3 convolution.

[0028] Furthermore, the feature pyramid reconstruction module and the construction process of the four feature pyramid networks are as follows: upsampling is performed from the bottom layer features of the backbone network and respectively associated with the feature map [Q 12 ,Q 13 ,Q 14 ,Q 15 ] fusion, build the first top-down feature pyramid network; start upsampling from the bottom layer of the backbone network and respectively combine it with the feature map [Q 22 ,Q 23 ,Q 24 ,Q 25 ] fusion, build a second top-down feature pyramid network; start upsampling from the bottom-level features of the backbone network and combine them with the feature map [Q 32 ,Q 33 ,Q 34 ,Q 35 ] fusion, build a third top-down feature pyramid network; start upsampling from the bottom-level features of the backbone network and combine them with the feature map [Q 42 ,Q 43 ,Q 44 ,Q 45 ] are fused to construct the fourth top-down feature pyramid network.

[0029] Accordingly, a computer-readable storage medium stores one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any of the above methods.

[0030] Accordingly, a computing device includes:

[0031] One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the above methods.

[0032] Compared with the prior art, the present invention has the following beneficial effects

[0033] The multi-scale fusion feature pyramid network structure proposed in the present invention uses convolution kernels of different sizes to extract features from the backbone network and reconstruct a feature pyramid network with different receptive fields, so that the detection network can learn the feature representation of instances of different scales of the same category during the training process. Description of the drawings:

[0034] Figure 1 This is the structural diagram of this method;

[0035] Figure 2 This is the structure diagram of the deep feature enhancement network DFEN;

[0036] Figure 3 To visualize the detection results of the method of the present invention in the CascadeR-CNN target detection framework; Specific implementation method:

[0037] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0038] Embodiment 1:

[0039] like Figure 1 As shown, the method of the present invention comprises the following specific steps:

[0040] 1) Backbone network feature extraction: Extract features from the backbone network in the following manner, the method comprising:

[0041] First, the middle layer feature map [P2, P3, P4, P5] of each block of the backbone network (ResNet) is taken out, and then the convolution layers with convolution kernel size of 3x3, 1x1 and 3x3 and expansion rates of 2 and 3 are used to extract features from the middle layer feature map [P2, P3, P4, P5] of the backbone network respectively ( Figure 1 The MLF in the visual network structure only shows the process of reconstructing the feature pyramid using two different convolutions), and the new feature map [Q i2 ,Qi3 ,Q i4 ,Q i5 ], where i is 4, indicating that four different convolution kernels are used to extract feature maps with different receptive fields.

[0042] 2) Feature pyramid reconstruction: For the obtained feature map [Q i2 ,Q i3 ,Q i4 ,Q i5 ], through top-down upsampling operations and lateral connections to reconstruct feature pyramid networks with different receptive fields, the specific steps are as follows:

[0043] First, the feature [Q 12 ,Q 13 ,Q 14 ,Q 15 ] is taken out, and up-sampled from the bottom layer features of the backbone network and respectively compared with the feature map [Q 12 ,Q 13 ,Q 14 ,Q 15 ] to build the first top-down feature pyramid network. Similarly, the bottom-level features of the backbone network are upsampled and combined with the feature map [Q 22 ,Q 23 ,Q 24 ,Q 25 ] fusion, construct the second top-down feature pyramid network, and the construction method of the third and fourth feature pyramid networks is the same as above. The features of the four feature pyramid networks can be expressed as [N 12 ,N 13 ,N 14 ,N 15 ],[N 22 ,N 23 ,N 24 ,N 25 ],[N 32 ,N 33 ,N 34 ,N 35 ],[N 42 ,N 43 ,N 44 ,N 45 ], they can be expressed as [N i2 ,N i3 ,N i4 ,N i5 ], where i=1,2,3,4.

[0044] 3) Feature fusion: The obtained [N i2 ,N i3 ,N i4 ,Ni5 ] The feature map is fused on each layer. The specific operations are as follows:

[0045]

[0046] Here, n is 4, which means that we reconstruct the new fused feature pyramid on feature maps of four different scales. i is 1, 2, 3, 4, which means that we use four different sizes of convolution kernels to reconstruct the feature pyramid. Here, j = 2, 3, 4, 5, which means that the features of different pyramid levels are fused respectively.

[0047] 4) Reconstruct the bottom-up feature pyramid: For the feature maps [M2, M3, M4, M5] obtained in step 3), extract features through convolution operations with a convolution kernel size of 3x3 to construct lateral connections, downsample the feature map M2 and fuse it with the lateral connection feature maps of the corresponding scale to construct a bottom-up feature pyramid [O2, O3, O4, O5].

[0048] 5) Adding a deep feature enhancement network: Based on the network we built, we add a deep feature enhancement network (network structure such as Figure 2 As shown in Figure 2, this promotes the information flow between the shallow feature maps of the network and the deep feature maps of the network. Our deep feature enhancement network is named DFEN, which is essentially a downsampling network composed of deformable convolutions. Specifically:

[0049] N i5 '=N i5 +DFEN1(P2)

[0050] O i5 '=O i5 +DFEN2(M2);

[0051] We add a deep feature enhancement network DFEN to the underlying feature map P2 of the backbone network and the underlying feature map M2 of the fused feature pyramid network, respectively. The structures of the two are exactly the same.

[0052] 6) Prediction: Use 3x3 convolution to extract features from the feature pyramid feature map [O2, O3, O4, O5] obtained in step 4 for network prediction.

[0053] Application Example 1:

[0054] This application example adopts a new multi-scale fusion feature pyramid network structure in Example 1 and applies it to our own object detection dataset and the public MSCOCO object detection dataset to verify its effectiveness. The experimental results are shown in Table 1 and Table 2. The results of our method and the feature pyramid FPN method under the CascadeR-CNN framework are visualized as follows: Figure 3 shown.

[0055] As shown in Table 1 below, combined with the COCO dataset evaluation indicators mAP, AP 50 ,AP 75 ,AP s ,AP m ,AP l As can be seen from the meaning, this method has a significant effect on the target detection dataset we constructed ourselves, especially mAP, AP 50 ,AP 75 ,AP m ,AP l In the Faster R-CNN framework, compared with the initial method, the use of multi-scale fusion feature pyramid network to extract features can achieve the best results, which are improved from 32.8, 57.1, 33.0, 11.8, 40.1 to 34.3, 58.1, 35.6, 12.9, 41.6 respectively. In the Cascade R-CNN framework, compared with the initial method, they are improved from 34.5, 56.1, 36.1, 11.4, 41.9 to 36.5, 58.2, 37.7, 13.4, 44.2 respectively, which fully verifies that the method of the present invention can enable the detection network to learn the feature representation of instances of the same category at different scales during training.

[0056] Table 1 Results of multi-scale fusion feature pyramid network on our own object detection dataset

[0057]

[0058] As shown in Table 2 below, combined with the COCO dataset evaluation indicators mAP, AP 50 ,AP 75 ,AP s ,,AP m ,AP l It can be seen from the meaning that this method is not only effective on our own dataset, but also has a good performance improvement when applied to the authoritative large-scale dataset MS COCO. In the Faster R-CNN framework, compared with the initial method, the use of multi-scale fusion feature pyramid network to extract features has achieved better results, mAP, AP 50 ,AP 75 ,AP s,AP m ,AP l They are improved from 37.4, 58.1, 40.4, 21.2, 41.0, 48.1 to 38.7, 59.9, 42.0, 22.4, 42.2, 50.3 respectively. Under the Cascade R-CNN framework, compared with the initial method, they are improved from 40.3, 58.6, 44.0, 22.5, 43.8, 52.9 to 41.8, 60.4, 45.2, 23.7, 45.2, 55.3 respectively, which fully verifies the effectiveness of this method on complex object detection datasets.

[0059] like Figure 3 As shown in the figure, we add our multi-scale fusion feature pyramid network to the Cascade R-CNN framework and visualize the detection results.

[0060] Table 2 Results of multi-scale fusion feature pyramid network on MS COCO dataset

[0061]

[0062] The above embodiments are provided only for the purpose of describing the present invention, and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Any equivalent substitutions and modifications made without departing from the spirit and principles of the present invention should be included within the scope of the present invention.

Claims

1. A multi-scale fusion feature extraction method for target detection, used to extract multi-scale features in an image, characterized in that: include: Step 1: Take out the intermediate layer feature map [P2, P3, P4, P5] of each block in the backbone network, and then use convolution kernel size 3x3, 1x1 convolution and convolution kernel size 3x3, dilation rate 2 and 3 respectively to extract features from the intermediate layer feature map [P2, P3, P4, P5] of the backbone network, and get the new feature map [Q i2 ,Q i3 ,Q i4 ,Q i5 ], i = 1, 2, 3, 4, indicating that four different convolution kernels are used to extract feature maps with different receptive fields; Step 2: Based on the new feature map [Q i2 ,Q i3 ,Q i4 ,Q i5 ] Construct four feature pyramid networks, and the features of the four feature pyramid networks are expressed as [N i2 ,N i3 ,N i4 ,N i5 ]; Step 3: [N i2 ,N i3 ,N i4 ,N i5 ] Perform feature fusion on each layer: Here, the value of n is 4, which means that the new fusion feature pyramid is reconstructed on the feature maps of four different scales, and the obtained feature maps are [M2, M3, M4, M5]; Step 4: Extract features through convolution operation with a convolution kernel size of 3x3 to construct lateral connections, downsample the feature map M2 and fuse it with the lateral connection feature maps of the corresponding scale to construct a bottom-up feature pyramid [O2, O3, O4, O5]; Step 5: Add a deep feature enhancement network DFEN to the underlying feature map P2 of the backbone network and the underlying feature map M2 of the fused feature pyramid network, respectively. The structures of the two are exactly the same. N i5 '=N i5 +DFEN1(P2) <h2 style=";text-align:left;direction:ltr">O<h2 style=";text-align:left;direction:ltr"> i5 <h2 style=";text-align:left;direction:ltr"> '=O<h2 style=";text-align:left;direction:ltr"> i5 <h2 style=";text-align:left;direction:ltr"> +DFEN2(M2); Step 6: Use 3x3 convolution to extract features from the feature pyramid feature map [O2, O3, O4, O5] obtained in step 4.

2. The multi-scale fusion feature extraction method for target detection according to claim 1, characterized in that: In step 2, the construction process of the four feature pyramid networks is as follows: upsampling is performed from the bottom layer of the backbone network and respectively associated with the feature map [Q 12 ,Q 13 ,Q 14 ,Q 15 ] fusion, build the first top-down feature pyramid network; start upsampling from the bottom layer of the backbone network and combine it with the feature map [Q 22 ,Q 23 ,Q 24 ,Q 25 ] fusion, build a second top-down feature pyramid network; start upsampling from the bottom-level features of the backbone network and combine them with the feature map [Q 32 ,Q 33 ,Q 34 ,Q 35 ] fusion, build a third top-down feature pyramid network; start upsampling from the bottom-level features of the backbone network and combine them with the feature map [Q 42 ,Q 43 ,Q 44 ,Q 45 ] are fused to construct the fourth top-down feature pyramid network.

3. A multi-scale fusion feature extraction system for target detection, used to extract multi-scale features in images, characterized in that: include: Backbone network feature extraction module: It is used to extract the intermediate layer feature map [P2, P3, P4, P5] of each block in the backbone network, and then use convolution kernel size 3x3, 1x1 convolution and convolution kernel size 3x3, dilation rate 2 and 3 respectively to extract features from the intermediate layer feature map [P2, P3, P4, P5] of the backbone network, and obtain new feature maps [Q i2 ,Q i3 ,Q i4 ,Q i5 ], i = 1, 2, 3, 4, indicating that four different convolution kernels are used to extract feature maps with different receptive fields; Feature pyramid reconstruction module: used to reconstruct the feature map based on the new feature map [Q i2 ,Q i3 ,Q i4 ,Q i5 ] Construct four feature pyramid networks, and the features of the four feature pyramid networks are expressed as [N i2 ,N i3 ,N i4 ,N i5 ]; Feature fusion module: used to transform [N i2 ,N i3 ,N i4 ,N i5 ] Perform feature fusion on each layer: Here, the value of n is 4, which means that a new fused feature pyramid is reconstructed on feature maps of four different scales; here, j = 2, 3, 4, 5, which means that feature fusion is performed on different pyramid levels respectively, and the final feature map [M2, M3, M4, M5] is obtained; Reconstruct bottom-up feature pyramid module: It is used to extract features through convolution operations with a convolution kernel size of 3x3 to construct lateral connections, downsample the feature map M2 and fuse it with the lateral connection feature maps of the corresponding scale to construct a bottom-up feature pyramid [O2, O3, O4, O5]; Add deep feature enhancement network module: It is used to add deep feature enhancement network DFEN to the underlying feature map P2 of the backbone network and the underlying feature map M2 of the fusion feature pyramid network. The structures of the two are exactly the same. N i5 '=N i5 +DFEN1(P2) <h2 style=";text-align:left;direction:ltr">O<h2 style=";text-align:left;direction:ltr"> i5 <h2 style=";text-align:left;direction:ltr"> '=O<h2 style=";text-align:left;direction:ltr"> i5 <h2 style=";text-align:left;direction:ltr"> +DFEN2(M2); Feature extraction module: used to extract features from the feature pyramid feature map [O2, O3, O4, O5] obtained in step 4 using 3x3 convolution.

4. The multi-scale fusion feature extraction system for target detection according to claim 3, characterized in that: The feature pyramid reconstruction module and the construction process of the four feature pyramid networks are as follows: upsampling starts from the bottom layer of the backbone network and respectively 12 ,Q 13 ,Q 14 ,Q 15 ] fusion, build the first top-down feature pyramid network; start upsampling from the bottom layer of the backbone network and combine it with the feature map [Q 22 ,Q 23 ,Q 24 ,Q 25 ] fusion, build a second top-down feature pyramid network; start upsampling from the bottom-level features of the backbone network and combine them with the feature map [Q 32 ,Q 33 ,Q 34 ,Q 35 ] fusion, build a third top-down feature pyramid network; start upsampling from the bottom-level features of the backbone network and combine them with the feature map [Q 42 ,Q 43 ,Q 44 ,Q 45 ] are fused to construct the fourth top-down feature pyramid network.

5. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions which, when executed by a computing device, cause the computing device to perform any one of the methods according to claims 1 to 2.

6. A computing device, characterized in that include: One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the methods according to claims 1 to 2.

Images