A new image object detection method based on feature pyramid network

Abstract

The invention discloses an image target detection method based on a novel feature pyramid network. First, a pyramid pooling module is used to process the input highest-level feature map to obtain multi-scale features containing context information of different-size regions of the feature map. Then, dimension reduction and average pooling are performed on the feature maps of all levels to obtain four feature maps with the same number of channels and the same size. Then, two 1x1 convolution layers are sent to calculate the position weights, and the four feature maps are position-weighted. Then, 3x3 convolution is used to process the highest-level feature map and five feature maps are added and fused from top to bottom. Finally, prediction is performed on the five feature maps to obtain target detection results. The invention can not only solve the problems of aliasing effect, dilution of high-level semantic information and limited access to low-level spatial information, but also generate more discriminative features without complicating the fusion structure of the feature pyramid network, and improve the detection accuracy of different types of detectors.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and in particular to an image target detection method based on a novel feature pyramid network. Background Art

[0002] Object detection is the simultaneous estimation of the position and category of objects in an image. It can be used in modern society in scenarios such as autonomous driving, helmet detection, pedestrian detection, etc. The current mainstream object detection algorithms have poor detection effects on objects of different scales, especially small objects. How to solve this problem and obtain more accurate detection results has become a current research hotspot.

[0003] The mainstream object detection algorithms can be roughly divided into two-stage detectors and single-stage detectors. The two-stage object detector first generates some predefined, dense sliding windows on the feature map through the proposal mechanism to determine the area where the object is most likely to appear, called the region of interest. These regions of interest are further processed by the detection network, and the object detection results are output in the form of bounding boxes and related object category probabilities. Finally, the non-maximum suppression algorithm is applied to the detection results to eliminate duplicate or highly overlapping results. The single-stage object detector is designed to directly predict the detection results in the proposal box after extracting features from the input image. The image is first input into the feature extraction network, usually a deep convolutional neural network, to obtain the feature map, and then a set of dense object proposal boxes are generated on the feature map and matched with the annotation box for sampling and labeling. Finally, the labeled proposal boxes are input into the classification network and regression network for training. These two types of detectors can achieve good detection results for objects of fixed scale, but the detection accuracy will be greatly reduced for objects of different scales, especially small objects.

[0004] In order to improve the detection effect of objects of different scales, He et al. proposed a feature pyramid network (FPN), which constructs a feature map pyramid through a top-down fusion path and a lateral connection method. The feature map pyramid combines low-resolution, strong semantic features with high-resolution, weak semantic features, and outputs five levels of feature maps. The feature maps of each level have different scales, but all contain rich semantic information. The detector predicts the target in these five feature maps at the same time, thereby improving the detection effect of objects of different scales. However, this method simply scales the feature maps of adjacent levels to the same scale through the nearest neighbor interpolation method and then adds and fuses them. It does not consider the semantic differences between feature maps of different levels, resulting in redundant or even erroneous information during fusion, which easily causes aliasing effects and poor detection effects. On the other hand, the top-down progressive fusion structure adopted will make the low-level feature information containing the details of the target spatial structure only accessible in the last fusion stage, while the high-level semantic information containing the complete target context information is gradually transmitted to the shallower layer and diluted.

[0005] Based on FPN, Liu et al. proposed PAFPN, which added a bottom-up fusion path to FPN, effectively alleviating the problems of dilution of high-level semantic information and limited access to low-level spatial structure information. However, the two-step progressive fusion structure will cause more serious aliasing effects and make the fusion structure complicated. Tan et al. proposed BiFPN, which achieves better detection results through bidirectional cross-scale connections and weighted feature fusion. However, the feature maps of each level are connected with the feature maps of other levels, which makes the fusion structure of the feature pyramid network more complicated. Pang et al. proposed Balanced FPN, which scales the feature maps of all levels to the size of the feature maps of the intermediate level and adds them up to take the average. It also introduces an attention module to refine the fused features, which can comprehensively balance various types of feature information on the feature maps of each level, but does not specifically consider the feature information on the feature maps of each level. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an image target detection method based on a novel feature pyramid network, which can not only effectively solve the aliasing effect, dilution of high-level semantic information and limited access to low-level spatial information caused by the fusion structure of the traditional feature pyramid network, but also generate more discriminative features without complicating the fusion structure of the feature pyramid network, thereby improving the detection accuracy of different types of target detectors.

[0007] To achieve the above purpose, the technical solution provided by the present invention is:

[0008] An image target detection method based on a novel feature pyramid network comprises the following steps:

[0009] S1, using the pyramid pooling module to process the highest level feature map C5 of the input feature pyramid network to obtain multi-scale features containing context information of regions of different sizes;

[0010] S2, use 1x1 convolution layer to reduce the dimension of all level feature maps of the input, so that the channel dimension of all feature maps is 256;

[0011] S3, perform average pooling on all levels of feature maps to make the resolution of all feature maps consistent with the size of the highest level feature map C5, and obtain feature map C′ i , where i = 2, 3, 4, 5;

[0012] S4, send all the feature maps after pooling processing to two series-connected 1x1 convolutional layers to calculate the position weight;

[0013] S5. Use the nearest neighbor interpolation method to upsample the position weight to the size of the feature map of the corresponding level to obtain a position weight map;

[0014] S6. Perform position weighting on all level feature maps to obtain position weighted feature maps;

[0015] S7, using the nearest neighbor interpolation method to scale the feature maps of each level from top to bottom to the size of the feature maps of the adjacent levels and add and fuse them, and at the same time use 3x3 convolution to process the position-weighted feature map of the highest level to obtain five feature maps of different levels for prediction;

[0016] S8. The target detection result can be obtained by predicting on the five different levels of feature maps obtained in step S7.

[0017] Furthermore, the step S1 specifically includes:

[0018] S1-1, divide the highest level feature map C5 into sub-regions of 1x1, 2x2, 3x3, and 6x6 sizes and perform average pooling to obtain 4 output feature maps of different sizes;

[0019] S1-2, send the four feature maps obtained in step S1-1 to the 1x1 convolution layer respectively to reduce the number of channels to 256;

[0020] S1-3, use the nearest neighbor interpolation method to upsample the feature map after reducing the number of channels to the size of C5, and concatenate it with C5 to obtain the final output feature map.

[0021] Furthermore, in step S4, the position weight calculation formula is as shown in formula (1):

[0022] W i =σ(F2(δ(F1(C′i )))) (1)

[0023] In formula (1), C′ i represents all level feature maps after dimensionality reduction and average pooling, where i = 2, 3, 4, 5, W i represents the calculated position weight, F1 and F2 represent 1x1 convolutional layers, defined as in formula (2), σ is the Sigma activation function, defined as in formula (3), δ is the ReLU activation function, defined as in formula (4);

[0024]

[0025] In formula (2), w1 is a convolution kernel of size 1x1, x is the input image, is the convolution operation;

[0026]

[0027] δ(x)=max(0,x) (4).

[0028] Furthermore, in step S6, the position weighted calculation formula is as shown in formula (5):

[0029] P i =W′ i ⊙C i (5)

[0030] In formula (5), P i Represents the position-weighted feature map, W′ i represents the position weight map, i = 2, 3, 4, 5, and ⊙ represents element-wise multiplication.

[0031] Furthermore, in step S7, the 3x3 convolution calculation formula is as shown in formula (6):

[0032]

[0033] In formula (6), w3 is a 3x3 convolution kernel with a step size of 2, P5 is the highest-level feature map after position weighting, and P6 is the new highest-level feature map obtained by performing a 3x3 convolution on P5. is the convolution operation.

[0034] Compared with the prior art, the principles and advantages of this technical solution are as follows:

[0035] 1) This technical solution proposes a position weighting method, which learns position weights from all channels of all input feature maps and assigns them to feature maps of each level. It not only considers the feature information of feature maps of each level in a targeted manner, but also comprehensively balances the feature information of feature maps of different levels. It has a guiding role in the fusion process of FPN, effectively refines the fused aliasing features, propagates low-level spatial structure information upward, and solves the dilution problem when high-level semantic information is propagated downward.

[0036] 2) This technical solution uses a pyramid pooling module to extract multi-scale features containing contextual information of regions of different sizes in the feature map. Combined with the position weighting method, it can make the features on the feature map pyramid output by FPN more discriminative.

[0037] 3) This technical solution does not complicate the fusion structure of FPN. It only adds two operations, position weighting and pyramid pooling, to the traditional FPN, which can effectively solve the problems caused by the fusion method of FPN. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the services required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 It is a principle flow chart of an image target detection method based on a novel feature pyramid network of the present invention;

[0040] Figure 2 This is the structural diagram of the pyramid pooling module. DETAILED DESCRIPTION

[0041] The present invention will be further described below in conjunction with specific embodiments:

[0042] like Figure 1 As shown, the image target detection method based on the novel feature pyramid network described in this embodiment includes the following steps:

[0043] S1, using the pyramid pooling module to process the highest level feature map C5 of the input feature pyramid network to obtain multi-scale features containing context information of regions of different sizes;

[0044] Assume that a deep convolutional neural network has been used to extract four feature maps C2, C3, C4, and C5 from the source image and input them into the feature pyramid network. Figure 2, the pyramid pooling module is used to divide the highest-level feature map C5 into sub-regions of 1x1, 2x2, 3x3, and 6x6 sizes and perform average pooling to obtain 4 output feature maps of different sizes. Then the four feature maps are sent to 1x1 convolution to reduce the number of channels to 256, and then the nearest neighbor interpolation method is used to upsample the feature maps with reduced channels to the size of C5, and concatenated with C5 to obtain the final output feature map containing the context information of different sizes of C5 regions.

[0045] S2, use 1x1 convolution layer to reduce the dimension of all level feature maps of the input, so that the channel dimension of all feature maps is 256;

[0046] S3, perform average pooling on all levels of feature maps to make the resolution of all feature maps consistent with the size of the highest level feature map C5, and obtain feature map C′ i , where i = 2, 3, 4, 5;

[0047] S4, all feature maps after pooling are sent to two 1x1 convolutional layers in series to calculate the position weight; the position weight calculation formula is shown in formula (1):

[0048] W i =σ(F2(δ(F1(C′ i )))) (1)

[0049] In formula (1), C′ i represents all level feature maps after dimensionality reduction and average pooling, where i = 2, 3, 4, 5, W i represents the calculated position weight, F1 and F2 represent 1x1 convolutional layers, defined as in formula (2), σ is the Sigmoid activation function, defined as in formula (3), δ is the ReLU activation function, defined as in formula (4);

[0050]

[0051] In formula (2), w1 is a convolution kernel of size 1x1, x is the input image, is the convolution operation;

[0052]

[0053] δ(x)=max(0,x) (4).

[0054] S5. Use the nearest neighbor interpolation method to upsample the position weight to the size of the feature map of the corresponding level to obtain a position weight map;

[0055] S6. Perform position weighting on all level feature maps to obtain position weighted feature maps;

[0056] The calculation formula of position weighting is shown in formula (5):

[0057] P i =W′ i ⊙C i (5)

[0058] In formula (5), P i Represents the position-weighted feature map, W′ i represents the position weight map, i = 2, 3, 4, 5, and ⊙ represents element-wise multiplication.

[0059] S7, using the nearest neighbor interpolation method to scale the feature maps of each level from top to bottom to the size of the feature maps of the adjacent levels and add and fuse them, and at the same time use 3x3 convolution to process the position-weighted feature map of the highest level to obtain five feature maps of different levels for prediction;

[0060] The calculation formula of 3x3 convolution is shown in formula (6):

[0061]

[0062] In formula (6), w3 is a 3x3 convolution kernel with a step size of 2, P5 is the highest-level feature map after position weighting, and P6 is the new highest-level feature map obtained by performing a 3x3 convolution on P5. is the convolution operation.

[0063] S8. The target detection result can be obtained by predicting on the five different levels of feature maps obtained in step S7.

[0064] This embodiment has the following advantages:

[0065] 1. A position weighted method is proposed to learn the position weights from all channels of all input feature maps and assign them to feature maps of each level. It not only considers the feature information of feature maps of each level in a targeted manner, but also comprehensively balances the feature information of feature maps of different levels. It guides the fusion process of FPN, effectively refines the fused aliasing features, propagates low-level spatial structure information upward, and solves the dilution problem of high-level semantic information when it propagates downward.

[0066] 2. Use the pyramid pooling module to extract multi-scale features containing contextual information of different size areas of the feature map. Combined with the position weighting method, the features on the feature map pyramid output by FPN can be made more discriminative.

[0067] 3. The fusion structure of FPN is not complicated. Only two operations, position weighting and pyramid pooling, are added to the traditional FPN, which can effectively solve the problems caused by the fusion method of FPN.

[0068] The embodiments described above are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, all changes made according to the shape and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image target detection method based on a novel feature pyramid network, characterized in that: The following steps are involved: S1, using the pyramid pooling module to process the highest level feature map C5 of the input feature pyramid network to obtain multi-scale features containing context information of regions of different sizes; S2, use 1x1 convolution layer to reduce the dimension of all level feature maps of the input, so that the channel dimension of all feature maps is 256; S3, perform average pooling on all levels of feature maps to make the resolution of all feature maps consistent with the size of the highest level feature map C5, and obtain feature map C′ i , where i = 2, 3, 4, 5; S4, send all the feature maps after pooling processing to two series-connected 1x1 convolutional layers to calculate the position weight; S5. Use the nearest neighbor interpolation method to upsample the position weight to the size of the feature map of the corresponding level to obtain a position weight map; S6. Perform position weighting on all level feature maps to obtain position weighted feature maps; S7, using the nearest neighbor interpolation method to scale the feature maps of each level from top to bottom to the size of the feature maps of the adjacent levels and add and fuse them, and at the same time use 3x3 convolution to process the position-weighted feature map of the highest level to obtain five feature maps of different levels for prediction; S8. The target detection result can be obtained by predicting on the five different levels of feature maps obtained in step S7.

2. The image target detection method based on a novel feature pyramid network according to claim 1, characterized in that: The step S1 specifically includes: S1-1, divide the highest level feature map C5 into sub-regions of 1x1, 2x2, 3x3, and 6x6 sizes and perform average pooling to obtain 4 output feature maps of different sizes; S1-2, send the four feature maps obtained in step S1-1 to the 1x1 convolution layer to reduce the number of channels to 256; S1-3. Use the nearest neighbor interpolation method to upsample the feature map after reducing the number of channels to the size of C5, and concatenate it with C5 to obtain the final output feature map.

3. The image target detection method based on a novel feature pyramid network according to claim 1 is characterized in that: In step S4, the position weight calculation formula is as shown in formula (1): W i =σ(F2(δ(F1(C′ i )))) (1) In formula (1), C′ i represents all level feature maps after dimensionality reduction and average pooling, where i = 2, 3, 4, 5, W i represents the calculated position weight, F1 and F2 represent 1x1 convolutional layers, defined as in formula (2), σ is the Sigmoid activation function, defined as in formula (3), δ is the ReLU activation function, defined as in formula (4); In formula (2), w1 is a convolution kernel of size 1x1, x is the input image, is the convolution operation; δ(x)=max(0,x) (4).

4. The image target detection method based on a novel feature pyramid network according to claim 1, characterized in that: In step S6, the calculation formula of position weighting is shown in formula (5): P i =W′ i ⊙C i (5) In formula (5), P i Represents the position-weighted feature map, W′ i represents the position weight map, i = 2, 3, 4, 5, and ⊙ represents element-wise multiplication.

5. The image target detection method based on a novel feature pyramid network according to claim 1, characterized in that: In step S7, the 3x3 convolution calculation formula is shown in formula (6): In formula (6), w3 is a 3x3 convolution kernel with a step size of 2, P5 is the highest-level feature map after position weighting, and P6 is the new highest-level feature map obtained by performing a 3x3 convolution on P5. is the convolution operation.

Images