Object detection method based on adaptive fusion of multi-level local and global features

Abstract

The present invention relates to a target detection method for adaptive fusion of multi-level local and global features, comprising the following steps: step 1: preparing a training data set for target detection; step 2: selecting a backbone network for target detection, building a candidate detection window extraction network, and constructing a multi-level local and global feature adaptive fusion network MLGNet on the basis of the backbone network, and setting the training loss function of the candidate window extraction network and MLGNet; step 3: initializing the network parameters of each part of the detector and the hyperparameters required for the training process; step 4: updating the weight of the detector using a back propagation algorithm; and obtaining the final detector after a set number of training times.

Description

Technical Field

[0001] The present invention relates to a target detection method in an intelligent system (such as intelligent driving, intelligent monitoring, intelligent interaction, etc.), in particular to a target detection method based on deep learning. Background Art

[0002] Object detection mainly refers to locating objects in images or videos and giving the specific category of the objects. In recent years, based on deep convolutional neural network technology, object detection has achieved great success and is widely used in intelligent driving, intelligent transportation, intelligent search, intelligent authentication and other fields. For example, smart cars need to detect obstacles in front before making decisions and controls, and intelligent interactive systems need to detect people who need to interact before recognizing related gestures and instructions.

[0003] Due to the powerful feature expression ability of deep convolutional neural networks, deep convolutional neural networks have achieved great success in tasks such as image classification, target detection, and semantic segmentation. For target detection based on deep learning, the relevant methods mainly include two categories: two-stage methods and single-stage methods. Compared with the single-stage method, the two-stage method has higher detection performance. This patent mainly focuses on the two-stage method. The two-stage method mainly includes two parts: a candidate detection window extraction network and a candidate detection window classification and regression network. In order to roughly extract objects that may exist in the image, the candidate window extraction network generates a certain number of candidate detection windows. Based on these candidate detection windows, the candidate window classification and regression network further classifies and regresses these candidate detection windows to obtain the final position and classification score of the detection window.

[0004] Among the two-stage methods, the more representative work is Faster R-CNN [1] . It extracts and classifies candidate windows through a shared basic network. Faster R-CNN uses the RoI pooling layer to extract the global features of the region of interest of the candidate detection window for classification and regression of the candidate detection window. Therefore, it ignores the local features of the object. In fact, for occluded objects, local features are more conducive to improving detection performance. At the same time, the RoI pooling layer scales the feature map of the original detection box area to a fixed size, which is not robust enough to the deformation of the object. In order to encode local information into features, Dai et al. [2] A position-sensitive pooling layer PSRoI is proposed. Specifically, PSRoI divides each region of interest into sub-regions of size k×k. The response value of each sub-region is the average corresponding value of the corresponding region on the corresponding channel of the position-sensitive feature map. Compared with Faster R-CNN, the object detector R-FCN based on PSRoI has similar detection performance but faster detection speed. Zhu et al. [3]The RoI layer and PSRoI layer are integrated to fully utilize global and local features. However, this method lacks the ability to mine multi-scale features and how to adaptively fuse these local and global features. To encode multi-scale features, He et al. [4] proposed to use the spatial pyramid structure to fuse multi-scale features. This method is more robust to object deformation. Zhao et al. [5] A similar structure is used to improve the performance of semantic segmentation. Liu et al. [6] The related ideas of spatial pyramid structure are used for single-stage object detection. Wang et al. [7] Three-dimensional convolution operation is used to fuse multi-scale features.

[0005] References:

[0006] [1]S.Ren, K.He, R.Girshick, and J.Sun, Faster R-CNN: Towards Real-TimeObject Detection with Region Proposal Networks, IEEE Trans.Pattern Analysis and Machine Intelligence, vol.39, no.6, pp.1137-1149, 2017.

[0007] [2] J.Dai, Y.Li, K.He, and J.Sun, R-FCN: Object Detection via Region-basedFully Convolutional Networks, Proc.Advances in Neural Information ProcessingSystems, 2015.

[0008] [3] Y. Zhu, C. Zhao, J. Wang, X. Zhao, Y. Wu, H. Lu, CoupleNet: Coupling Global Structure with Local Parts for Object Detection, IEEE Computer Vision and Pattern Recognition, 2017.

[0009] [4] K.He,

[0010] [5]H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, Pyramid Scene Parsing Network, Proc. IEEE Computer Vision and Pattern Recognition, 2017.

[0011] [6] S. Liu, D. Huang, Y. Wang, Receptive Field Block Net for Accurate and Fast Object Detection, Proc. European Conference on Computer Vision, 2018.

[0012] [7] X. Wang, S. Zhang, Z. Yu, L. Feng, and W. Zhang, Scale-Equalizing PyramidConvolution for Object Detection, Proc. IEEE Computer Vision and Pattern Recognition, 2020. Summary of the invention

[0013] The present invention proposes a target detection method with adaptive fusion of multi-level local and global features, which can make full use of multi-scale features and local and global features to improve the performance of target detection. The method proposed in the present invention uses a multi-level PSRoI layer to extract multi-level local features, and uses a multi-level RoI layer to extract multi-level global features. Based on these multi-level features, the proposed method predicts the respective weight coefficients to generate the final feature map. Therefore, the proposed method can learn multi-scale local and global features, which helps to improve the performance of target detection. The technical solution is as follows:

[0014] A target detection method with multi-level local and global feature adaptive fusion includes the following steps:

[0015] Step 1: Prepare a training data set for target detection, including training images and corresponding object annotations; the annotations are the coordinates of the detection box where the object is located and the category of the object;

[0016] Step 2: Select the backbone network for target detection, build a candidate detection window extraction network, and build a multi-level local and global feature adaptive fusion network MLGNet based on the backbone network, and set the training loss function of the candidate window extraction network and MLGNet;

[0017] Among them, the method of constructing a multi-level local and global feature self-adaptive fusion network MLGNet based on the backbone network is as follows:

[0018] For a given input image, a deep feature map is generated through a backbone network; for each candidate detection window, six branches are used to extract multi-level local and global features respectively: three branches are used to generate three sensitive feature maps at different positions through three different convolutional layers, and based on the three sensitive feature maps at different positions, three different PSRoI layers are used to extract 3×3, 5×5, and 7×7 sensitive feature maps of the candidate window's region of interest, respectively. Then, the three sensitive feature maps of different sizes and positions are upsampled to the same 7×7 feature map through bilinear interpolation; a feature map is generated using the convolutional layers of the other three branches, and based on the feature map, three different RoI layers are used to extract 3×3, 5×5, and 7×7 global feature maps of the candidate window's region of interest, respectively. Then, the three global feature maps of different sizes are upsampled to the same 7×7 feature map through bilinear interpolation;

[0019] Construct a feature adaptive fusion unit by concatenating the six 7×7 feature maps generated by the previous six branches in the direction of the feature channel to obtain the original concatenated feature map; for the original concatenated feature map, we perform the following operations: first, use a global average global pooling operation to generate a feature vector of the length of the feature channel; then use a fully connected layer to generate a feature vector of length 6; then pass a Sigmoid layer to normalize the feature vector of length 6; finally, use the six values ​​of the normalized feature vector to multiply the features corresponding to the above six branches respectively and concatenate them in the direction of the feature channel to obtain the enhanced concatenated feature map; add the enhanced concatenated feature map and the original concatenated feature map to obtain the final output feature map;

[0020] Form a multi-level local and global feature adaptive fusion network MLGNet;

[0021] Step 3: Initialize the network parameters of each part of the detector and the hyperparameters required for the training process;

[0022] Step 4: Use the back-propagation algorithm to update the weights of the detector; after a set number of training times, the final detector is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Multi-level local and global feature adaptive fusion network (MLGNet)

[0024] Figure 2 Feature Adaptive Fusion Unit

[0025] Figure 3 Specific implementation of the proposed method DETAILED DESCRIPTION

[0026] The main technical problem solved by the present invention is how to make full use of multi-scale features and local and global features to improve the performance of target detection. To address this problem, this patent proposes a target detection method that adaptively fuses multi-level local and global features. Specifically, the proposed method uses a multi-level PSRoI layer to extract multi-level local features, and a multi-level RoI layer to extract multi-level global features. Based on these multi-level features, the proposed method predicts the respective weight coefficients to generate the final feature map. Therefore, the proposed method can learn multi-scale local and global features, which helps to improve the performance of target detection.

[0027] We first introduce the proposed multi-level local and global feature adaptive fusion network, and then introduce how to use the proposed network for object detection.

[0028] (1) Multi-level local and global feature adaptive fusion network

[0029] Figure 1 The basic schematic diagram of the multi-level local and global feature adaptive fusion network (abbreviated as MLGNet) is given. For a given input image, it first generates a deep feature map through a backbone network (such as VGG16 and ResNet). For each candidate detection window, MLGNet uses six branches to extract multi-level local and global features respectively. The top three branches first generate three different position-sensitive feature maps through three different convolutional layers. Based on the three position-sensitive feature maps, three different PSRoI layers extract 3×3, 5×5, and 7×7 position-sensitive feature maps of the candidate window's region of interest, respectively, and then upsample the three different-sized feature maps to the same 7×7 feature map through bilinear interpolation. The bottom three branches first generate a feature map using a convolutional layer. Based on the feature map, three different RoI layers extract 3×3, 5×5, and 7×7 global feature maps of the candidate window's region of interest, respectively, and then upsample the three different-sized feature maps to the same 7×7 feature map through bilinear interpolation.

[0030] Finally, we concatenate the six 7×7 feature maps generated by the upper and lower six branches and use the adaptive fusion module to generate a new output feature map. The new output feature passes through a convolutional layer for subsequent classification and regression tasks. Figure 2 A feature adaptive fusion module is given. For the six concatenated feature maps, we first use a global average global pooling operation to generate a feature vector with a length of the channel, then use a fully connected layer to generate a feature vector with a length of 6, then pass it through a sigmoid layer for normalization, and finally use the normalized vector to multiply the six branches respectively to obtain the enhanced feature map. The enhanced feature map is added to the input feature map of the feature adaptive fusion module to obtain the new output feature.

[0031] The proposed MLGNet is able to extract multi-level local and global features and perform adaptive fusion, thereby improving the expressiveness and robustness of subsequent classification and regression features.

[0032] (2) Object detection based on multi-level local and global feature adaptive fusion network

[0033] In this section, we introduce how to apply the proposed multi-level local and global feature adaptive fusion network MLGNet to target detection. Target detection mainly consists of two different steps: training phase and testing phase. The training phase is used to learn the network parameters of MLGNet, and the testing phase uses the trained MLGNet to detect images and determine whether there are objects in the image. First, let's introduce the training process:

[0034] Step 1: Prepare a training dataset for object detection (such as general object detection data MS COCO). The data needs to contain a certain number of training images and corresponding object annotations. The annotations are the coordinates of the detection box where the object is located and the category of the object.

[0035] Step 2: Select the backbone network of the target detector, build the candidate detection window extraction network and the proposed multi-level local and global feature adaptive fusion network MLGNet. Set the training loss function of the candidate window extraction network and MLGNet.

[0036] Step 3: Initialize the network parameters of each part of the detector and the hyperparameters required for the training process. The network parameters can be initialized randomly, and the hyperparameters of the training process include the number of iterations, learning rate, batch size, etc.

[0037] Step 4: Use the back propagation algorithm to update the weight of the detector. After the set number of training times, the final detector is obtained.

[0038] Then, we introduce the testing process:

[0039] Step 1: Prepare the test image. Use the candidate detection window network to extract several candidate detection windows, and use the multi-level local and global feature adaptive fusion network MLGNet to accurately classify and regress these candidate detection windows.

[0040] Step 2: Use the non-maximum suppression algorithm NMS to post-process the output results of MLGNet to generate the final detection results.

[0041] Figure 3 The specific implementation method of the method proposed in the present invention is given, and the main steps are as follows:

[0042] Step 1: Select a target detection dataset based on the application scenario, which includes several images and corresponding annotation information, such as the location information of the target and the category information to which it belongs.

[0043] Step 2: Select the backbone network of the target detector to build the candidate detection window extraction network and the proposed multi-level local and global feature adaptive fusion network MLGNet.

[0044] Step 3: Initialize the network parameters of each part of the detector and the hyperparameters required for the training process.

[0045] Step 4: Use the back propagation algorithm to update the weight of the detector. After the set number of training times, the final detector is obtained.

[0046] Step 5: Prepare the test image. Use the candidate detection window network to extract several candidate windows, and use the multi-level local and global feature adaptive fusion network MLGNet to accurately classify and regress these detection windows.

[0047] Step 6: Use the non-maximum suppression algorithm to post-process the network output results to generate the final detection results.

Claims

1. A target detection method with multi-level local and global feature adaptive fusion, comprising the following steps: Step 1: Prepare a training data set for target detection, including training images and corresponding object annotations; the annotations are the coordinates of the detection box where the object is located and the category of the object; Step 2: Select the backbone network for target detection, build a candidate detection window extraction network, and build a multi-level local and global feature adaptive fusion network MLGNet based on the backbone network, and set the training loss function of the candidate window extraction network and MLGNet; in, The method of constructing a multi-level local and global feature adaptive fusion network MLGNet based on the backbone network is as follows: For a given input image, a deep feature map is generated through a backbone network; for each candidate detection window, six branches are used to extract multi-level local and global features respectively: three branches are used to generate three sensitive feature maps at different positions through three different convolutional layers, and based on the three sensitive feature maps at different positions, three different PSRoI layers are used to extract 3×3, 5×5, and 7×7 sensitive feature maps of the candidate window's region of interest, respectively. Then, the three sensitive feature maps of different sizes and positions are upsampled to the same 7×7 feature map through bilinear interpolation; a feature map is generated using the convolutional layers of the other three branches, and based on the feature map, three different RoI layers are used to extract 3×3, 5×5, and 7×7 global feature maps of the candidate window's region of interest, respectively. Then, the three global feature maps of different sizes are upsampled to the same 7×7 feature map through bilinear interpolation; Construct a feature adaptive fusion unit by concatenating the six 7×7 feature maps generated by the previous six branches in the direction of the feature channel to obtain the original concatenated feature map; for the original concatenated feature map, we perform the following operations: first, use a global average global pooling operation to generate a feature vector of the length of the feature channel; then use a fully connected layer to generate a feature vector of length 6; then pass a Sigmoid layer to normalize the feature vector of length 6; finally, use the six values ​​of the normalized feature vector to multiply the features corresponding to the above six branches respectively and concatenate them in the direction of the feature channel to obtain the enhanced concatenated feature map; add the enhanced concatenated feature map and the original concatenated feature map to obtain the final output feature map; Form a multi-level local and global feature adaptive fusion network MLGNet; Step 3: Initialize the network parameters of each part of the detector and the hyperparameters required for the training process; Step 4: Update the weights of the detector using the back-propagation algorithm; After a set number of training times, the final detector is obtained.

Images