A target detection method based on residual encoding and decoding parallel network

Abstract

The present invention relates to a target detection method based on a residual codec parallel network, comprising a training phase and a testing phase, wherein the training phase comprises the following steps: preparing a training data set for target detection; selecting a basic network model; establishing a residual codec unit; and constructing a residual codec parallel network REDNet, wherein the method is as follows: based on the established residual codec unit, adding the proposed residual codec unit to different convolution blocks of the basic network model, adding a parallel residual codec unit next to the convolution block, and the convolution layers in the convolution blocks of the parallel basic network model and the convolution layers in the residual codec unit share weights to obtain the residual codec parallel network REDNet; a two-stage target detector is given; the residual codec parallel network REDNet is embedded in the given two-stage target detector, and the final detector is trained and generated.

Description

Technical Field

[0001] The present invention relates to a target detection method in computer vision fields such as video surveillance and unmanned driving, and in particular to a target detection method based on deep learning. Background Art

[0002] In recent years, with the rapid development of science and technology, a large amount of image and video data has begun to emerge. At the same time, computing devices, such as parallel computing devices and graphics cards, have powerful computing capabilities. Based on these factors, deep convolutional neural networks have begun to achieve great success in computer vision tasks, including image classification, object detection, semantic segmentation, etc. Object detection methods based on deep convolutional neural networks are widely used in intelligent driving, assisted driving, identity recognition, human-computer interaction and other fields. For example, in intelligent driving, the car needs to intelligently identify obstacles in front of the car, including pedestrians, cyclists, vehicles, etc., before making decision-making controls; and in unmanned supermarkets and other occasions, the monitoring system needs to intelligently identify pedestrians entering and leaving the supermarket.

[0003] The reason for the great success of deep convolutional neural networks is that they have rich feature expression capabilities through a series of convolution and nonlinear operations. Object detection methods based on deep convolutional neural networks can be divided into single-stage object detection methods and two-stage object detection methods. The two-stage object detection method mainly includes two steps: candidate window generation and candidate window classification. Among them, candidate window generation aims to generate a large number of candidate detection windows covering objects that may exist in the image, while candidate window classification aims to classify and regress these candidate windows to obtain the final detection results. Single-stage object detection directly classifies and regresses through dense candidate windows. Generally speaking, the two-stage object detection method has higher detection performance. This patent mainly focuses on the two-stage object detection method.

[0004] Existing two-stage target detection method Faster R-CNN [1] Based on typical deep convolutional neural networks, such as AlexNet, VGG, and ResNet. Based on these basic networks, Faster R-CNN first extracts candidate detection boxes and then classifies these candidate detection boxes. However, the receptive field of the feature maps output by these basic networks is relatively limited, and the features are not robust enough. [2][3] In order to enhance the context information and increase the receptive field of the feature map, researchers have made some improvements. Lin et al. [4] The encoder-decoder network is used to merge the features of the shallow network with the features of the deep network. Chen et al. [5] The receptive field is increased by parallelizing convolutional layers with different dilation rates. Peng et al. [6] Using large kernel operations to increase the semantics and receptive field of convolutional networks. Tan et al.[7] The semantics of features are enhanced by using network search and adaptive fusion. Qiao et al. [8] The recursive operation is used to enhance the receptive field of features through multiple cycles. Most of these methods are built after the backbone network, and the semantic level and robustness of the features are enhanced through subsequent processing. In fact, we believe that the semantic level of the network and the robustness of the features can be increased through parallel branches while building the backbone network.

[0005] References:

[0006] [1]S.Ren, K.He, R.Girshick, and J.Sun, Faster R-CNN: Towards Real-TimeObject Detection with Region Proposal Networks, IEEE Trans.Pattern Analysis and Machine Intelligence, vol.39, no.6, pp.1137-1149, 2017.

[0007] [2] W.Luo, Y.Li, R.Urtasun, and R.Zemel, Understanding the effective receptive field in deep convolutional neural networks, Proc.Advances in NeuralInformation Processing Systems, 2016.

[0008] [3]T.Lin, P.Dollar, R.Girshick, K.He, B.Hariharan, and S.Belongie, FeaturePyramid Networks for Object Detection, Proc.IEEE Computer Vision and PatternRecognition, 2017.

[0009] [4] T. Kong, F. Sun, A. Yao, H. Liu, M. Lu, and Y. Chen, RON: Reverse Connection with Objectness Prior Networks for Object Detection, Proc. IEEE Computer Vision and Pattern Recognition, 2017.

[0010] [5]L.-C.Chen, G.Papandreou, F.Schroff, and H.Adam, Rethinking atrousconvolution for semantic image segmentation, arXiv, 2017

[0011] [6] C.Peng,

[0012] [7] M.Tan, R.Pang, QVLe, EfficientDet: Scalable and Efficient ObjectDetection, Proc. IEEE Computer Vision and Pattern Recognition, 2020.

[0013] [8] S. Qiao, L.-C. Chen, A. Yuille, DetectoRS: Detecting Objects withRecursive Feature Pyramid and Switchable Atrous Convolution, arXiv, 2020. Summary of the invention

[0014] The purpose of this invention is to provide a target detection method of a parallel residual encoding and decoding unit, so as to increase the receptive field and semantic level of the target detection network, thereby improving the performance of target detection. To address this problem, this patent proposes that our backbone network contains two different branches. One branch is composed of several ordinary convolutional layers, and the other branch contains several convolutional layers and upsampling layers. At the same time, the two different branches enhance the robustness of the network and reduce the parameters of the network through weight sharing.

[0015] In this section, we first introduce the proposed parallel residual encoder-decoder unit and the network it constructs, and then introduce how to use the proposed network for object detection.

[0016] A target detection method based on a residual codec parallel network includes a training phase and a testing phase, wherein the training phase includes the following steps:

[0017] (1) Prepare a training dataset for object detection, which includes images for training and annotations of objects in the images. The annotations include the coordinates of the detection box where the object is located and the category to which the object belongs.

[0018] (2) Select the basic network model;

[0019] (3) Establishing a residual encoding and decoding unit for processing a given input feature map, the method is as follows: performing the first encoding and decoding operation, that is, downsampling the input feature map by 1 times through a convolutional layer with a step size of 2, and then using an upsampling operation to restore the downsampled features to the size of the input feature map; for the feature map outputted for the first time, continuing to perform the second encoding and decoding operation to generate a new feature map; the feature map outputted by the first encoding and decoding operation is treated as a residual and added to the feature map outputted by the current encoding and decoding operation to obtain a feature map after residual addition; for the feature map after residual addition, continuing to perform the third encoding and decoding operation to generate a new output feature map again, and adding the new output feature map generated this time to the output of the second encoding and decoding operation to obtain the final output feature map;

[0020] (4) Construct a residual codec parallel network REDNet. The method is as follows: Based on the established residual codec unit, the proposed residual codec unit is added to different convolution blocks of the basic network model, and a parallel residual codec unit is added next to the convolution block. The convolution layer in the convolution block of the parallel basic network model and the convolution layer in the residual codec unit share weights to obtain the residual codec parallel network REDNet.

[0021] (5) Given a two-stage target detector;

[0022] (6) Embed the residual encoder-decoder parallel network REDNet into the given two-stage target detector, train and generate the final detector. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Residual codec unit. 's2' represents a convolutional layer with a stride of 2.

[0024] Figure 2 Residual Encoder-Decoder Parallel Network (REDNet).

[0025] Figure 3 Specific implementation of the proposed method DETAILED DESCRIPTION

[0026] The main technical problem solved by the present invention is how to increase the receptive field and semantic level of the target detection network, thereby improving the performance of target detection. To address this problem, this patent proposes a target detection method of a parallel residual encoding and decoding unit. Specifically, the backbone network includes two different branches. One branch is composed of several ordinary convolutional layers, and the other branch includes several convolutional layers and upsampling layers. At the same time, the two different branches enhance the robustness of the network and reduce the parameters of the network through weight sharing.

[0027] The following first introduces the proposed parallel residual codec unit and the network it constructs, and then introduces how to use the proposed network for object detection.

[0028] (1) Residual Encoding and Decoding and Parallel Networks

[0029] Figure 1 A basic schematic diagram of the residual encoding and decoding unit is given. Figure 1 The detailed structure of the residual codec unit is given on the left side. For a given input feature map, it first downsamples the input feature map by 1 through a convolutional layer with a stride of 2, and then uses an upsampling operation, such as bilinear difference or nearest neighbor difference, to restore the downsampled features to the size of the input feature map. Based on the above two steps (referred to as the codec operation), we can enhance the contextual information and receptive field without reducing the spatial resolution. For the output feature map, it continues to generate a new feature map through a codec operation. At the same time, the feature map output by the previous codec operation can be treated as a residual and added to the feature map output by the current codec operation. For the feature map after residual addition, it continues to pass through the codec unit to generate a new output feature map. The output feature map is added to the output of the previous codec to obtain the final output feature map. Figure 2 A simplified schematic diagram of the residual encoding and decoding unit is given on the left and right. Figure 1 The three-layer residual codec unit is given. In fact, the number of layers of our residual codec unit can be set to the number of layers we need, that is, the same as the number of convolutional layers of the convolutional block corresponding to the backbone network.

[0030] Based on the proposed residual codec unit, Figure 2The schematic diagram of the constructed residual codec parallel network is given. For a given backbone network, we add a parallel residual codec unit next to each convolution block. Specifically, the backbone network contains five different convolution blocks Block1, Block2, Block3, Block4, Block5. In order to reduce the computational consumption, we do not add parallel residual units to convolution blocks Block1 and Block2, but only add parallel residual codec units RED3, RED4, RED5 to convolution blocks Block3, Block4, Block5. Compared with general convolution blocks, the proposed residual codec unit can increase the receptive field and context information. Therefore, the proposed residual codec parallel network has a larger receptive field, and can extract features with better semantics. In order to reduce network parameters, the convolution layers in the parallel general convolution blocks and the convolution layers in the proposed residual codec unit share weights. Therefore, compared with the traditional convolutional neural network, the proposed method does not increase the parameters of the network.

[0031] (2) Object detection method based on residual encoding and decoding parallel network

[0032] This paper introduces how to apply the proposed residual codec parallel network (abbreviated as REDNet) to object detection. Object detection mainly consists of two different steps: training phase and testing phase. The training phase is used to learn the network parameters of REDNet, and the testing phase uses the parameters learned in the training phase to detect whether there are objects in the image. First, we introduce the training process:

[0033] Step 1: Prepare a training dataset for object detection, which includes a certain number of images and annotations of objects in the images. The annotations mainly include the coordinates of the upper left and lower right corners of the detection box where the object is located, as well as the category to which the object belongs.

[0034] Step 2: Construct the proposed residual codec parallel network REDNet. Specifically, select a basic network model, such as VGG, ResNet, etc. Then add the proposed residual codec unit to different convolutional blocks of the basic network.

[0035] Step 3: We embed the residual encoder-decoder parallel network REDNet into a given two-stage target detector. For example, we select the two-stage target detector Faster R-CNN and paste the candidate detection window generation network RPN and the candidate detection window classification network Fast R-CNN in Faster R-CNN onto the feature map output by REDNet.

[0036] Step 4: Initialize the parameters of the detector and set some hyperparameters of the training process, such as the number of iterations, learning rate, etc. Based on these parameters and the corresponding database, we start training and generate the final detector.

[0037] Then, we introduce the testing process:

[0038] Step 1: Prepare the image to be tested. For a given image, feature extraction is performed based on the proposed residual codec parallel network REDNet. Then, the two-stage object detector Faster R-CNN is used to extract and classify candidate detection windows.

[0039] Step 2: Based on the results output by the detector, we use the non-maximum suppression algorithm NMS to merge the detection results, remove redundant detection results and generate the final detection results.

[0040] Figure 3 The specific implementation method of the method proposed in the present invention is given, and the main steps are as follows:

[0041] Step 1: Select or collect the corresponding target detection dataset according to the application scenario. The dataset contains a certain number of images and the object annotation information corresponding to these images. The annotation information mainly refers to the location information of the object, such as the coordinates of the upper left corner and the lower right corner, and the category information to which the object belongs.

[0042] Step 2: Select the backbone network and configure the proposed residual codec parallel network according to the backbone network. Then, embed the set residual codec parallel network into the object detector.

[0043] Step 3: Set the network parameters of the residual codec parallel network and the detector, as well as some hyperparameters required for the training process, such as the number of iterations, learning rate, number of positive and negative samples, loss function, etc.

[0044] Step 4: Based on the given hyperparameters, use the back propagation algorithm to update the network weight parameters to obtain the final detector.

[0045] Step 5: For a test image, use the trained detector to detect the image and generate the corresponding detection result, and further use the non-maximum suppression algorithm to obtain the final target detection result.

Claims

1. A target detection method based on a residual codec parallel network, comprising a training phase and a testing phase, wherein the training phase comprises the following steps: (1) Prepare a training dataset for object detection, which includes images for training and annotations of objects in the images. The annotations include the coordinates of the detection box where the object is located and the category to which the object belongs. (2) Select the basic network model; (3) Establishing a residual encoding and decoding unit for processing a given input feature map, the method is as follows: performing the first encoding and decoding operation, that is, downsampling the input feature map by 1 times through a convolutional layer with a step size of 2, and then using an upsampling operation to restore the downsampled features to the size of the input feature map; for the feature map outputted for the first time, continuing to perform the second encoding and decoding operation to generate a new feature map; the feature map outputted by the first encoding and decoding operation is treated as a residual and added to the feature map outputted by the current encoding and decoding operation to obtain a feature map after residual addition; for the feature map after residual addition, continuing to perform the third encoding and decoding operation to generate a new output feature map again, and adding the new output feature map generated this time to the output of the second encoding and decoding operation to obtain the final output feature map; (4) Construct a residual codec parallel network REDNet. The method is as follows: Based on the established residual codec unit, the proposed residual codec unit is added to different convolution blocks of the basic network model, and a parallel residual codec unit is added next to the convolution block. The convolution layer in the convolution block of the parallel basic network model and the convolution layer in the residual codec unit share weights to obtain the residual codec parallel network REDNet. (5) Given a two-stage target detector; (6) Embed the residual encoder-decoder parallel network REDNet into the given two-stage target detector, train and generate the final detector.

Images