Object detection method based on convolutional neural network

Abstract

The present invention discloses a target detection method based on a convolutional neural network, which mainly solves the problems of the existing target detection method, namely, the network is complex, the speed is slow, and the detection effect of small targets is poor. The specific steps of the present invention are as follows: (1) constructing a convolutional neural network; (2) generating a target detection training set; (3) training the convolutional neural network; (4) detecting the target to be detected. The present invention constructs a convolutional neural network composed of a feature extraction module, a feature enhancement module, and a recognition and positioning module, which overcomes the problem that the existing target detection method has a good detection effect for large targets, a poor detection effect for small targets, and a slow speed, so that the present invention can not only recognize large targets in real time, but also recognize small targets in real time.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, relates to image detection, and in particular to a target detection method based on a convolutional neural network, which can be used to detect stationary targets in natural images. Background Art

[0002] A large number of existing target detection methods have achieved very high scores in public datasets, but there are still many challenges in target detection in real-world tasks, such as poor detection of small targets. For example, in intelligent sports teaching, detecting the ball will help coaches grasp the accuracy of students playing the ball. However, when the camera is placed in the sports room, the ball only occupies a few pixels, and because the small ball in motion will be deformed or blocked due to some specific actions of the athletes, and there are also different lighting, motion blur and other influencing factors, the existing target detection network is not suitable for target recognition scenarios similar to this small ball recognition.

[0003] Yue Wenzhen et al. disclosed a target detection method and a target detection network in their patent document "A target detection method and a target detection network" (application number CN201910702231.X, publication number CN110428002A). The method first prepares a target detection training set, which includes a number of material images; establishes a target detection network, including: establishing a network structure, designing a feature map, and establishing a loss function; using the training material images as materials to train the target detection network; and using the trained target detection network to perform target detection. Although this method has made improvements to the network structure, feature map, and loss function, respectively, taking into account both accuracy and efficiency, the method still has the following shortcomings: there is no feature enhancement module for small targets. For small target detection with fewer pixels or occlusions in the image, this method is not very effective because its extracted features are not obvious.

[0004] Wei Liu et al. published a method for image target detection SSD in their paper (SSD: Single Shot MultiBox Detector). The network uses VGG16 as the base model, converts the fully connected layers FC6 and FC7 of VGG16 into 3x3 convolutional layers Conv6 and 1x1 convolutional layers Conv7, removes all Dropout layers and FC8 layers, and changes the pooling layer pool5 from the original 2x2 with stride = 2 to 3x3 with stride = 1. The Atrous algorithm (hole algorithm) is added to obtain a denser score map, and then a new convolutional layer is added on the basis of VGG16 to obtain more feature maps for detection. This method belongs to the one-stage method and is a multi-box prediction. Although the running speed of this network is comparable to that of YOLO, and the detection accuracy is comparable to that of Faster RCNN, the method still has the following shortcomings: the basic size and shape of the prior box in the network cannot be directly obtained through learning, but need to be set manually. However, the size and shape of the prior boxes used by each layer of features in the network are different, which makes the debugging process highly dependent on experience. In addition, the recall rate of this network for small targets is not high, resulting in low final detection accuracy. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a target detection method based on convolutional neural network to solve the problems of complex network, slow speed and poor detection effect on small targets in the existing target detection method.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] Object detection methods based on convolutional neural networks include:

[0008] Step 1: Build a convolutional neural network

[0009] The convolutional neural network is composed of a feature extraction module, a feature enhancement module, and an identification and positioning module connected in sequence; the feature extraction module is used to extract target features, the feature enhancement module is used to enhance the extracted target features, and the identification and positioning module is used to identify and locate the enhanced target features;

[0010] Step 2: Generate target detection training set

[0011] The target detection training set consists of a plurality of images to be detected containing targets, and the targets in each image to be detected are marked. The marking is to mark the coordinates of each vertex of each circumscribed rectangular box where the target is located and the category it represents;

[0012] Step 3: training the convolutional neural network

[0013] Step 4: Detect the target to be detected

[0014] The image containing the target to be detected is input into the trained convolutional neural network, and the detection results of the category of the target to be detected in the image and the position of each circumscribed rectangular box where the target is located are output.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] First, since the present invention constructs a convolutional neural network consisting of a feature extraction module, a feature enhancement module, and a recognition and positioning module, and since the network has good feature extraction performance and the feature enhancement module has a more accurate recognition ability for the features of small targets, it overcomes the problem that the existing target detection method has good large target detection effect and poor small target detection effect, and is particularly suitable for detecting a large number of weak targets in a complex background.

[0017] Second, since the present invention constructs an end-to-end target detection network with a relatively simple overall network structure and overcomes the problem of slow detection speed of existing target detection methods, the present invention can quickly detect vehicles or pedestrians in real time and realize the function of real-time target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the process of the present invention.

[0019] Figure 2 It is the feature extraction module and feature enhancement module of the present invention.

[0020] Figure 3 It is the specific vehicle or pedestrian detection result of the present invention. DETAILED DESCRIPTION

[0021] The present invention is further described below in conjunction with the accompanying drawings.

[0022] The present invention is a target detection method based on convolutional neural network, which uses a convolutional neural network to extract and enhance target features, and corresponds the vector output by the network to the category and position, so as to identify and locate the targets in the image data set, so that the network can accurately detect each target in the image. Figure 1 , the present invention specifically comprises the following steps:

[0023] Step 1: Build a convolutional neural network.

[0024] The convolutional neural network of the present invention is composed of a feature extraction module, a feature enhancement module, and a recognition and positioning module connected in sequence. Among them:

[0025] The feature extraction module is used to extract target features, refer to Figure 2 In the embodiment of the present invention, there are 12 layers in total, and the structure is: Focus layer→first convolution layer→first CSP layer→second convolution layer→second CSP layer→third convolution layer→third CSP layer→fourth convolution layer→fourth CSP layer→fifth convolution layer→pooling layer→fifth CSP layer.

[0026] The parameters of each layer are set as follows: the number of convolution kernels in the first to fifth convolution layers is set to 64, 128, 256, 512, 768 respectively, the size of the convolution kernel is set to 3×3, the step size is set to 2, and each convolution layer performs convolution operation, batch normalization and leaky ReLU activation processing; the Focus layer performs a downsampling and convolution operation. When downsampling, the width and height are reduced to half of the original, and the number of channels is increased to four times the original. The convolution kernel size of the convolution operation is 1×1, and the step size is 1; each CSP layer has two-way operation, the upper way is through the convolution operation and then through the residual structure, and the lower way is a convolution operation. The two ways are spliced ​​and then through the convolution operation. The convolution kernel size is set to 3×3 each time, and the step size is set to 2; the pooling layer has four ways, one is input, and the sizes of the pooling area kernels of the other three ways are 5×5, 9×9, and 13×13 respectively, and the step size is set to 2. The maximum pooling method is adopted. The convolution operation is performed after the four-way structure is spliced. The convolution kernel size is 3×3 and the step size is 2. Normalization processing is to normalize each batch of data. For example, the normalization processing method can use min-batch SGD.

[0027] This method can achieve the downsampling effect well by using the Focus layer, and can reduce the number of parameters while widening the number of channels. It can extract the features of the detection target in the image well by using multiple convolutional layers and CSP layers. Using the pooling layer instead of the fully connected layer can greatly reduce the amount of data and network parameters, thereby improving the detection speed.

[0028] The feature enhancement module is used to enhance the extracted target features. Figure 2 In the embodiment of the present invention, there are 21 layers in total, and the structure is as follows: the sixth convolution layer → the first upsampling layer → the first channel splicing layer → the sixth CSP layer → the seventh convolution layer → the second upsampling layer → the second channel splicing layer → the seventh CSP layer → the eighth convolution layer → the third upsampling layer → the third channel splicing layer → the eighth CSP layer → the ninth convolution layer → the fourth channel splicing layer → the ninth CSP layer → the tenth convolution layer → the fifth channel splicing layer → the tenth CSP layer → the eleventh convolution layer → the sixth channel splicing layer → the eleventh CSP layer.

[0029] Among them, the convolution kernel sizes of the sixth convolution layer, the seventh convolution layer, and the eighth convolution layer are all 1×1, and the step size is 1. The convolution kernel sizes of the ninth convolution layer, the tenth convolution layer, and the eleventh convolution layer are all 3×3, and the step size is 2. The structure of each CSP layer is the same as the structure of the CSP layer in the feature extraction module. The input of the first channel splicing layer is the output of the first upsampling layer and the fourth CSP layer, the input of the second channel splicing layer is the output of the second upsampling layer and the third CSP layer, the input of the third channel splicing layer is the output of the third upsampling layer and the second CSP layer, and the input of the fourth channel splicing layer is the output of the eighth and ninth convolution layers.

[0030] This method can generate feature maps of various sizes through multiple convolutional layers, upsampling layers and channel splicing layers. The multi-scale features generated by this network are richer than those generated by existing networks, enabling better feature fusion capabilities for small targets.

[0031] The recognition and positioning module is used to recognize and locate the enhanced target features. In the embodiment of the present invention, there are 4 layers in total, and its structure is four convolutional layers, namely: the twelfth convolutional layer, the thirteenth convolutional layer, the fourteenth convolutional layer, and the fifteenth convolutional layer.

[0032] Among them, the inputs of the twelfth convolutional layer, the thirteenth convolutional layer, the fourteenth convolutional layer, and the fifteenth convolutional layer are connected to the outputs of the eighth CSP layer, the ninth CSP layer, the tenth CSP layer, and the eleventh CSP layer, respectively. The number of convolution kernels in each layer is set to 256, 512, 768, and 1024, respectively, the size of the convolution kernel is set to 1×1, and the step size is set to 1.

[0033] This module processes the feature maps output by the feature enhancement module. After passing through four convolutional layers, it can generate feature maps of 20*20, 40*40, 80*80 and 160*160. These feature maps can not only accurately detect large targets, but also detect small targets well, greatly improving the small target detection performance.

[0034] Finally, the feature extraction module, the feature enhancement module, the recognition and positioning module are connected in sequence to form the convolutional neural network of the present invention.

[0035] The function of the input volume of each convolutional layer of the present invention is as follows:

[0036] V in =H1*W1*D1

[0037] Since each layer of the network will output a feature map, H1, W1 and D1 are the height, width and number of output channels of the feature map output by the previous layer of the network;

[0038] The function of the output volume of the convolutional layer is:

[0039] V out =H2*W2*D2

[0040] Among them, H2, W2 and D2 are the height, width and number of output channels of the feature map output by this layer of network.

[0041] The formulas for H2, W2, and D2 are as follows:

[0042] H2=(H1-F+2*P) / S+1

[0043] W2=(W1-F+2*P) / S+1

[0044] D2=K

[0045] Among them, F is the size of the convolution kernel, S is the step size, P is the size of zero padding, and K is the number of convolution kernels.

[0046] The function of batch normalization processing of the present invention is as follows:

[0047]

[0048] Among them, z i is the i-th feature weight before updating, s i represents the updated i-th feature weight, and C is the number of channels. Based on this setting, the feature information in the image can be well extracted through each convolution layer operation.

[0049] Step 2: Generate target detection training set.

[0050] The target detection training set of the present invention is composed of a plurality of images to be detected containing targets, and the targets in each image to be detected are annotated, that is, the coordinates of each vertex of each circumscribed rectangular box where the target is located and the category it represents are annotated. For example, a picture can be captured every 10 frames from a continuously shot video containing the target to be detected as a picture to be detected, and a total of 10,000 pictures can be captured, and the acquired pictures can be annotated. For example, the target category to be detected in the picture can be annotated with labelImg software, and the coordinates and category of the annotated box can be determined, that is, the target detection training set is formed.

[0051] In one embodiment of the present invention, the target detection training set is generated by the following method:

[0052] The first step is to form a picture data set with at least 10,000 pictures of vehicles or pedestrians to be detected. Each picture to be detected is captured every 10 frames from the continuously shot videos containing vehicles or pedestrians, and a total of 10,000 pictures are captured;

[0053] The second step is to label the targets in each image in the image dataset. Use labelImg software to label the bus, truck, car, and person in the image, determine the coordinates and category of the label box, and form a target detection training set.

[0054] Step 3: Train the convolutional neural network:

[0055] The training set is input into the convolutional neural network, and the weights of the convolutional neural network are updated using the gradient descent method until the Loss value drops below 3.0 to obtain a trained convolutional neural network.

[0056] The present invention defines the Loss function as:

[0057] Loss=Loss box +Loss obj +Loss cls

[0058] Among them, Loss box To evaluate the position loss of the target box and the predicted box, Loss obj To evaluate the confidence loss of the target box and the predicted box, Loss cls To evaluate the category loss of the target box and the predicted box;

[0059] Loss box The formula is as follows:

[0060]

[0061] Among them, IOU is the intersection-union ratio of the predicted box and the real box, Distance_2 is the Euclidean distance between the two center points of the predicted box and the real box, Distance_C is the diagonal distance between the predicted box and the real box, and v is a parameter to measure the consistency of the aspect ratio, which is defined as:

[0062]

[0063]

[0064] Among them, A and B are the areas of the predicted box and the real box respectively, and w gt and h gt Indicates the width and height of the real box, w p and w p Indicates the width and height of the prediction box;

[0065] Loss obj The formula is:

[0066]

[0067] in, Indicates whether it is a positive sample, with a value of 0 or 1. Indicates traversing all prediction boxes, S*S=20*20+40*40+80*80+160*160, B=4, Indicates whether it is a negative sample, with a value of 0 or 1. When it is 1, it indicates a negative sample, and when it is 0, it indicates that it is not a negative sample. represents the sample value, C i represents the predicted value;

[0068] Loss cls The formula is:

[0069]

[0070] Where i and j are the indexes of anchors in a mini-batch, i increases from 0 to S*S, j increases from 0 to the value of B, and p i is the probability that the anchor is predicted as the target. When the anchor is the target, is 1, otherwise it is 0. Indicates traversing all prediction boxes, S*S=20*20+40*40+80*80+160*160, B=4, Indicates whether it is a positive sample, with a value of 0 or 1. When it is 1, it indicates a positive sample, and when it is 0, it indicates a non-positive sample. Represents the true probability value of the target, p i (c) represents the target prediction probability value.

[0071] Step 4: Detect the target to be detected:

[0072] Each picture containing a vehicle or pedestrian is input into the trained convolutional neural network in turn, and the detection results of the category of the vehicle or pedestrian in the picture and the position of each circumscribed rectangular box where the target is located are output. Figure 3 is the specific vehicle or pedestrian detection result, where car represents the detected car, truck represents the detected truck, and the number represents the probability. It can be seen that this method can well detect small target vehicles or pedestrians with high detection accuracy, and almost every small target can be detected.

Claims

1. A target detection method based on a convolutional neural network, characterized in that: include: Step 1: Build a convolutional neural network The convolutional neural network is composed of a feature extraction module, a feature enhancement module, and an identification and positioning module connected in sequence; the feature extraction module is used to extract target features, the feature enhancement module is used to enhance the extracted target features, and the identification and positioning module is used to identify and locate the enhanced target features; The feature extraction module has 12 layers in total, and its structure is as follows: Focus layer → first convolution layer → first CSP layer → second convolution layer → second CSP layer → third convolution layer → third CSP layer → fourth convolution layer → fourth CSP layer → fifth convolution layer → pooling layer → fifth CSP layer; the parameters of each layer are set as follows: the number of convolution kernels in the first to fifth convolution layers is set to 64, 128, 256, 512, 768 respectively, the size of the convolution kernel is set to 3×3, the step size is set to 2, and each convolution layer performs convolution operation, batch normalization and Leaky ReLU activation processing; the Focus layer performs a downsampling and convolution operation, and the width and height are reduced to half of the original during downsampling, the number of channels is increased to four times the original, and the convolution kernel size of the convolution operation is 1×1, and the step size is 1; each CSP layer has two-way operation, the upper way is through the convolution operation and then through the residual structure, and the lower way is a convolution operation. The two ways are spliced ​​and then through the convolution operation. The size of each convolution kernel is set to 3×3, and the step size is set to 2; the pooling layer has four ways, one is input, and the sizes of the pooling area kernels of the other three ways are 5×5, 9×9, and 13×13 respectively, and the step size is set to 2, and the maximum pooling method is adopted. After the four-way structure is spliced, the convolution operation is performed, the convolution kernel size is 3×3, and the step size is 2. Normalization is performed on each batch of data, and the processing method used is min-batch SGD; The feature enhancement module has a total of 21 layers, and its structure is as follows: the sixth convolution layer → the first upsampling layer → the first channel splicing layer → the sixth CSP layer → the seventh convolution layer → the second upsampling layer → the second channel splicing layer → the seventh CSP layer → the eighth convolution layer → the third upsampling layer → the third channel splicing layer → the eighth CSP layer → the ninth convolution layer → the fourth channel splicing layer → the ninth CSP layer → the tenth convolution layer → the fifth channel splicing layer → the tenth CSP layer → the eleventh convolution layer → the sixth channel splicing layer → the eleventh CSP layer; among which the sixth convolution layer, the seventh convolution layer and the eighth convolution layer The convolution kernel size is 1×1, and the step size is 1. The convolution kernel size of the ninth, tenth, and eleventh convolution layers is 3×3, and the step size is 2. The structure of each CSP layer is the same as the structure of the CSP layer in the feature extraction module. The input of the first channel splicing layer is the output of the first upsampling layer and the fourth CSP layer, the input of the second channel splicing layer is the output of the second upsampling layer and the third CSP layer, the input of the third channel splicing layer is the output of the third upsampling layer and the second CSP layer, and the input of the fourth channel splicing layer is the output of the eighth and ninth convolution layers. The recognition and positioning module has a total of 4 layers, and its structure is four convolutional layers, namely: the 12th convolutional layer, the 13th convolutional layer, the 14th convolutional layer, and the 15th convolutional layer; wherein the inputs of the 12th convolutional layer, the 13th convolutional layer, the 14th convolutional layer, and the 15th convolutional layer are respectively connected to the outputs of the 8th CSP layer, the 9th CSP layer, the 10th CSP layer, and the 11th CSP layer, the number of convolution kernels of each layer is set to 256, 512, 768, and 1024, respectively, the size of the convolution kernel is set to 1×1, and the step size is set to 1; Step 2: Generate target detection training set The target detection training set consists of a plurality of images to be detected containing targets, and the targets in each image to be detected are marked. The marking is to mark the coordinates of each vertex of each circumscribed rectangular box where the target is located and the category it represents; Step 3: training the convolutional neural network Step 4: Detect the target to be detected The image containing the target to be detected is input into the trained convolutional neural network, and the detection results of the category of the target to be detected in the image and the position of each circumscribed rectangular box where the target is located are output.

2. The target detection method based on convolutional neural network according to claim 1, characterized in that: The function of the input volume of each convolutional layer is as follows: In in =H1*W1*D1 Each layer of the network will output a feature map, H1, W1 and D1 are the height, width and number of output channels of the feature map output by the previous layer of the network; The function of the output volume of the convolutional layer is: <h2 style=";text-align:left;direction:ltr">V<h2 style=";text-align:left;direction:ltr"> out <h2 style=";text-align:left;direction:ltr"> (H2*W2*D2) Among them, H2, W2 and D2 are the height, width and number of output channels of the feature map output by this layer of network; The formulas for H2, W2, and D2 are as follows: H2=(H1-F+2*P) / S+1 W2=(W1-F+2*P) / S+1 D2=K Among them, F is the size of the convolution kernel, S is the step size, P is the size of zero padding, and K is the number of convolution kernels.

3. The target detection method based on convolutional neural network according to claim 1, characterized in that: The function of the batch normalization process is as follows: Among them, z i is the i-th feature weight before updating, s i represents the updated i-th feature weight, and C is the number of channels.

4. The target detection method based on convolutional neural network according to claim 1, characterized in that: Each of the to-be-detected pictures is captured every 10 frames from a continuously shot video containing the to-be-detected object, and a total of 10,000 pictures are captured.

5. The target detection method based on convolutional neural network according to claim 2, characterized in that: In step 2, the acquired images are labeled, the target categories to be detected in the images are labeled using labelImg software, the coordinates and categories of the labeled boxes are determined, and a target detection training set is formed.

6. The target detection method based on convolutional neural network according to claim 1, characterized in that: In step 3, the target detection training set is input into the convolutional neural network, and the weights of the convolutional neural network are updated using the gradient descent method until the Loss value drops below 3.0, thereby obtaining a trained convolutional neural network.

7. The target detection method based on convolutional neural network according to claim 6, characterized in that: Define the Loss function as: Loss=Loss box +Loss obj +Loss cls Among them, Loss box To evaluate the position loss of the target box and the predicted box, Loss obj To evaluate the confidence loss of the target box and the predicted box, Loss cls To evaluate the category loss of the target box and the predicted box; Loss box The formula is as follows: Among them, IOU is the intersection-union ratio of the predicted box and the real box, Distance_2 is the Euclidean distance between the two center points of the predicted box and the real box, Distance_C is the diagonal distance between the predicted box and the real box, and v is a parameter to measure the consistency of the aspect ratio, which is defined as: Among them, A and B are the areas of the predicted box and the real box respectively, and w gt and h gt Indicates the width and height of the real box, w p and w p Indicates the width and height of the prediction box; Loss obj The formula is: in, Indicates whether it is a positive sample, the value is 0 or 1, Indicates traversing all prediction boxes, S*S=20*20+40*40+80*80+160*160, B=4, Indicates whether it is a negative sample, with a value of 0 or 1. represents the sample value, C i represents the predicted value; Loss cls The formula is: Where i and j are the indexes of anchors in a mini-batch, i increases from 0 to S*S, j increases from 0 to the value of B, and p i is the probability that the anchor is predicted as the target. When the anchor is the target, is 1, otherwise it is 0. Indicates traversing all prediction boxes, S*S=20*20+40*40+80*80+160*160, B=4, Indicates whether it is a positive sample, the value is 0 or 1, Represents the true probability value of the target, p i (c) represents the target prediction probability value.

Images