Target detection and classification method and computer readable storage medium
By building a neural network model containing deep residual network and feature pyramid network, combining data augmentation and non-maximum suppression, the detection problems of small and medium-sized targets and high-blocking vehicles in the logistics park are solved, and the accurate detection and classification of vehicles in the logistics park are achieved. , improve management efficiency.
Patent Information
- Application Number
- CN202111137707.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-27
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2041-09-27
AI Technical Summary
The prior art is difficult to accurately detect and classify small targets and high-blocking logistics vehicles in logistics parks, especially because the vehicle is small in size and easily confused with the background in the video images captured by the surveillance camera, which makes it difficult to detect.
A target detection and classification method is adopted, including building a neural network model, using deep residual network, feature pyramid network, regional candidate network and classification and regression subnet modules, through feature extraction, feature fusion and anchor box generation, combined with data enhancement and non-maximum suppression to improve detection accuracy of small-size and high-occlusion targets.
Accurate detection and classification of small-size and high-shading targets is achieved, detection capabilities are enhanced, calculation costs are reduced, and management efficiency is improved in logistics park vehicle management.
Smart Images
Figure CN113887602B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of target recognition technology, and in particular to a target detection and classification method and a computer-readable storage medium. Background Art
[0002] In recent years, with the development of transportation and logistics, more and more logistics vehicles serve people's work and life, resulting in a large number of logistics vehicles entering or parking in the logistics park every day, which greatly increases the difficulty of the park's management of logistics vehicles. If the type of logistics vehicles can be identified, it will be convenient for the logistics park to manage vehicles.
[0003] Currently, logistics parks are equipped with surveillance cameras, which can be used to identify vehicles through video images captured by surveillance cameras. However, there are often many vehicle targets in the park, and because surveillance cameras are generally set at high places and have a long depth of field, some vehicles occupy a very small proportion of the picture, that is, there is a problem of small targets, and the color and model of small targets are easily confused with background elements, which increases the difficulty of vehicle detection and classification. At the same time, there is also a phenomenon that a large proportion of some vehicles are occluded, that is, there is a problem of strong occlusion, which results in limited information available to the target, and also increases the difficulty of detection and classification.
[0004] In addition, some of the targets to be identified are easily confused with each other, which further increases the difficulty of classification. For example, trucks need to be classified into fine-grained identification of barge type, plate type and box type. Pedestrians, motorcycles and bicycles are very similar in small targets. The backs of tricycles and vans are very similar. The front and sides of tricycles and motorcycles are very similar. Some models of vans, such as mini-vans and sedans, are very similar.
[0005] Although there are many vehicle detection methods, they are unable to perform well or the detection effect is greatly reduced for the detection scenarios of small targets and highly occluded multi-type vehicle targets. Therefore, how to accurately detect and classify small targets and highly occluded logistics park vehicles has become a problem to be solved. Summary of the invention
[0006] The technical problem to be solved by the present invention is to provide a target detection and classification method and a computer-readable storage medium, which can realize the detection and classification of small-sized and highly occluded targets.
[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: a target detection and classification method, comprising:
[0008] Acquire a sample set, wherein the sample set includes a plurality of sample images containing the target to be measured;
[0009] Annotate the target to be measured in each sample image respectively to obtain an annotation file corresponding to each sample image, wherein the annotation file includes the bounding box coordinates of the target to be measured and the category of the target to be measured;
[0010] Constructing a neural network model, and training the neural network model according to the sample images and their corresponding annotation files;
[0011] Obtaining a picture to be tested, and inputting the picture to be tested into a trained neural network model to obtain a detection result;
[0012] The neural network model includes a deep residual network module, a feature pyramid network module, a region candidate network module, and a classification and regression sub-network module;
[0013] The deep residual network module is used to extract features from the input image, obtain C1 feature map, C2 feature map, C3 feature map, C4 feature map and C5 feature map from bottom to top, and output the C2 feature map, C3 feature map, C4 feature map and C5 feature map to the feature pyramid network;
[0014] The feature pyramid network module is used to perform feature extraction, feature fusion and convolution processing on the C2 feature map, the C3 feature map, the C4 feature map and the C5 feature map to obtain the P2 feature map, the P3 feature map, the P4 feature map and the P5 feature map, and perform a 3×3 convolution processing with a step size of 2 on the C5 feature map to obtain the P6 feature map;
[0015] The region candidate network module is used to map the anchor frame in the input image according to the P2 feature map, the P3 feature map, the P4 feature map, the P5 feature map and the P6 feature map, and generate a candidate frame corresponding to each anchor frame according to the preset n scales and m aspect ratios, where m is greater than 3;
[0016] The classification and regression subnetwork module is used to filter and merge the candidate boxes according to the preset elimination score, intersection-over-union ratio threshold and score ranking to obtain the predicted box and its category.
[0017] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method described above when the program is executed by a processor.
[0018] The beneficial effects of the present invention are as follows: in a feature pyramid network module, by increasing the use of a P2 feature map, the detection of small-sized targets and strongly occluded targets can be achieved, and by reducing the use of a P7 feature map, the computing cost can be saved, thereby enhancing the detection capability of smaller targets, and the detection capability of larger targets does not decrease significantly; in a region candidate network module, by setting more aspect ratios to generate denser prediction boxes, the missed detection of small-sized targets and strongly occluded targets can be avoided; in a classification and regression subnetwork module, by setting appropriate non-maximum suppression parameters, it can be better applied to small-sized and highly occluded target detection and classification scenarios, thereby improving the accuracy of detection and classification. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 A flow chart of a target detection and classification method of the present invention;
[0020] Figure 2 This is a flow chart of a method according to Embodiment 1 of the present invention;
[0021] Figure 3 This is a schematic diagram of a bounding box marked in a sample image according to the first embodiment of the present invention;
[0022] Figure 4 The annotation files before and after the update of the first embodiment of the present invention;
[0023] Figure 5 This is a schematic diagram of the detection results of Example 1 of the present invention;
[0024] Figure 6 This is a schematic diagram of the existing RetinaNet structure;
[0025] Figure 7 It is a structural diagram of the existing deep residual network ResNet;
[0026] Figure 8 This is a schematic diagram of the feature extraction network structure of the existing feature pyramid network FPN;
[0027] Fig. 9 This is a schematic diagram of a feature extraction network structure according to Embodiment 1 of the present invention;
[0028] Fig.10 This is a schematic diagram of the structure of a neural network model according to the first embodiment of the present invention;
[0029] Fig.11 This is a comparison chart of COCOs evaluation indicators of the neural network model of Example 1 of the present invention and five existing models. DETAILED DESCRIPTION
[0030] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is a detailed description in conjunction with the implementation methods and the accompanying drawings.
[0031] See also Figure 1 , a target detection and classification method, comprising:
[0032] Acquire a sample set, wherein the sample set includes a plurality of sample images containing the target to be measured;
[0033] Annotate the target to be measured in each sample image respectively to obtain an annotation file corresponding to each sample image, wherein the annotation file includes the bounding box coordinates of the target to be measured and the category of the target to be measured;
[0034] Constructing a neural network model, and training the neural network model according to the sample images and their corresponding annotation files;
[0035] Obtaining a picture to be tested, and inputting the picture to be tested into a trained neural network model to obtain a detection result;
[0036] The neural network model includes a deep residual network module, a feature pyramid network module, a region candidate network module, and a classification and regression sub-network module;
[0037] The deep residual network module is used to extract features from the input image, obtain C1 feature map, C2 feature map, C3 feature map, C4 feature map and C5 feature map from bottom to top, and output the C2 feature map, C3 feature map, C4 feature map and C5 feature map to the feature pyramid network;
[0038] The feature pyramid network module is used to perform feature extraction, feature fusion and convolution processing on the C2 feature map, the C3 feature map, the C4 feature map and the C5 feature map to obtain the P2 feature map, the P3 feature map, the P4 feature map and the P5 feature map, and perform a 3×3 convolution processing with a step size of 2 on the C5 feature map to obtain the P6 feature map;
[0039] The region candidate network module is used to map the anchor frame in the input image according to the P2 feature map, the P3 feature map, the P4 feature map, the P5 feature map and the P6 feature map, and generate a candidate frame corresponding to each anchor frame according to the preset n scales and m aspect ratios, where m is greater than 3;
[0040] The classification and regression subnetwork module is used to filter and merge the candidate boxes according to the preset elimination score, intersection-over-union ratio threshold and score ranking to obtain the predicted box and its category.
[0041] From the above description, it can be seen that the beneficial effect of the present invention is that it can realize the detection and classification of small-sized and highly occluded targets.
[0042] Furthermore, after labeling the target to be measured in each sample image respectively and obtaining a labeling file corresponding to each sample image, the method further includes:
[0043] Each sample image is scaled to a preset size, and each annotation file is updated accordingly according to the scaled sample image.
[0044] From the above description, it can be seen that the GPU utilization rate can be improved by adjusting the size of the sample image.
[0045] Furthermore, after labeling the target to be measured in each sample image respectively and obtaining a labeling file corresponding to each sample image, the method further includes:
[0046] Perform data enhancement processing on the sample set.
[0047] From the above description, it can be seen that by performing data enhancement processing, the versatility of the data set is increased and overfitting is prevented.
[0048] Furthermore, the training of the neural network model according to the sample images and their corresponding annotation files is specifically as follows:
[0049] Traverse the sample set and obtain a sample image in sequence;
[0050] Inputting the sample image into the neural network model to obtain a prediction result corresponding to the sample image, wherein the prediction result includes a prediction box and its category;
[0051] Calculate the loss value corresponding to the sample image according to the prediction result and the annotation file corresponding to the sample image;
[0052] According to the loss value, updating the neural network model through a back propagation algorithm;
[0053] After traversing the sample set, the average loss value corresponding to each sample image is calculated to obtain the average loss value;
[0054] Determine whether the average loss value is less than a preset threshold;
[0055] If not, continue to execute the step of traversing the sample set and obtaining a sample image in sequence;
[0056] If yes, the training is ended and the trained neural network model is obtained.
[0057] From the above description, we can see that when the loss value is lower than a certain threshold, the training ends.
[0058] Furthermore, the training of the neural network model according to the sample images and their corresponding annotation files is specifically as follows:
[0059] Traversing the sample set, and obtaining a sample image from the sample set in sequence;
[0060] Inputting the sample image into the neural network model to obtain a prediction result corresponding to the sample image, wherein the prediction result includes a prediction box and its category;
[0061] Calculate the loss value corresponding to the sample image according to the prediction result and the annotation file corresponding to the sample image;
[0062] According to the loss value, updating the neural network model through a back propagation algorithm;
[0063] After traversing the sample set, the number of training steps is increased by one, and the initial value of the number of training steps is 0;
[0064] Determining whether the number of training steps reaches the predicted number;
[0065] If not, continue to execute the step of traversing the sample set and sequentially acquiring a sample image from the sample set;
[0066] If yes, the training is ended and the trained neural network model is obtained.
[0067] From the above description, we can see that when the number of training steps reaches a certain number, the training ends.
[0068] Furthermore, the loss value corresponding to the sample image is calculated based on the prediction result and the annotation file corresponding to the sample image as follows:
[0069] According to the prediction result and the annotation file corresponding to the sample image, the classification loss value and the regression loss value are calculated by the Focal Loss function and the Smooth L1 Loss function respectively, and the classification loss value and the regression loss value are added to obtain the loss value corresponding to the sample image.
[0070] Furthermore, the feature pyramid network is specifically used for:
[0071] Performing convolution processing with a convolution kernel of 1×1 on the C5 feature map to obtain an M5 feature map;
[0072] A feature map obtained by performing convolution processing with a convolution kernel of 1×1 on the C4 feature map is subjected to feature fusion with a feature map obtained by upsampling the M5 feature map by a factor of 2, to obtain an M4 feature map;
[0073] A feature map obtained by performing convolution processing with a convolution kernel of 1×1 on the C3 feature map is fused with a feature map obtained by upsampling the M4 feature map by a factor of 2 to obtain an M3 feature map;
[0074] A feature map obtained by performing convolution processing with a convolution kernel of 1×1 on the C2 feature map is subjected to feature fusion with a feature map obtained by upsampling the M3 feature map by a factor of 2, to obtain an M2 feature map;
[0075] Perform convolution processing with a convolution kernel of 3×3 on the M2 feature map, the M3 feature map, the M4 feature map, and the M5 feature map, respectively, to obtain a P2 feature map, a P3 feature map, a P4 feature map, and a P5 feature map;
[0076] The C5 feature map is subjected to a 3×3 convolution process with a step size of 2 to obtain a P6 feature map.
[0077] From the above description, it can be seen that by using the features of FPN layers 2 to 6, the detection capability of smaller targets can be enhanced.
[0078] Furthermore, n=3, the three scales are {2 0 , 2 1 / 3 , 2 2 / 3}, m = 5, the 5 aspect ratios are {2 0 , 2 1 , 2 1 / 2 , 2 1 / 3 , 2 2 / 3}.
[0079] From the above description, it can be seen that by setting more aspect ratios to generate denser prediction boxes, it is possible to avoid missing small-sized objects and strongly occluded objects.
[0080] Furthermore, the elimination score is 9.99999993922529e-09, the intersection-over-union ratio threshold is 0.6, and the score ranking is 100.
[0081] From the above description, it can be seen that by setting appropriate non-maximum suppression parameters, it can be better applied to small-sized and highly occluded target detection and classification scenarios, thereby improving the accuracy of detection and classification.
[0082] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method described above when the program is executed by a processor.
[0083] Embodiment 1
[0084] Please refer to Figure 2-11, Embodiment 1 of the present invention is: a target detection and classification method, which can be applied to the detection and classification of vehicles in a logistics park, such as Figure 2 As shown, the following steps are included:
[0085] S1: Obtain a sample set, wherein the sample set includes a plurality of sample images containing a target to be measured;
[0086] Specifically, we obtain videos collected by different surveillance cameras on different roads in the logistics park, select some representative clips, such as clips with a large number and categories of targets, clips with small targets, clips with strongly occluded targets, etc., and then use ffmpeg to edit the videos and convert them into picture sets.
[0087] By selectively selecting clips from the video to create a sample set, better results can be achieved in detection and recognition.
[0088] In this embodiment, there are a total of 10 types of targets to be detected, namely, truck, flatbed truck, van, motorcycle, school bus, car, bicycle and pedestrians who are not riding bicycles.
[0089] S2: Label the target to be measured in each sample image respectively to obtain a labeling file corresponding to each sample image, wherein the labeling file includes the bounding box coordinates of the target to be measured and the category of the target to be measured, wherein the bounding box coordinates may be the coordinates of the four corners of the bounding box.
[0090] The labeling method can be manual labeling. Specifically, the tool labelImg can be used to label the bounding boxes and categories of various types of targets to be tested in each sample image, and generate a labeling file associated with the sample image. The manually labeled bounding boxes can be as follows: Figure 3 As shown ( Figure 3 Only the four corner points of the bounding box are marked in ). Since manual marking will take a lot of time, a semi-automatic marking method can also be used. Specifically, the bounding boxes and categories of various types of targets to be tested are predicted using existing detection and classification models, and annotation files of associated sample images are generated, and then manual adjustments are made.
[0091] S3: scaling each sample image to a preset size, and updating each annotation file accordingly according to each scaled sample image.
[0092] If the resolution of the original sample image is relatively large, for example, 1920×1080, the GPU utilization rate fluctuates greatly and sometimes is very low when the model is subsequently input for training. Therefore, the size of the sample image needs to be adjusted to improve the GPU utilization rate. In this embodiment, each sample image is scaled to a resolution of 640×640. Since the sample image is scaled, the bounding boxes corresponding to various types of targets to be measured are also changed. Therefore, the bounding box coordinates in the annotation file of each sample image are updated accordingly.
[0093] Furthermore, the annotation file may also include the size and address of the sample image, and this step may also adjust the address of the sample image. Figure 4 As shown, Figure 4 The left side is the annotation file before the update, and the right side is the annotation file after the update. Figure 4 It can be seen that the sample image is named 1, the address (path) is adjusted, the size is changed from 1920×1080 to 640×640, and the four corner coordinates of the bounding box of a target to be tested with a category of Truck are updated from (324, 639, 495, 869) to (108, 378, 165, 514).
[0094] In this embodiment, each sample image and its corresponding annotation file are directly used as a training set, that is, the sample set is directly used as a training set for subsequent training of the neural network model.
[0095] In other embodiments, the sample images and their corresponding annotation files can be used to divide the training set and the validation set. For example, assuming there are 197 sample images, 167 sample images and their corresponding annotation files can be used as the training set, and 30 sample images and their corresponding annotation files can be used as the validation set. Then, they are converted into the tfrecord format accepted by tensorflow.
[0096] Furthermore, in order to increase the versatility of the dataset and prevent overfitting, data enhancement processing is performed on the training set and the validation set, such as random adjustment of contrast, hue, saturation, color blocks, cropping, padding, scaling, integrated color, flipping, etc.
[0097] S4: Construct a neural network model, and train the neural network model according to the sample images and their corresponding annotation files.
[0098] In this embodiment, the neural network model includes a deep residual network module, a feature pyramid network module, a region candidate network module, and a classification and regression sub-network module. By inputting a sample image into the neural network model, the prediction box and its category of the target to be detected in the image can be predicted.
[0099] In this embodiment, the condition for the end of training is that the average loss value is less than a preset threshold, or the number of training steps reaches a preset number.
[0100] When the training end condition is the former, this step specifically includes the following steps:
[0101] S401: traverse the sample set and obtain a sample image from the sample set in sequence.
[0102] S402: Inputting the sample image into the neural network model to obtain a prediction result corresponding to the sample image, wherein the prediction result includes a prediction box and its category, and further includes a classification confidence.
[0103] S403: Calculate a loss value corresponding to the sample image according to the prediction result and the annotation file corresponding to the sample image.
[0104] Specifically, according to the prediction result and the annotation file corresponding to the sample image, the classification loss value and the regression loss value are calculated by the Focal Loss function and the Smooth L1 Loss function respectively, and the classification loss value and the regression loss value are added to obtain the loss value corresponding to the sample image.
[0105] From the following description of RetinaNet50, we can see that after being processed by the regression subnetwork, each true box (i.e., the bounding box in the standard file) corresponds to an Anchor, so that each Anchor knows the true box it is bound to and the corresponding category, and then calculates the regression loss according to the Smooth L1 Loss function.
[0106] After being processed by the classification subnetwork, the classification confidence of each Anchor relative to each category can be obtained. Then, according to the preset hyperparameters α and γ and the category bound to each Anchor, the classification loss is calculated by the Focal Loss function.
[0107] Finally, the classification loss and regression loss are added to get the loss value.
[0108] S404: According to the loss value, the neural network model is updated by a back propagation algorithm, and then the next sample image is obtained from the sample set for forward propagation and reverse update until the sample set is traversed.
[0109] S405: After traversing the sample set, the average value of the loss value corresponding to each sample image is calculated to obtain an average loss value.
[0110] S406: Determine whether the average loss value is less than a preset threshold value. If so, end the training to obtain a trained neural network model; if not, continue to execute step S411, that is, perform the next iterative training.
[0111] When the training end condition is the latter, this step specifically includes the following steps:
[0112] S411: traverse the sample set and obtain a sample image from the sample set in sequence.
[0113] S412: Inputting the sample image into the neural network model to obtain a prediction result corresponding to the sample image, wherein the prediction result includes a prediction box and its category, and further includes a classification confidence.
[0114] S413: Calculate the loss value corresponding to the sample image according to the prediction result and the annotation file corresponding to the sample image; this step is consistent with the above step S403.
[0115] S414: According to the loss value, the neural network model is updated by a back propagation algorithm; then the next sample image is obtained from the sample set for forward propagation and reverse update until the sample set is traversed.
[0116] S415: After traversing the sample set, the number of training steps is increased by one, that is, epochs=epochs+1, where the initial value of epochs is 0.
[0117] S416: Determine whether the number of training steps reaches a preset number. If so, end the training to obtain a trained neural network model; if not, continue to execute step S411, that is, perform the next iterative training.
[0118] S5: Obtain a picture to be tested, and input the picture to be tested into the trained neural network model to obtain a detection result. The detection result may include a detection frame, a category, and a classification confidence of the target to be tested, such as Figure 5 shown.
[0119] Furthermore, for step S4, after selecting a baseline model, the baseline model can be adjusted to obtain a neural network model.
[0120] For example, five target detection and recognition models are selected, namely RetinaNet152 (SSD+ResNet152+FPN), EfficientNet-d1 (SSD+EfficientNet-b1+BiFPN), RetinaNet50 (SSD+ResNet50+FPN), faster_rcnn_resnet50, and faster_rcnn_inception. Then, these five models are trained according to the same training set and validation set, and the average accuracy and recall rate are calculated. By comparing the average accuracy and recall rate of these five models, the benchmark model is determined, and finally RetinaNet50 is selected.
[0121] Then, the RetinaNet50 model is fine-tuned to obtain the neural network model of this embodiment.
[0122] The existing RetinaNet50 learning rate is a constant 0.01, which is reduced by a factor of 10 at 60,000 steps and another factor of 10 at 80,000 steps. The learning rate of stochastic gradient descent with warm restart (SGDR) is used instead. The warm-up learning rate is set to 0.013333000242710114, and it linearly reaches the base learning rate of 0.03999999910593033 after 2,000 steps. After that, cosine decay is entered until 35,000 steps.
[0123] The existing RetinaNet structure is as follows Figure 6 As shown in the figure, it is mainly composed of three parts: deep residual network (Residual network, ResNet), feature pyramid network (Feature Pyramid Network, FPN) and classification and regression sub-network. Among them, the class subnet in the classification and regression sub-network is responsible for classification, and the box subnet is responsible for positioning. After the image passes through ResNet, the features are initially extracted, the features are reorganized in FPN, and finally sent to the classification and regression sub-network to obtain the final detection result.
[0124] The idea of ResNet is to assume that F(x) represents the optimal solution mapping and consider fitting a residual mapping, where F(x) = F(x) + x. The specific implementation method is to add a shortcut connection in the convolutional feedforward network, such as Figure 7 As shown in the figure, the shortcut connection adds the network's own mapping to the output of the superimposed slave layer, which does not add additional parameters to the network and does not affect the complexity of the calculation. This structure of ResNet makes it possible to train deep neural networks.
[0125] In order to strengthen the feature extraction of ResNet, RetinaNet50 uses Feature Pyramid Network (FPN) to combine high-level semantic features with low-level detail features on targets of various scales, thereby enhancing the network's ability to express features. Each layer of the pyramid can be used to detect objects of different sizes. Based on the characteristics of image samples, which have less semantic information in low-level features but accurate target positions, and more semantic information in high-level features but rough target positions, FPN fuses feature maps of different layers through bottom-up and top-down connections, making small targets easy to identify. The layers of the pyramid are P3 to P7, where Pi represents the layer of the pyramid (Pi represents a value 2 smaller than the original input). i resolution). Figure 8 As shown in the figure, the existing FPN generates P3, P4, P5 and P6 feature maps based on C2, C3, C4 and C5 output by ResNet, and generates P7 feature map based on P6 feature map, where "0.5x" is used to generate feature maps C of different resolutions, "2x" means two times upsampling, aligning the network to ensure the fusion generation of the intermediate layer M, "3×3" and "1×1" represent the size of the convolution kernel, and "s=2" represents the convolution with a stride of 2. It can be seen that P3-P5 is calculated based on C3-C5, P6 is obtained by performing a 3x3 convolution with a stride of 2 on C5, and P7 is obtained by using ReLU and performing a 3x3 convolution with a stride of 2 on P6. Large feature maps predict small objects, and small feature maps predict large objects. By adding P6 and P7, the model can better detect large objects, thanks to a larger receptive field. This top-down and lateral connection method can ensure that each layer has a suitable resolution and strong semantic features, and cooperate with the target detection algorithm to improve the detection performance of objects.
[0126] From the above description, it can be seen that the FPN in the existing RetinaNet50 uses the features of its 3rd to 7th layers. For the application scenario of this embodiment, it is necessary to detect targets that are severely occluded and occupy a very small proportion of the image, but it is not necessary to detect targets that occupy a very large proportion of the image. Among them, the size of small targets and strongly occluded targets is smaller than the minimum feature that can be extracted by P3, and there are almost no targets larger than the size of P6. Therefore, in this embodiment, the features of FPN layers 2 to 6 are used, such as Fig. 9 As shown in the figure, by increasing the use of P2 feature maps, the network's receptive field range is increased, the use of P7 feature maps is reduced, and the computational cost is saved. In the end, the detection ability of smaller targets is enhanced, and the detection ability of larger targets is not significantly reduced.
[0127] After obtaining the above P3-P7 feature maps, the last step of the existing RetinaNet50 is to enter the classification and regression subnetwork to process the prediction of target classification and target positioning respectively.
[0128] The mapping of the center of the P feature map output by FPN to the original image is called the basic prediction box. A point on the feature map corresponds to several basic prediction boxes of different scales on the original image. Then, based on the basic prediction boxes, dense prediction box anchors with different aspect ratios are generated. In RetinaNet50, there are 3 scales, namely {2 0 , 2 1 / 3 , 2 2 / 3}, and 3 aspect ratios, namely {2 0 , 2 1 , 2 1 / 2}.
[0129] RetinaNet50 is a one-stage network that integrates detection and classification. The density of the initial prediction box greatly affects the accuracy. For the application scenario of this embodiment, there are strongly occluded targets, and more dense prediction box anchors need to be generated to avoid missing targets. Therefore, in this embodiment, the number of aspect ratios needs to be greater than 3. For example, 5 aspect ratios are preset, namely {2 0 , 2 1 , 2 1 / 2 , 2 1 / 3 , 2 2 / 3}.
[0130] After generating anchors, the classification and regression sub-network connects a layer of the feature layer to the full convolutional network. After each convolution process, Relu is used as the excitation function, and finally the Sigmoid function is used to predict the classification or positioning of the target. The classification sub-network predicts the probability of target appearance and the probability that the target belongs to one of the ten types of targets for each anchor. The regression sub-network is used for positioning and generates the offset of the candidate region to the coordinates of the adjacent real region for each anchor.
[0131] The classification subnetwork predicts the probability of the target appearing at each position for A anchors and K categories. The classification branch includes four 3x3 convolutions (ReLU activation function, channel is 256), and finally a 3x3 convolution with output channel KA. Finally, sigmoid activation can get the probability of each anchor predicting each category. Each position is equivalent to KA binary classification problems.
[0132] The regression sub-network is parallel to the classification sub-network. Unlike the classification sub-network, the final output is the linear output of 4A. Figure 6 For each unknown space in A anchors, these 4 outputs predict the relative offset between the anchor and the ground truth.
[0133] At this time, the Ground Truth is involved, that is, the original marked real target position, hereinafter referred to as GT. Assume that the high threshold of the intersection-over-Union (IoU) is thigh and the low threshold is tlow. A dual threshold strategy based on IOU is adopted: calculate the IOU of the Anchor and all GTs, take the maximum IOU value, if it is greater than thigh, then this Anchor is considered to be a positive sample and is responsible for predicting the GT with the largest IOU; if it is lower than tlow, then this Anchor is considered to be a negative sample; if the IOU is between [tlow, thigh], it is ignored and does not participate in the training. In this way, each GT may match multiple Anchors, but the maximum IOU value of a GT and all Anchors may be less than thigh. Although the threshold condition is not met, it should be ensured that this GT is matched by the Anchor with the largest IOU value.
[0134] In RetinaNet50, non-maximum suppression (NMS) filters the initial prediction box by classification confidence to obtain credible candidate target locations. RetinaNet50's NMS eliminates boxes with a score lower than 0.5, a minimum intersection-over-union ratio lower than 0.5, and a score ranking higher than 1000.
[0135] For the application scenario of this embodiment, due to severe target occlusion, the minimum elimination score is reduced to 9.99999993922529e-09; due to the high similarity between the parts of the targets, the minimum intersection-over-union ratio is set to 0.6; due to the limited number of targets, the score ranking is reduced to 100.
[0136] In summary, if Fig.10 As shown, the neural network model of this embodiment includes a deep residual network module (ResNet), a feature pyramid network module (FPN), a region candidate network module, and a classification and regression sub-network module;
[0137] Among them, the deep residual network module is used to extract features of the input image, obtain C1 feature map, C2 feature map, C3 feature map, C4 feature map and C5 feature map from bottom to top, and output the C2 feature map, C3 feature map, C4 feature map and C5 feature map to the feature pyramid network.
[0138] The feature pyramid network module is used to: perform convolution processing with a convolution kernel of 1×1 on the C5 feature map to obtain an M5 feature map; perform feature fusion on the feature map obtained by performing convolution processing with a convolution kernel of 1×1 on the C4 feature map and the feature map obtained by performing 2-fold upsampling on the M5 feature map to obtain an M4 feature map; perform feature fusion on the feature map obtained by performing convolution processing with a convolution kernel of 1×1 on the C3 feature map and the feature map obtained by performing 2-fold upsampling on the M4 feature map to obtain an M4 feature map. to the M3 feature map; perform feature fusion on the feature map obtained by performing convolution processing with a convolution kernel of 1×1 on the C2 feature map and the feature map obtained by upsampling the M3 feature map by 2 times to obtain the M2 feature map; perform convolution processing with a convolution kernel of 3×3 on the M2 feature map, the M3 feature map, the M4 feature map and the M5 feature map respectively to obtain the P2 feature map, the P3 feature map, the P4 feature map and the P5 feature map; perform 3×3 convolution processing with a step size of 2 on the C5 feature map to obtain the P6 feature map.
[0139] The region candidate network module is used to map the anchor frame in the input image according to the P2 feature map, the P3 feature map, the P4 feature map, the P5 feature map and the P6 feature map, and generate the candidate frame corresponding to each anchor frame according to the preset n scales and m aspect ratios, where m is greater than 3; in this embodiment, n=3, and the 3 scales are {2 0 , 2 1 / 3 , 2 2 / 3}, m = 5, the 5 aspect ratios are {2 0 , 2 1 , 2 1 / 2 , 2 1 / 3 , 2 2 / 3 That is to say, one anchor box can generate 15 candidate boxes.
[0140] The classification and regression subnetwork module is used to filter and merge the candidate boxes according to a preset elimination score, intersection-over-union ratio threshold and score ranking to obtain a predicted box and its category; in this embodiment, the elimination score is 9.99999993922529e-09, the intersection-over-union ratio threshold is 0.6, and the score ranking is 100.
[0141] Furthermore, the above five target detection and recognition models (RetinaNet152, EfficientNet-d1, RetinaNet50, faster_rcnn_resnet50, faster_rcnn_inception) and the neural network model of this embodiment are trained and verified according to the same training set and verification set. The COCOs evaluation indicators of the six models are as follows: Fig.11As shown, it can be seen that the various COCOs evaluation indicators of the neural network model of this embodiment are better than those of the other five models.
[0142] This embodiment modifies the number of FPN layers, uses the features of the 2nd to 6th layers of FPN, increases the use of the P2 feature map, increases the receptive field of the network, and reduces the use of the P7 feature map to save computing costs, thereby enhancing the detection capability of smaller targets, and the detection capability of larger targets does not decrease significantly. By setting more aspect ratios to generate denser prediction boxes, it is possible to avoid missing small targets and strongly occluded targets. By setting appropriate non-maximum suppression parameters, it can be better applied to small and highly occluded target detection and classification scenarios to improve the accuracy of detection and classification.
[0143] Embodiment 2
[0144] The present embodiment is a computer-readable storage medium corresponding to the above-mentioned embodiment, on which a computer program is stored. When the computer program is executed by the processor, the various processes in the target detection and classification method embodiment described above are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0145] In summary, the object detection and classification method and computer-readable storage medium provided by the present invention improve the GPU utilization rate by adjusting the size of the sample image; increase the versatility of the data set and prevent overfitting by performing data enhancement processing; in the feature pyramid network module, by increasing the use of the P2 feature map, the detection of small-sized targets and strongly occluded targets can be achieved, and by reducing the use of the P7 feature map, the computing cost can be saved, thereby enhancing the detection capability of smaller targets, and the detection capability of larger targets is not significantly reduced; in the region candidate network module, by setting more aspect ratios to generate denser prediction boxes, the missed detection of small-sized targets and strongly occluded targets is avoided; in the classification and regression subnetwork module, by setting appropriate non-maximum suppression parameters, it can be better applied to small-sized and highly occluded target detection and classification scenarios, thereby improving the accuracy of detection and classification.
[0146] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A target detection and classification method, characterized in that: include: Acquire a sample set, wherein the sample set includes a plurality of sample images containing the target to be measured; Annotate the target to be measured in each sample image respectively to obtain an annotation file corresponding to each sample image, wherein the annotation file includes the bounding box coordinates of the target to be measured and the category of the target to be measured; Constructing a neural network model, and training the neural network model according to the sample images and their corresponding annotation files; Obtaining a picture to be tested, and inputting the picture to be tested into a trained neural network model to obtain a detection result; The neural network model includes a deep residual network module, a feature pyramid network module, a region candidate network module, and a classification and regression sub-network module; The deep residual network module is used to extract features from the input image, obtain C1 feature map, C2 feature map, C3 feature map, C4 feature map and C5 feature map from bottom to top, and output the C2 feature map, C3 feature map, C4 feature map and C5 feature map to the feature pyramid network; The feature pyramid network module is used to perform feature extraction, feature fusion and convolution processing on the C2 feature map, the C3 feature map, the C4 feature map and the C5 feature map to obtain the P2 feature map, the P3 feature map, the P4 feature map and the P5 feature map, and perform a 3×3 convolution processing with a step size of 2 on the C5 feature map to obtain the P6 feature map; The region candidate network module is used to map the anchor frame in the input image according to the P2 feature map, the P3 feature map, the P4 feature map, the P5 feature map and the P6 feature map, and generate a candidate frame corresponding to each anchor frame according to the preset n scales and m aspect ratios, where m is greater than 3; The classification and regression subnetwork module is used to filter and merge the candidate boxes according to the preset elimination score, intersection-over-union ratio threshold and score ranking to obtain the predicted box and its category; The feature pyramid network is specifically used for: Performing convolution processing with a convolution kernel of 1×1 on the C5 feature map to obtain an M5 feature map; A feature map obtained by performing convolution processing with a convolution kernel of 1×1 on the C4 feature map is subjected to feature fusion with a feature map obtained by upsampling the M5 feature map by a factor of 2, to obtain an M4 feature map; A feature map obtained by performing convolution processing with a convolution kernel of 1×1 on the C3 feature map is fused with a feature map obtained by upsampling the M4 feature map by a factor of 2 to obtain an M3 feature map; A feature map obtained by performing convolution processing with a convolution kernel of 1×1 on the C2 feature map is subjected to feature fusion with a feature map obtained by performing 2-fold upsampling on the M3 feature map to obtain an M2 feature map; Perform convolution processing with a convolution kernel of 3×3 on the M2 feature map, the M3 feature map, the M4 feature map, and the M5 feature map, respectively, to obtain a P2 feature map, a P3 feature map, a P4 feature map, and a P5 feature map; The C5 feature map is subjected to a 3×3 convolution process with a step size of 2 to obtain a P6 feature map.
2. The target detection and classification method according to claim 1, characterized in that: After labeling the target to be measured in each sample image respectively to obtain a labeling file corresponding to each sample image, the method further includes: Each sample image is scaled to a preset size, and each annotation file is updated accordingly according to the scaled sample image.
3. The target detection and classification method according to claim 1, characterized in that: After labeling the target to be measured in each sample image respectively to obtain a labeling file corresponding to each sample image, the method further includes: Perform data enhancement processing on the sample set.
4. The target detection and classification method according to claim 1, characterized in that: The training of the neural network model according to the sample image and its corresponding annotation file is specifically as follows: Traverse the sample set and obtain a sample image in sequence; Inputting the sample image into the neural network model to obtain a prediction result corresponding to the sample image, wherein the prediction result includes a prediction box and its category; Calculate the loss value corresponding to the sample image according to the prediction result and the annotation file corresponding to the sample image; According to the loss value, updating the neural network model through a back propagation algorithm; After traversing the sample set, the average loss value corresponding to each sample image is calculated to obtain the average loss value; Determine whether the average loss value is less than a preset threshold; If not, then according to the loss value, the neural network model is updated by a back propagation algorithm, and the step of traversing the sample set and obtaining one sample image in turn is continued; If yes, the training is ended and the trained neural network model is obtained.
5. The target detection and classification method according to claim 1, characterized in that: The training of the neural network model according to the sample image and its corresponding annotation file is specifically as follows: Traversing the sample set, and obtaining a sample image from the sample set in sequence; Inputting the sample image into the neural network model to obtain a prediction result corresponding to the sample image, wherein the prediction result includes a prediction box and its category; Calculate the loss value corresponding to the sample image according to the prediction result and the annotation file corresponding to the sample image; According to the loss value, updating the neural network model through a back propagation algorithm; After traversing the sample set, the number of training steps is increased by one, and the initial value of the number of training steps is 0; Determining whether the number of training steps reaches the predicted number; If not, continue to execute the step of traversing the sample set and sequentially acquiring a sample image from the sample set; If yes, the training is ended and the trained neural network model is obtained.
6. The target detection and classification method according to claim 4 or 5, characterized in that: The loss value corresponding to the sample image is calculated according to the prediction result and the annotation file corresponding to the sample image as follows: According to the prediction result and annotation file corresponding to the sample image, the classification loss value and the regression loss value are calculated by the Focal Loss function and the SmoothL1 Loss function respectively, and the classification loss value and the regression loss value are added to obtain the loss value corresponding to the sample image.
7. The target detection and classification method according to claim 1, characterized in that: n=3, the 3 scales are {20, 21 / 3, 22 / 3}, m=5, the 5 aspect ratios are {20, 21, 21 / 2, 21 / 3, 22 / 3}.
8. The target detection and classification method according to claim 1, characterized in that: The elimination score is 9.99999993922529e-09, the intersection-over-union ratio threshold is 0.6, and the score ranking is 100.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.