A Separate Traffic Sign Detection and Recognition Method

Abstract

The present invention discloses a separated traffic detection method, which completes the detection and recognition of traffic signs through a YOLOv3‑SPP‑SE network for traffic sign detection and a MobileNetv3‑small network for traffic sign classification; first, a video stream is decoded into a picture for detection and scaled to a corresponding size; then the picture is input into a pruned YOLOv3‑SPP‑SE to detect traffic signs; then the coordinates of the traffic sign in the picture are obtained according to the traffic sign detection result, and the traffic sign image is sent to MobileNetv3‑small for classification and recognition; finally, the detection and recognition results are output. The present invention can accurately detect and recognize traffic signs in natural scenes, and can maintain real-time detection and recognition in mobile scenes with limited calculations; separated detection and recognition can compress the detection network to a greater extent, reduce the calculation cost, and has good application prospects.

Description

Technical Field

[0001] The present invention relates to an image processing method for traffic sign detection and recognition, and specifically, to a traffic sign detection and recognition method based on deep learning. Background Art

[0002] Traffic sign detection and recognition system is a key subsystem for realizing intelligent transportation system, and is widely used in assisted driving, intelligent navigation, autonomous driving, smart transportation, traffic sign maintenance and other fields. At the same time, for safety reasons, the traffic sign detection and recognition system needs to be both accurate and fast.

[0003] Traditional algorithms use color-based image segmentation, shape-based image segmentation, maximum stable extremum region and other methods to locate traffic signs in images; use Haar, Hog, SIFT, LBP and other hand-designed operators to extract features of traffic signs; use SVM, Adaboost and other classifiers to classify the extracted features. Traditional traffic sign detection and recognition algorithms have poor robustness and are difficult to solve problems such as various lighting conditions, complex backgrounds, and small targets in natural scenes.

[0004] Existing deep learning-based methods can detect and recognize traffic signs at the same time. When faced with a long-tail dataset such as traffic signs, it is difficult to guarantee the recall rate and accuracy of the tail data. At the same time, due to the large number of categories, the model is difficult to compress, and a large amount of continuous computing resources are required when the model is running, making it difficult to have both algorithm performance and speed in mobile scenarios.

[0005] Compared with traditional algorithms, separated traffic sign detection and recognition is more robust. Compared with existing deep learning-based methods, it has a higher recall rate for traffic signs with lower frequency of occurrence and more flexible computing requirements. It also has higher accuracy and real-time performance. Summary of the invention

[0006] In view of the shortcomings of the existing methods, the purpose of the present invention is to provide a separated traffic sign detection and recognition method, embed the underlying YOLOv3-SPP model into the SE structure, improve the loss function of the original model; prune and compress the improved detection model; train a lightweight traffic sign recognition model; and cascade the detection and recognition models to obtain the final traffic sign detection and recognition system.

[0007] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] Step 1: Perform cropping preprocessing on the Tsinghua-Tencent100K dataset to reduce the image size for training the detection network;

[0009] Step 2: Use the processed data set to train the improved YOLOv3-SPP model, and select the model with the best performance during the training process as the base model;

[0010] Step 3: Perform sparse training on the base model to induce the γ scale factor of the Batch Normalization layer to become sparse, thus obtaining a sparse model.

[0011] Step 4: Use the channel pruning method of fused pruning mask to prune the sparse model to obtain the pruned YOLOv3-SPP network structure and corresponding weights;

[0012] Step 5: For the embedded Squeeze-and-Excitation structure of the pruned model, the SE structure is initialized randomly, and the rest is initialized with the pruned weights;

[0013] Step 6: Fine-tune YOLOv3-SPP-SE on the processed dataset to obtain the final model for detection;

[0014] Step 7: Sort the traffic signs and traffic sign recognition datasets in the Tsinghua-Tencent100K dataset to obtain the traffic sign recognition dataset;

[0015] Step 8: Use the sorted traffic sign recognition dataset to train the MobileNetv3-small model to obtain the traffic sign recognition model;

[0016] Step 9: Cascade the pruned detection model and recognition model to obtain the final model.

[0017] A further technical solution is that the specific steps of preprocessing the Tsinghua-Tencent100K dataset in step 1 are as follows:

[0018] Step 1-1: For an image containing a target, take the top, left, bottom, and right values ​​of all the targets;

[0019] Step 1-2: Calculate the midpoint between the top and the top boundary, the midpoint between the leftmost end and the left boundary, the midpoint between the bottom and the bottom boundary, and the midpoint between the rightmost end and the right boundary respectively;

[0020] Step 1-3: Use the 4 points obtained in step 1-2 to crop the original image;

[0021] Step 1-4: Calculate the coordinates of the traffic sign based on the cropping coordinates and the size of the original image;

[0022] Step 1-5: Randomly divide the processed data set into training set and validation set in a ratio of 2:1.

[0023] A further technical solution is to improve the loss function of YOLOv3-SPP in step 2, which is described as follows:

[0024] Using the confidence loss and classification loss of the FocalLoss measurement model can balance the loss of positive and negative samples and difficult and easy samples. The formula is as follows:

[0025]

[0026]

[0027]

[0028] In the above formula Used to adjust the weights of positive and negative samples. It is used to adjust the weight of difficult and easy samples. y represents the true label, p represents the predicted label, α and y are hyperparameters with values ​​of 0.25 and 2 respectively.

[0029] Using GIoU Loss to measure the border loss of the model can solve the problem of inconsistent border evaluation during training and evaluation. The formula is as follows:

[0030]

[0031]

[0032] In the above formula, A and B represent two borders to be measured, and C represents the smallest bounding rectangle of the two borders A and B. A further technical solution is that the sparse training process in step 3 is described in detail as follows:

[0033] Add all Batch Normalization layers to the overall loss The penalty term of the scale factor. The penalty formula uses the L1 norm, inducing Sparse; the penalty coefficient λ is set to 0.001 to ensure that the gradient in the sparse training process is always dominated by the gradient of the detection network. The specific expression is as follows:

[0034]

[0035]

[0036] In the above formula, the first term represents the original loss of the detection network, and the second term represents The penalty term of the scale factor, λ, is set to 0.001. A further technical solution is that the specific steps of pruning in step 4 are as follows:

[0037] Step 4-1: Take all the γ values ​​and sort them from small to large;

[0038] Step 4-2: According to the channel pruning ratio, select the γ value under the ratio from the selected and sorted γ as the global threshold; Step 4-3: According to the global threshold, calculate the pruning mask of each convolutional layer while retaining the minimum number of channels;

[0039] Step 4-4: Group all convolutional layers into 5 groups based on each downsampling.

[0040] Step 4-5: Take the union of the pruning masks of the convolutional layers in each group to obtain 5 groups of pruning masks;

[0041] Step 4-6: Use different pruning masks to prune the convolutional layers in the group.

[0042] A further technical solution is to embed the SE structure described in step 5, which is specifically described as follows:

[0043] In the first residual block after each downsampling, the residual branch in the residual block is embedded in the SE structure. First, the number of channels of the branch output is obtained; then the feature map of the branch output is globally averaged pooled; then it passes through a two-layer fully connected network, the network input dimension is consistent with the number of channels, the first layer output dimension is one-fourth of the number of channels, ReLU activation is used, and the second layer output dimension is consistent with the number of channels, Sigmoid activation is used; finally, the feature map of the residual branch is weighted using the fully connected output.

[0044] A further technical solution is the fine-tuning training described in step 6, which is specifically described as follows:

[0045] Use the weights retained after pruning to restore the structure corresponding to the pruned model, and use the parameters of the SE structure to randomly initialize; fine-tune the recovery accuracy on the sorted traffic sign recognition dataset.

[0046] A further technical solution is to organize the traffic sign classification dataset described in step 7, specifically to crop the traffic sign targets in the Tsinghua-Tencent100K dataset and merge them with the traffic sign recognition dataset by category; select traffic sign categories with more than 200 occurrences; and upsample the selected categories with less than 500 categories to 500 by random cropping, random blurring, random Gaussian noise, etc.

[0047] A further technical solution is that the classification model training described in step 8 is specifically described as scaling the sorted recognition data set images to 256*256 and normalizing them, and then sending them to the MobileNetv3-small network for training.

[0048] A further technical solution is that the cascade operation described in step 9 is specifically described as packaging the detection results output by the detection model into a Batch, and sending the Batch to the classification model for parallel processing.

[0049] Compared with the prior art, the present invention has the following significant advantages: (1) Single-class detection is easier to learn. Compared with multi-class detection, single-class detection only learns the common features of traffic signs, which can reduce the learning difficulty of the detection network; (2) Single-class detection reduces the requirements for the data set. It avoids the processing of long-tail data sets; (3) All traffic signs can be detected. If new traffic sign detection and recognition categories are added, only a new classification model needs to be retrained; (4) The calculation requirements are flexible and do not require continuous large-scale calculations; (5) The algorithm is highly robust and can detect and recognize in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is the overall flow chart of the algorithm of the present invention;

[0051] Figure 2 It is a flow chart of the present invention for pruning YOLOv3-SPP; Figure 3 The present invention is used to embed the SE structure diagram for the pruning model; Figure 4 It is a result diagram of traffic sign detection and recognition of the present invention. DETAILED DESCRIPTION

[0052] like Figure 1 As shown, the steps of traffic sign detection and recognition of the present invention are as follows: Step 1: Decode an image from an input video stream;

[0053] Step 2: Send the image to the pruned YOLOv3-SPP-SE model to detect whether there is a traffic sign;

[0054] Step 3: If the traffic sign is included, the traffic sign is sent to the classification model in parallel for segmentation and recognition; if the traffic sign is not included, the original image is returned;

[0055] Step 4: The classification model performs segmentation and recognition on the output of the detection model in parallel. The specific implementation is as follows.

[0056] Step 1: Decode the image.

[0057] Decode the image from the video stream frame by frame, scale the long side of the image to 608, and scale the short side to an integer multiple of 32, using gray padding in between to maintain the aspect ratio of the image content.

[0058] Step 2: Prune the YOLOv3-SPP-SE model to detect traffic signs in the image. The YOLOv3-SPP-SE model used for detection in this process is obtained as follows: Figure 2 .

[0059] Step 2-1: Perform basic training on the detection network to obtain a basic model. Train the YOLOv3-SPP network on the processed Tsinghua-Tencent100K dataset, and select the optimal model in the training process as the basic model. The FocalLoss measurement model's confidence loss and classification loss are used in the training process to balance the loss of positive and negative samples and difficult and easy samples. The formula is as follows:

[0060]

[0061]

[0062]

[0063] In the above formula, αt is used to adjust the weight of positive and negative samples, (1− pt)y is used to adjust the weight of difficult and easy samples, y represents the true label, p represents the predicted label, α and y are hyperparameters with values ​​of 0.25 and 2 respectively.

[0064] Using the border loss of the GIoULoss measurement model can solve the problem of inconsistent border evaluation during training and evaluation. The formula is as follows:

[0065]

[0066]

[0067] In the above formula, A and B represent two borders that need to be measured, and C represents the smallest circumscribed rectangle of the two borders A and B.

[0068] Step 2-2: Perform sparse training on the basic model to obtain a sparse model. Sparse training is to add an additional penalty term for the γ scale factor of the BatchNormalization layer during the training process to induce γ to become sparse, that is, to add a penalty term for the γ scale factor of all Batch Normalization layers to the overall loss. The penalty formula uses the L1 norm to induce γ to be sparse; the penalty coefficient λ is set to 0.001 to ensure that the gradient in the sparse training process is always dominated by the gradient of the detection network. The specific expression is as follows:

[0069]

[0070] g(γ) = llγl1

[0071] In the above formula, the first term represents the original loss of the detection network, and the second term represents the penalty term of the γ scale factor, and the value of λ is 0.001.

[0072] Step 2-3: Prune the sparse model to obtain the pruned model. First, sort all γ from small to large, and calculate the pruning threshold according to the required channel pruning ratio; then calculate the pruning mask of each convolutional layer before the BatchNormalization layer according to the pruning threshold; then, according to the 5 downsamplings of the feature extraction network Darknet-53, take each downsampling as a group, and take the union of the pruning masks of all convolutional layers in the group according to the downsampling group label; finally, use the union pruning mask to prune the convolutional layers in the group. In order to ensure the continuity of the entire network, keep 0.01 of all channels in a certain layer.

[0073] Step 2-4: Embed the residual block of the pruned model into the SE structure to obtain the final detection model. The SE structure is embedded in the residual block after each downsampling in the feature extraction network, such as Figure 3 In the first residual block after each downsampling, the residual branch in the residual block is embedded in the SE structure. First, the number of channels of the branch output is obtained; then the feature map of the branch output is globally averaged pooled; then it passes through a two-layer fully connected network, the network input dimension is consistent with the number of channels, the first layer output dimension is one-fourth of the number of channels, ReLU activation is used, and the second layer output dimension is consistent with the number of channels, Sigmoid activation is used; finally, the feature map of the residual branch is weighted using the fully connected output.

[0074] Step 2-5: Fine-tune the final detection model to restore accuracy. Use the weights retained after pruning to restore the corresponding structure, and use the randomly initialized SE structure to fine-tune and restore accuracy.

[0075] Step 3: Determine the traffic sign area.

[0076] If the detection result contains traffic signs, each detection result of the detection network is scaled to 256*256, and all detection results are packaged into a Batch; if not, the original image is returned.

[0077] Step 4: Traffic sign recognition.

[0078] Use the trained MobileNetv3-small to classify and identify the detection results in parallel.

[0079] The specific details of training MobileNetv3-small are as follows: the images in the sorted recognition dataset are scaled to 256*256 and normalized; then data enhancement methods such as random angle rotation within 10 degrees, color jitter, brightness jitter, random blur, and random center cropping are used to simulate the various forms of traffic signs in natural scenes. The loss function used in the training phase is the cross entropy function with softmax; the optimizer uses SGD with a momentum of 0.9 and an initial learning rate of 0.001; the 90% decay in the entire epoch is reduced to one-tenth of the original; the batch size is 24.

[0080] Figure 4 The results of the traffic sign detection and recognition of the present invention are given. Among them, the ts label is the sign detected by the detector but not recognized by the classification model. From the figure, it can be seen that the traffic sign detection and recognition system proposed in this paper has good detection performance when facing small target traffic signs, distorted targets, occluded targets, and blurred targets in natural scenes.

Claims

1. A separate traffic sign detection and recognition method, characterized in that Follow these steps: Step 1: Crop and preprocess the Tsinghua-Tencent 100K image dataset; Step 2: Use the processed data set to train the YOLOv3-SPP model and select the model with the best performance during the training process as the base model; Step 3: Perform sparse training on the base model to induce the γ scale factor of the Batch Normalization layer to become sparse, thus obtaining a sparse model. Step 4: Use the channel pruning method with fused pruning mask to prune the sparse model to obtain the pruned YOLOv3-SPP network structure and corresponding weights; The specific steps of pruning in step 4 are as follows: Step 4-1: Take all γ values ​​and sort them from small to large; Step 4-2: According to the channel pruning ratio, select the γ value under the ratio from the selected and sorted γ as the global threshold; Step 4-3: According to the global threshold, calculate the pruning mask of each convolutional layer while retaining the minimum number of channels; Step 4-4: Group all convolutional layers into 5 groups with each downsampling; Step 4-5: Take the union of the pruning masks of the convolutional layers in each group to obtain 5 groups of pruning masks; Step 4-6: Use different pruning masks to prune the convolutional layers in the group; Step 5: Embed the pruned model into the SE structure to obtain YOLOv3-SPP-SE. The SE structure is a Squeeze-and-Excitation structure. Specifically, in the first residual block after each downsampling in the feature extraction network, the residual branch in the residual block is embedded into the SE structure; first, the number of channels of the output of the branch is obtained; then, the feature map of the output of the branch is globally averaged pooled; then, it passes through a two-layer fully connected network, the input dimension of which is consistent with the number of channels, the output dimension of the first layer is one-fourth of the number of channels, and ReLU activation is used, and the output dimension of the second layer is consistent with the number of channels, and Sigmoid activation is used; finally, the feature map of the residual branch is weighted using the fully connected output; random initialization is used for initialization, and the rest is initialized with pruning weights; Step 6: Fine-tune YOLOv3-SPP-SE on the processed dataset to obtain the final model for detection; Step 7: Organize and obtain the traffic sign recognition dataset; Step 8: Use the sorted traffic sign recognition dataset to train the MobileNetv3-small model to obtain the traffic sign recognition model; Step 9: Cascade the pruned detection model and recognition model to obtain the final model; Step 10: Use the final model for traffic sign detection and recognition.

2. A separate traffic sign detection and recognition method according to claim 1, characterized in that: In step 2, the YOLOv3-SPP model is improved, and Focal Loss is used to measure the confidence loss and classification loss of the detection network; GIoU Loss is used to measure the border loss of the detection network.

3. A separate traffic sign detection and recognition method according to claim 1, characterized in that: The sparse training process in step 3 specifically includes adding a penalty term for the γ scale factor of all Batch Normalization layers to the overall loss; the penalty formula uses the L1 norm to induce γ sparsity; the penalty coefficient λ is set to 0.0001.

4. A separate traffic sign detection and recognition method according to claim 1, characterized in that: The fine-tuning process includes: using the weights retained after pruning to restore the structure corresponding to the pruned model, using the parameters of the SE structure to be randomly initialized; and fine-tuning the recovery accuracy on the sorted traffic sign recognition dataset.

Images