A traffic sign detection method and device, system, storage medium

Abstract

The application discloses a traffic sign detection method and device, system and storage medium, and comprises the following steps: S1, acquiring a traffic sign dataset; S2, training an improved YOLOv5s model according to the traffic sign dataset; and S3, inputting a real-time video stream monitored by a vehicle into the trained improved YOLOv5s model for traffic sign detection. The technical scheme of the application solves the problems of low detection accuracy and poor real-time performance of small target traffic signs, and effectively reduces the missing detection rate of small target traffic signs without introducing new parameters.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of deep learning, and particularly relates to a traffic sign detection method and device, system and storage medium. BACKGROUND

[0002] Traffic sign detection is one of the key technologies of intelligent transportation systems. Through accurate identification of traffic signs, self-driving cars can better understand the road environment and make corresponding decisions and driving plans, thereby improving driving safety and efficiency. Traffic sign detection is of great significance to the development of self-driving car technology, and has important theoretical significance and broad application prospects. Traffic signs guide vehicles to travel in a standard manner, improve driving efficiency, and promote road safety. For example, dangerous prediction can be performed through prohibition signs, obstacle avoidance processing can be performed through warning signs, and control preprocessing can be performed through indication signs. However, the environment in which traffic signs are located is very complex, and the detection process is easily affected by weather conditions, environmental lighting, and angle changes.

[0003] Since traffic signs usually have distinctive color features (red, yellow, and blue) and regular shape structures (triangular, circular, and square), traffic sign detection methods based on traditional handcrafted features usually detect different traffic signs according to the color or shape attributes of the features. Traditional detection methods are easily affected by changes in lighting and detection backgrounds, have low detection accuracy, poor real-time performance, and weak generalization ability.

[0004] With the advent of Fast R-CNN, Faster R-CNN, and YOLO series algorithms, general object detection methods based on deep learning have rapidly developed. Many traffic sign detection algorithms have emerged on the basis of these general object detection algorithms. Although traffic sign detection technology based on deep learning is constantly developing, in actual vehicle-mounted camera images, small target traffic signs often account for less than 0.1% of the total image area, and similar objects to the detection target often appear in the detection background, which can interfere with detection, cause false detection, and current traffic sign detection algorithms usually require a large amount of computing resources, and detection real-time performance needs to be improved.

[0005] Currently, the main problem of small target traffic sign detection is that there is a lack of a perfect traffic sign dataset, the small target area is too small to be detected, there are many types of traffic signs, light changes, and obstacle obstructions. The current detection algorithm has the problems of being difficult to simultaneously ensure detection speed and detection accuracy, and weak generalization ability of the detection model. SUMMARY

[0006] The technical problem to be solved by the present application is to provide a traffic sign detection method and device, system and storage medium, which effectively reduce the small target traffic sign missing detection rate without introducing new parameters.

[0007] To achieve the above object, the present application adopts the following technical solution:

[0008] A traffic sign detection method comprises the following steps:

[0009] Step S1, acquiring a traffic sign dataset;

[0010] Step S2, training an improved YOLOv5s model according to the traffic sign dataset;

[0011] Step S3, inputting a real-time video stream of vehicle monitoring into the trained improved YOLOv5s model for traffic sign detection.

[0012] As a preferred, in step S2, the YOLOv5s model is subjected to lightweight design, feature prediction scale optimization and feature fusion enhancement to obtain the improved YOLOv5s model; wherein, the YOLOv5s model is subjected to channel pruning, and the 3x3 convolution is replaced by depth separable convolution to realize lightweight design; the 160x160 shallow layer feature is introduced in multi-scale fusion to optimize the feature prediction scale; the 5C module is used to fuse the shallow and deep layer feature maps to enhance the features.

[0013] As a preferred, the improved YOLOv5s model uses a loss function EIoU Loss instead of the loss function GIoU Loss of the YOLOv5s model.

[0014] The present application also provides a traffic sign detection device, comprising:

[0015] An acquisition module is configured to acquire a traffic sign dataset;

[0016] A training module is configured to train an improved YOLOv5s model according to the traffic sign dataset;

[0017] A detection module is configured to input a real-time video stream of vehicle monitoring into the trained improved YOLOv5s model for traffic sign detection.

[0018] As preferred, the improved YOLOv5s model is obtained by lightening design, feature prediction scale optimization and feature fusion enhancement of the YOLOv5s model; wherein, the YOLOv5s model is pruned in channels, and the 3x3 convolution is replaced by the depth separable convolution to realize the lightening design; the 160x160 shallow layer feature is introduced in the multi-scale fusion to optimize the feature prediction scale; the 5C module is used to fuse the shallow layer and deep layer features to strengthen the features.

[0019] As preferred, the improved YOLOv5s model uses the loss function EIoU Loss instead of the loss function GIoU Loss of the YOLOv5s model.

[0020] The application also provides a traffic sign detection system, comprising a memory and a processor, wherein the memory stores a computer program which is run by the processor, and the computer program performs the traffic sign detection method when being run by the processor.

[0021] The application also provides a storage medium, wherein the storage medium stores a computer program which performs the traffic sign detection method when being run.

[0022] The application prunes the unimportant and redundant modules in the YOLOv5s, introduces the depth separable convolution, reduces the model size, reduces the required computing resources of the model, and improves the real-time performance of the traffic sign detection. The feature prediction scale optimization fully utilizes the shallow layer features, improves the perception ability to small targets, and reduces unnecessary computing overhead. The 5C module is used in the feature fusion part to weaken the mutual interference between the shallow layer features and the deep layer features, and avoid inaccurate target positioning. The EIoU loss function is introduced to effectively solve the problem that the penalty term of the ratio change of the long and short sides is invalid, and effectively improve the small target detection accuracy. The application has better traffic sign detection performance, and through the accurate identification of the traffic sign, the unmanned vehicle can better understand the road environment, make corresponding decisions and driving planning, and thus improve the driving safety and efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only belong to the embodiments of the application, and those skilled in the art can obtain other drawings according to the provided drawings without any creative labor.

[0024] Figure 1 The traffic sign detection method flowchart of the embodiments of the application;

[0025] Figure 2A schematic diagram of the improved YOLOv5s model;

[0026] Figure 3 A schematic diagram of the structure for optimizing feature prediction scale. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] Example 1:

[0030] like Figure 1 As shown, an embodiment of the present invention provides a traffic sign detection method including:

[0031] Step S1: Obtain the traffic sign dataset;

[0032] Step S2: Train an improved YOLOv5s model based on the traffic sign dataset;

[0033] Step S3: Input the real-time video stream from the vehicle monitoring system into the trained improved YOLOv5s model for traffic sign detection.

[0034] As an embodiment of the present application, Chinese Traffic Sign Detection Benchmark 2021 (CCTSDB 2021) is selected, which is an improved version of Chinese Traffic Sign Detection Benchmark 2017 (CCTSDB 2017), adding more than 4000 images, removing simple samples, increasing complex samples, and making a complete test data set. CCTSDB 2021 has a total of 16356 training images and 2000 test images, and a total of three types of traffic signs. The training set contains 13876 prohibition signs, 4598 warning signs, and 8363 mandatory signs, including road traffic scenes under different weather conditions, such as foggy, snowy, rainy, night, overcast and sunny. Among the 2000 test images, there are 1500 positive sample images and 500 negative sample images. The test set is divided into five categories according to the size of the traffic signs. CCTSDB 2021 better restores the traffic sign detection scene in the real road environment, which can effectively improve the robustness of the model.

[0035] As an embodiment of the present application, in step S2, the YOLOv5s model is designed to be lightweight, the feature prediction scale is optimized, the feature fusion is enhanced, and the loss function is optimized to obtain an improved YOLOv5s model, as shown in Figure 2 , and the specific steps are as follows:

[0036] 1. Lightweight design

[0037] Traffic sign detection systems are usually deployed on unmanned vehicles or other transportation tools with limited computing resources. Therefore, when pursuing high detection accuracy, the computational load of the model should be minimized to make the model easy to deploy on devices with limited computing resources. YOLOv5 includes four models of different sizes: YOLOv5s, YOLOv5x, YOLOv5m, and YOLOv5l. The present application selects the smallest YOLOv5s as the baseline model to meet the high real-time demand of traffic sign detection. The YOLOv5s model is pruned, and depth separable convolution is applied to reduce the model size and computational load. When pruning the model, the network model is first loaded, and the neurons in the network are evaluated, and then the unimportant and large network layers are deleted. Pruning can improve the detection speed of the model, but in order to obtain better detection real-time, depth separable convolution is introduced to replace part of the 3x3 convolution with depth separable convolution. Depth separable convolution is mainly used to reduce network parameters and improve the computational efficiency of the model.

[0038] The depth separable convolution includes two processes of depthwise convolution and pointwise convolution. The depthwise convolution allocates a convolution kernel to each channel of the feature map for convolution. If a KxKxM convolution is performed on an image with a size of HxWxN, the computational power consumed is HxWxNxKxKxM. The depthwise convolution first performs a KxKxN depthwise convolution on the feature map, and then performs a 1x1xM pointwise convolution, and the total computational power consumed is HxWxNxKxK+NxMxHxW. Given the same input features, the depth separable convolution consumes 1 / M+1 / K of the calculation amount of the conventional convolution 2 After lightening the model, the model becomes smaller and the calculation amount is reduced, and the detection speed is greatly improved.

[0039] 2. Feature prediction scale optimization

[0040] In the CCTSDB 2021 dataset, small target traffic signs account for the highest proportion. In actual traffic sign detection scenarios, most traffic signs are small targets. However, YOLOv5s only uses three scales of features, i.e., 20x20, 40x40 and 80x80, for detection, and the use of shallow layer information is less, which is easy to cause the loss of position information of small target traffic signs. Small target detection relies on the detection of larger bottom layer feature maps, and rich spatial position information can be captured from shallow layer features to fully extract the features of small target traffic signs. In order to fully utilize the shallow layer features to detect small target traffic signs, two strategies are adopted to improve the network prediction scale. The first strategy is to introduce 160x160 shallow layer features in the multi-scale fusion part to increase the number of detection feature scales of YOLOv5s. The prediction scale of YOLOv5 becomes 20x20, 40x40, 80x80 and 160x160, which are respectively responsible for the detection of traffic targets of four different size levels, i.e., large, medium, small and extremely small. The newly added detection branch is composed of shallow layer features (features before the P3 layer) and deep layer features after a large number of convolutions, which fully utilizes the rich fine-grained information in the shallow layer features and the rich semantic information in the deep layer features, and theoretically can effectively improve the detection effect of small target traffic signs. The second improvement strategy is to introduce shallow layer features in the multi-scale fusion part, but not to increase the number of detection feature scales of YOLOv5s. Considering that the number of large targets in general traffic sign datasets is small, the feature scale responsible for the detection of large target traffic signs is deleted when the feature scale responsible for the detection of small and extremely small targets is introduced. This theoretically will not affect the detection effect of the network, and finally reduces unnecessary calculation costs on the premise of not affecting the detection effect of the network. This strategy aims to improve the ability of the network to detect small targets. The network frameworks of the two prediction scale improvements are as follows Figure 3The light YOLOv5s simple structure is shown, F1-F4 represents the fusion of each layer backbone feature and deep feature, P1-P4 represents the further processing of the fused features, F1 is the newly added feature layer, which fuses the information of shallow and deep layers, P1 further extracts the information of F1 feature layer, and the features of P1 are processed to obtain 160x160 detection head. By introducing 160x160 features and integrating them with deeper features, the network can better capture spatial position information and extract necessary features to detect small target traffic signs.

[0041] 3. Feature fusion enhancement

[0042] YOLOv5s is composed of four parts: input, Backbone, Neck, and Prediction. The main function of Backbone is feature extraction, and Neck is responsible for fusing features from Backbone. Backbone and neck part use a large number of C3 structures and convolution layers. C3 is a convolution module that can effectively extract feature information by stacking convolution. However, using a large number of convolutions will cause the loss of key image information in deep feature maps. To solve this problem, a more efficient module is constructed to improve feature extraction quality, increase information flow, and enhance network expression ability. This module is composed of 5 convolutions, so it is named 5C. 5C can be divided into three branches. The first branch first passes through a 1x1 convolution, reducing the channel number by half, and then passes through a 3x3 convolution for further feature extraction. The second branch first passes through a 3x3 convolution, and then passes through a 1x1 convolution to reduce the channel number by half. The third branch inputs the feature map through a 1x1 convolution for dimension reduction, and the channel number becomes half of the original channel number, and the feature map size remains unchanged. Their expressions are as follows:

[0043] Y1 = f Conv1×1 (f Conv3×3 (I)) (1)

[0044] Y2 = f Conv3×3 (f Conv1×1 (I)) (2)

[0045] Y3 = f Conv1×1 (I) (3)

[0046] In the formula, I is the input feature map, Y1, Y2 and Y3 are the output feature maps of the three branches respectively, f Conv3×3 represents a 3x3 two-dimensional convolution, f Conv1×1 represents a 1x1 two-dimensional convolution.

[0047] The sizes of Y1, Y2 and Y3 are the same, Y4 is obtained by performing an add operation on Y1 and Y2, the channel number of the feature map of Y4 is unchanged, and the information amount under each dimension of the feature map is increased. Then, Y3 and Y4 are concatenated to fuse the features and increase the feature dimension, so as to ensure that the channel number of the output feature map is the same as that of the input feature map.

[0048]

[0049] Y = Y3 Y4 (5)

[0050] In the formula, represents the addition of the feature map, the channel number is unchanged, represents the addition of the channel number, the feature map is unchanged, Y4 represents the feature map output after Y1 and Y2 are added, Y is the feature map output by the 5C module. Y3 and Y4 have different receptive fields, so that the network can learn rich features and make up for the decrease in accuracy due to insufficient feature learning ability. This module further extracts the input features, while retaining the characteristics of the input features and enriching the information level of the features. Since a nonlinear activation function is added after the 1x1 convolution, the module increases the nonlinear characteristics of the entire model and improves the overall expression ability of the model. In order to explore the performance of the 5C module, two modules similar to 3C are constructed for comparison with 3C. The ablation experiment proves that using the 3C module in the feature fusion part of YOLOv5s can improve the detection performance of the model.

[0051] 4. Loss function optimization

[0052] In order to make full use of the target information in the data set, the model is further optimized from the loss function aspect, and EIoU Loss is used instead of the original loss function GIoU Loss in the model. By obtaining more feature information, the detection accuracy of the model is improved. When there are two prediction boxes with the same width and height and located on the same horizontal plane, GIoU Loss degenerates into IoU Loss, which has the problems of slow convergence and inaccurate regression. DIoU Loss further solves the problems existing in GIoU Loss. CIoU Loss increases the consideration of aspect ratio on the basis of DIoU Loss, but there are problems such as ambiguous description of aspect ratio and not considering the balance of difficult samples. EIoU Loss solves the problems existing in CIoU Loss by respectively calculating the difference value of width and height to replace the aspect ratio on the basis of CIoU Loss, and introduces Focal Loss to solve the problem of sample imbalance. EIoU Loss minimizes the width and height difference between the target box and the anchor box, has faster convergence speed and better positioning effect, and its formula is as follows:

[0053]

[0054] wherein, c w and c h denote the width and height of the minimum bounding box of the target frame and the prediction frame, respectively, p 2 denotes the Euclidean distance between two points, b and b gt represent the center coordinates of the target frame and the prediction frame, respectively, d is the distance between the center points of the target frame and the prediction frame, and c is the diagonal distance of the minimum bounding box. By introducing the EIoU Loss, the model's extraction of target features is more thorough, further improving the model's detection performance.

[0055] Further, in step S2, the traffic sign dataset is input into the improved YOLOv5s model for training; the experiment is performed on a platform of NVIDIA GeForce GTX 3080ti, 16GB RAM; the input image size is set to 640x640, the stochastic gradient descent (SGD) optimizer is used for model training, the batch size is 32, the weight decay is 0.0005, and the learning rate is 0.01.

[0056] Embodiment 2:

[0057] The embodiment of the present application also provides a traffic sign detection device, comprising:

[0058] an acquisition module configured to acquire a traffic sign dataset;

[0059] a training module configured to train an improved YOLOv5s model according to the traffic sign dataset;

[0060] a detection module configured to input a real-time video stream of vehicle-mounted monitoring into the trained improved YOLOv5s model for traffic sign detection.

[0061] As an embodiment of the present application, the YOLOv5s model is designed to be lightweight, the feature prediction scale is optimized, and the feature fusion is enhanced to obtain an improved YOLOv5s model; wherein, the YOLOv5s model is pruned in channels, and the 3x3 convolution is replaced by a depth separable convolution to achieve lightweight design; in multi-scale fusion, 160x160 shallow features are introduced for feature prediction scale optimization; a 5C module is used to fuse the shallow and deep feature maps for feature enhancement.

[0062] As an embodiment of the present application, the improved YOLOv5s model uses a loss function EIoU Loss instead of the loss function GIoU Loss of the YOLOv5s model.

[0063] Embodiment 3:

[0064] The embodiment of the present application also provides a traffic sign detection system, comprising a memory and a processor, the memory has a computer program run by the processor stored thereon, and the computer program executes the traffic sign detection method when being run by the processor.

[0065] Embodiment 4:

[0066] The embodiment of the present application also provides a storage medium, the storage medium has a computer program stored thereon, and the computer program executes the traffic sign detection method when being run.

[0067] The above-described embodiments are only used to describe the preferred modes of the present application, and do not limit the scope of the present application, and various modifications and improvements to the technical solutions of the present application made by those skilled in the art without departing from the design spirit of the present application shall fall within the protection scope of the present application defined by the claims.

Claims

1. A method for detecting traffic signs, characterized in that, include: Step S1: Obtain the traffic sign dataset; Step S2: Train an improved YOLOv5s model based on the traffic sign dataset; Step S3: Input the real-time video stream from the vehicle monitoring system into the trained improved YOLOv5s model for traffic sign detection; In step S2, the YOLOv5s model undergoes lightweight design, feature prediction scale optimization, and feature fusion enhancement to obtain an improved YOLOv5s model. Specifically, channel pruning is performed on the YOLOv5s model, and 3x3 convolutions are replaced with depthwise separable convolutions to achieve lightweight design. In multi-scale fusion, 160×160 shallow features are introduced for feature prediction scale optimization. The 5C module is used to fuse the shallow and deep feature maps for feature enhancement. The 5C module consists of 5 convolutions. It is divided into three branches. The first branch first performs a 1×1 convolution, halving the number of channels, followed by a 3×3 convolution for further feature extraction. The second branch first performs a 3×3 convolution, followed by a 1×1 convolution, halving the number of channels again. The third branch inputs the feature map and performs a 1×1 convolution for dimensionality reduction, reducing the number of channels to half the original number while keeping the feature map size unchanged. The expression is as follows: Y1=f Conv1×1 (f Conv3×3 (I)) (1) Y2=f Conv3×3 (f Conv1×1 (I)) (2) Y3=f Conv1×1 (I) (3) Where I is the input feature map, Y1, Y2, and Y3 are the output feature maps of the three branches, respectively, and f Conv3×3 f represents a 3×3 two-dimensional convolution. Conv1×1 Represents a 1×1 two-dimensional convolution; Y1, Y2 and Y3 have the same size. Add Y1 and Y2 to get Y4, and then concat Y3 and Y4 to fuse the features. Y = Y3 ⊙ Y4 (5) in, The symbol represents the sum of the feature maps, with the number of channels remaining unchanged. ⊙ represents the sum of the number of channels, with the feature map remaining unchanged. Y4 represents the feature map output after performing ⊙ on Y1 and Y2. Y is the feature map output by the 5C module. The improved YOLOv5s model uses the EIoU Loss function instead of the GIoU Loss function of the YOLOv5s model.

2. A traffic sign detection device for implementing the traffic sign detection method of claim 1, characterized in that, include: The acquisition module is used to acquire traffic sign datasets; The training module is used to train an improved YOLOv5s model based on a traffic sign dataset; The detection module is used to input the real-time video stream from the vehicle monitoring system into the trained improved YOLOv5s model for traffic sign detection.

3. A traffic sign detection system, characterized in that, include: A memory and a processor, wherein the memory stores a computer program executed by the processor, the computer program performing the traffic sign detection method as described in claim 1 when executed by the processor.

4. A storage medium, characterized in that, The storage medium stores a computer program, which executes the traffic sign detection method as described in claim 1 when it runs.

Images