A method and device for detecting damaged traffic signs

Abstract

The present application relates to a method and device for detecting damaged traffic signs, which includes collecting data on a number of damaged traffic signs and normal traffic signs to construct a data set; labeling the data set to obtain a training set; presetting a YOLOV5 model and designing a fine-grained loss function module for the YOLOV5 model, wherein the fine-grained loss function module includes a discriminative component and a diversity component, the discriminative component is used to ensure that feature maps at different levels contain sufficient discriminative information, and the diversity component is used to maintain the diversity and interactivity of feature maps at different levels; inputting the training set into the YOLOV5 model, combining the fine-grained loss function module to perform model training until the preset conditions are met, and outputting the trained model as a target detection model. The present application has the effect of capturing more local subtle discriminative features, better distinguishing damaged traffic signs from normal traffic signs, and improving the detection accuracy of damaged traffic signs.

Description

Technical Field

[0001] The present application relates to the field of intelligent transportation technology, and in particular to a method and device for detecting damaged traffic signs. Background Art

[0002] Road traffic signs are crucial for ensuring safe and smooth traffic flow. However, due to a variety of factors, including natural, human, and administrative factors, they can be damaged or missing. This not only damages the public image of road management departments but also poses significant safety risks to drivers and pedestrians. To promptly detect and repair damaged traffic signs, ensure their proper function, and provide scientific evidence and decision-making support for road management departments, the problem of detecting and identifying damaged traffic signs has become a pressing issue.

[0003] Currently, there is a lack of technology for detecting and identifying damaged traffic signs. Furthermore, because damaged traffic signs are similar in appearance to intact ones, and the damage states are complex and diverse, traditional object detection methods struggle to capture the detailed local damage features of damaged traffic signs, making it difficult to distinguish them from intact ones. This results in low object detection accuracy.

[0004] With respect to the above-mentioned related technologies, the inventors found that existing target detection methods have the problem of difficulty in extracting subtle local features of damaged traffic signs, resulting in low detection accuracy. Summary of the Invention

[0005] In order to improve the detection accuracy of damaged traffic signs, the present application provides a damaged traffic sign detection method and device.

[0006] In a first aspect, the present application provides a method for detecting damaged traffic signs.

[0007] This application is achieved through the following technical solutions:

[0008] A method for detecting damaged traffic signs comprises the following steps:

[0009] Collect data from several damaged traffic signs and normal traffic signs to build a data set;

[0010] Annotate the data set to obtain a training set;

[0011] Preset the YOLOV5 model and design a fine-grained loss function module for the YOLOV5 model, wherein the fine-grained loss function module includes a discriminative component and a diversity component, the discriminative component is used to ensure that feature maps at different levels contain sufficient discriminative information, and the diversity component is used to maintain the diversity and interactivity of feature maps at different levels;

[0012] Input the training set into the YOLOV5 model, perform model training in combination with the fine-grained loss function module until the preset conditions are met, and output the trained model as the target detection model;

[0013] Damaged traffic signs are detected based on the target detection model.

[0014] In a preferred example, the present application can be further configured as follows: the discriminative component includes a first SE channel attention mechanism layer, a positive and negative sample pair selection layer, a cross-channel maximum pooling layer, a global average pooling layer and a first Softmax layer;

[0015] The first SE channel attention mechanism layer is used to reconstruct the weight of the expanded feature map, increase the weight of the key feature channel and reduce the weight of the redundant feature channel; the positive and negative sample pair selection layer is used to determine the target level negative sample feature map of the damage mark and the target level positive sample feature map of the normal mark; the cross-channel maximum pooling layer is used to perform a cross-channel maximum pooling on the negative sample feature map and the positive sample feature map to obtain a dimensionality reduction feature map;

[0016] The global average pooling layer is used to perform global average pooling on the dimensionality reduction feature map in the spatial domain to obtain a compressed feature map containing discriminative information of positive and negative samples;

[0017] The first Softmax layer is used to perform cross entropy calculation based on the compressed feature map in combination with the true label, and predict the discriminative loss value of any group of positive and negative sample feature maps.

[0018] In a preferred example, the present application can be further configured as follows: the calculation formula of the discriminative loss value of any set of positive and negative sample feature maps includes:

[0019]

[0020] Where, is the discriminative loss function value of any set of positive and negative sample feature maps, is the negative sample feature map, is the positive sample feature map, j = 1, 2, 3 represent the low, medium and high level feature maps respectively, L CE is the cross entropy loss function, y is the true label, is the dimension reduction and compression operation of the negative sample feature map, is the dimensionality reduction and compression operation of the positive sample feature map, and e is an exponential function.

[0021] In a preferred example, the present application may be further configured as follows: the discriminative component further includes a cross entropy calculation layer and an averaging layer, the input end of the cross entropy calculation layer is connected to the output end of the first Softmax layer, and the output end of the cross entropy calculation layer is connected to the input end of the averaging layer;

[0022] The cross entropy calculation layer is used to predict the discriminative loss value of the positive and negative sample feature maps at any level;

[0023] The averaging layer is used to design a discriminative loss function based on the discriminative loss values of the positive and negative sample feature maps at any level.

[0024] In a preferred example, the present application can be further configured as follows: the expression of the discriminative loss function includes:

[0025]

[0026]

[0027] Where, L dis is the discriminative loss function value, is the discriminative loss value of the positive and negative sample feature maps at any level, N0 is the number of damaged traffic signs, and i is used to count the number of each damaged traffic sign, that is, every damaged traffic sign from the 1st to the N0th is involved in the calculation.

[0028] In a preferred example, the present application can be further configured as follows: the diversity component includes a second SE channel attention mechanism layer, a maximum pooling layer, a flattening layer, a cosine similarity calculation layer, and a second Softmax layer.

[0029] The second SE channel attention mechanism layer is used to reconstruct the weights of the expanded feature map, increase the weights of key feature channels and reduce the weights of redundant feature channels;

[0030] The maximum pooling layer is used to align the size of the feature map of the expanded reconstruction weight, highlight the part with the maximum response value in the feature map, and obtain a tensor of uniform size for feature maps at different levels;

[0031] The flattening layer is used to flatten and compress the tensor in the spatial dimension;

[0032] The cosine similarity calculation layer is used to obtain the cosine similarity between the two tensors;

[0033] The second Softmax layer is used to calculate the similarity between feature maps of different levels according to the scalar of the cosine similarity, and design a diversity loss function based on the similarity.

[0034] In a preferred example, the present application can be further configured as follows: the expression of the diversity loss function includes:

[0035] L div =αSim 1,3 ―βSim 2,3 ―γSim 1,2

[0036] Where, L div is the diversity loss function value, Sim 1,3 is the similarity between the low-level feature map and the high-level feature map, Sim 2,3 is the similarity between the middle-level feature map and the high-level feature map, Sim 1,2 is the similarity between the low-level feature map and the middle-level feature map, and α, β, and γ are all hyperparameters greater than zero.

[0037] In a second aspect, the present application provides a damaged traffic sign detection device.

[0038] This application is achieved through the following technical solutions:

[0039] A damaged traffic sign detection device, comprising:

[0040] The raw data module is used to collect data from several damaged traffic signs and normal traffic signs to construct a data set;

[0041] A training data module, used to label the data set to obtain a training set;

[0042] A fine-grained module is used to preset the YOLOV5 model and design a fine-grained loss function module for the YOLOV5 model. The fine-grained loss function module includes a discriminative component and a diversity component. The discriminative component is used to ensure that feature maps at different levels contain sufficient discriminative information, and the diversity component is used to maintain the diversity and interactivity of feature maps at different levels.

[0043] A training module is used to input the training set into the YOLOV5 model, perform model training in combination with the fine-grained loss function module until the preset conditions are met, and output the trained model as the target detection model;

[0044] A detection module is used to detect damaged traffic signs based on the target detection model.

[0045] In a third aspect, the present application provides a computer device.

[0046] This application is achieved through the following technical solutions:

[0047] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any one of the above-mentioned methods for detecting damaged traffic signs are implemented.

[0048] In a fourth aspect, the present application provides a computer-readable storage medium.

[0049] This application is achieved through the following technical solutions:

[0050] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any one of the above-mentioned methods for detecting damaged traffic signs.

[0051] In summary, compared with the prior art, the technical solution provided by this application has at least the following beneficial effects:

[0052] Collect data from several damaged traffic signs and normal traffic signs, construct a data set as the original data for model training; label the data set to obtain a training set for subsequent model training; preset the YOLOV5 model, and design the fine-grained loss function module of the YOLOV5 model, wherein the fine-grained loss function module includes a discriminative component and a diversity component, input the training set into the YOLOV5 model, and perform model training in combination with the fine-grained loss function module until the preset conditions are met, output the trained model as the target detection model, and perform damaged traffic sign detection based on the target detection model; wherein the diversity component can maintain the diversity and interactivity of feature maps at different levels, and the discriminative component can make the feature maps at different levels contain sufficient discriminative information, ensuring that the feature maps extracted and transformed from each layer contain sufficient Discriminative information, and plays a guiding role in diversity components to reduce the loss of key information in feature maps at different levels, so that the improved model can pay attention to local areas with discriminative information at different levels to capture more local subtle discriminative features, and then better capture the local subtle damage features of damaged traffic signs, so as to better distinguish damaged traffic signs from normal traffic signs, greatly improving the detection accuracy of damaged traffic signs, and achieving better fine-grained target detection effects; at the same time, the target detection model of the present application does not require complex network design, and can improve the detection accuracy without increasing the number of parameters and calculations in the reasoning process, that is, it will not affect the speed of the detection reasoning process of the model, and can be extended to other target detection algorithms containing multiple levels, and is more applicable. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of the main flow of a damaged traffic sign detection method provided by an exemplary embodiment of the present application.

[0054] Figure 2 A schematic diagram of the working principle of a damaged traffic sign detection method provided as another exemplary embodiment of the present application.

[0055] Figure 3 A schematic diagram of the grid structure of a fine-grained loss function module of an improved YOLOV5 model for a damaged traffic sign detection method provided as an exemplary embodiment of the present application.

[0056] Figure 4 A schematic diagram of the overall grid structure of an improved YOLOV5 model for a damaged traffic sign detection method provided in another exemplary embodiment of the present application.

[0057] Figure 5 A schematic diagram of the grid structure of the GSConv module of the improved YOLOV5 model for a damaged traffic sign detection method provided in an exemplary embodiment of the present application.

[0058] Figure 6 A schematic diagram of the grid structure of the GS bottleneck submodule and the VoVGSCSP submodule in the GSConv module of an improved YOLOV5 model for a damaged traffic sign detection method provided by an exemplary embodiment of the present application.

[0059] Figure 7 A graph showing the visualization results of GradCAM heat maps at various levels during the detection process of a damaged traffic sign detection method provided by an exemplary embodiment of the present application.

[0060] Figure 8 An example diagram of damaged traffic sign detection provided by a damaged traffic sign detection method according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0061] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed, but as long as they are within the scope of the claims of the present application, they are protected by the patent law.

[0062] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0063] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document, unless otherwise specified, generally indicates an "or" relationship between the related objects.

[0064] The detection and recognition of damaged traffic signs is an important issue in intelligent transportation systems. It involves separating areas that may be traffic signs from complex scenes and classifying them. Existing technologies mainly use deep learning methods to perform unified target detection on all traffic signs, and extract the specific information they represent, such as speed limit, height limit, etc. through classifier recognition. However, since damaged traffic signs are similar to normal and intact traffic signs in appearance, and the damage conditions are complex and diverse, it is difficult to capture the more detailed local damage features of damaged traffic signs using traditional target detection methods, and it is impossible to distinguish them from normal traffic signs, resulting in low accuracy. In addition, there is also the problem of difficult acquisition of damaged traffic sign data.

[0065] To this end, this application proposes a method for detecting damaged traffic signs. This method is based on the improved YOLOV5 fine-grained damaged traffic sign detection. By adding an innovatively designed fine-grained loss function, the Mutual Feature Layers Loss module, to the training process of the traditional target detection model, the fine-grained loss function enables the model to capture more local subtle discriminative features, thereby achieving better fine-grained target detection results. This method has the advantages of simplicity, high performance, high precision, and easy migration. At the same time, this method does not require complex network design and can improve the model detection accuracy while not affecting the speed of the model detection reasoning process. It can also be extended to other target detection algorithms with multiple layers.

[0066] The embodiments of the present application are described in further detail below with reference to the accompanying drawings.

[0067] Reference Figure 1 , an embodiment of the present application provides a method for detecting damaged traffic signs, and the main steps of the method are described as follows.

[0068] S1: Collect data of several damaged traffic signs and normal traffic signs to build a data set;

[0069] S2: labeling the data set to obtain a training set;

[0070] S3: Preset the YOLOV5 model and design a fine-grained loss function module of the YOLOV5 model, wherein the fine-grained loss function module includes a discriminative component and a diversity component, the discriminative component is used to make the feature maps of different levels contain sufficient discriminative information, and the diversity component is used to maintain the diversity and interactivity of the feature maps of different levels;

[0071] S4: Input the training set into the YOLOV5 model, perform model training in combination with the fine-grained loss function module until the preset conditions are met, and output the trained model as the target detection model;

[0072] S5: Detect damaged traffic signs based on the target detection model.

[0073] Reference Figure 2 and Figure 3 In one embodiment, the discriminative component includes a first SE channel attention mechanism layer, a positive and negative sample pair selection layer, a cross-channel maximum pooling layer, a global average pooling layer and a first Softmax layer;

[0074] The first SE channel attention mechanism layer is used to reconstruct the weights of the expanded feature map, increase the weights of key feature channels and reduce the weights of redundant feature channels;

[0075] The positive and negative sample pair selection layer (P / N pair selection) is used to determine the target level negative sample feature map of the damaged mark and the target level positive sample feature map of the normal mark

[0076] The cross-channel maximum pooling layer (CCMP) is used to perform a cross-channel maximum pooling on the negative sample feature map and the positive sample feature map to obtain a dimensionality reduction feature map;

[0077] The global average pooling layer (GAP) is used to perform global average pooling on the dimensionality reduction feature map in the spatial domain to obtain a compressed feature map containing discriminative information of positive and negative samples;

[0078] The first Softmax layer is used to perform cross entropy calculation based on the compressed feature map in combination with the true label, and predict the discriminative loss value of any group of positive and negative sample feature maps.

[0079] In one embodiment, the calculation formula for the discriminative loss value of any set of positive and negative sample feature maps includes:

[0080]

[0081] Where, is the discriminative loss function value of any set of positive and negative sample feature maps, is the negative sample feature map, is the positive sample feature map, j = 1, 2, 3 represent the low, medium and high level feature maps respectively, L CE is the cross entropy loss function, y is the true label, is the dimension reduction and compression operation of the negative sample feature map, is the dimensionality reduction and compression operation of the positive sample feature map, and e is an exponential function.

[0082] In one embodiment, the discriminative component further includes a cross entropy calculation layer and an averaging layer, wherein the input end of the cross entropy calculation layer is connected to the output end of the first Softmax layer, and the output end of the cross entropy calculation layer is connected to the input end of the averaging layer;

[0083] The cross-entropy calculation layer is used to predict the discriminative loss value of the positive and negative sample feature maps at any level;

[0084] The average layer is used to design a discriminative loss function based on the discriminative loss values of the positive and negative sample feature maps at any level.

[0085] In one embodiment, the expression of the discriminative loss function includes:

[0086]

[0087]

[0088] Where, L dis is the discriminative loss function value, is the discriminative loss value of the positive and negative sample feature maps at any level, N0 is the number of damaged traffic signs, and i is used to count the number of each damaged traffic sign, that is, every damaged traffic sign from the 1st to the N0th is involved in the calculation.

[0089] In one embodiment, the diversity component includes a second SE channel attention mechanism layer, a maximum pooling layer, a flattening layer, a cosine similarity calculation layer, and a second Softmax layer.

[0090] The second SE channel attention mechanism layer is used to reconstruct the weights of the expanded feature map, increase the weights of key feature channels and reduce the weights of redundant feature channels;

[0091] The maximum pooling layer (Maxpool) is used to align the size of the feature map of the expanded reconstruction weight, highlight the part with the maximum response value in the feature map, and obtain a tensor of uniform size for feature maps at different levels;

[0092] The flattening layer (Flatten) is used to flatten and compress the tensor in the spatial dimension;

[0093] The cosine similarity calculation layer (Compute Cosine Similarity) is used to obtain the cosine similarity between the tensors;

[0094] The second Softmax layer is used to calculate the similarity between feature maps of different levels according to the scalar of the cosine similarity, and design a diversity loss function based on the similarity.

[0095] In one embodiment, the expression of the diversity loss function includes:

[0096] L div =αSim 1,3 ―βSim 2,3 ―γSim 1,2

[0097] Where, L div is the diversity loss function value, Sim 1,3 is the similarity between the low-level feature map and the high-level feature map, Sim 2,3 is the similarity between the middle-level feature map and the high-level feature map, Sim 1,2 is the similarity between the low-level feature map and the middle-level feature map, and α, β, and γ are all hyperparameters greater than zero.

[0098] The above embodiments are described in detail as follows.

[0099] Reference Figure 2 , data from several damaged and normal traffic signs were collected to form a dataset. In this example, the experimental data was derived from images of normal and damaged traffic signs collected from various roads in Guangzhou. A total of 1,134 traffic sign images were collected through manual photography, including 503 images of damaged traffic signs and 631 images of normal traffic signs. Damaged traffic signs included signs with surface wear, edge damage, tilt, and collapse.

[0100] Large amounts of high-quality training data are crucial for improving machine learning performance. Therefore, data augmentation preprocessing of datasets of damaged and normal traffic signs can be performed using the Mosaic data augmentation method to improve data robustness. Furthermore, data augmentation methods such as horizontal and vertical flipping, random rotation, random cropping, deformation, and scaling can be used to increase the sample size of types with limited data. This approach, in turn, increases the dataset size and improves the model's generalization capabilities.

[0101] The dataset after data augmentation preprocessing is manually annotated to form a training set. In this embodiment, the processed dataset is annotated using the data annotation tool labelme. The annotation categories include: BROKEN (broken traffic sign) and NORMAL (normal traffic sign). The annotation format is YOLO txt format, and the annotation file contains the position, size, and category information of the annotation box.

[0102] Next, the pre-processed dataset is divided into a training set and a validation set using random sampling, for example, with a ratio of 8:2. The samples in the training set will be used for model training, while the samples in the validation set will not participate in the model training process and will only be used for performance evaluation of the final model.

[0103] Reference Figure 3 Based on the original loss function, the YOLOV5 model introduces a fine-grained loss function, the Mutual Feature Layer Loss module. This module includes a discriminative component, which ensures that feature maps at different levels contain sufficient discriminative information, and a diversity component, which maintains the diversity and interactivity of feature maps at different levels. The extracted feature maps at different levels are fed into the discriminative component and then into the same diversity component. This allows the feature maps at different levels to focus on different local areas of the target, capturing subtle local damage features that may exist in traffic signs. This process only adds a small amount of computation to the training process and does not increase the computational complexity of the inference process, which would slow down the inference process.

[0104] Among them, the specific implementation method of the diversity component of the fine-grained loss function Mutual Feature Layer Loss module is as follows: the low, medium and high-level feature maps used for final detection in YOLOV5 are first subjected to a convolution change, and the channels are expanded to 512 dimensions while keeping the width and height unchanged, so that the converted feature map is sufficient to accommodate a rich variety of damaged features. Since the increase in channels of the converted feature map may increase redundant information, the SE channel attention mechanism is used to reconstruct the weights of the expanded feature map, so that the model gives greater weight to important feature channels and reduces attention to redundant parts. Then, a Maxpool maximum pooling layer is used to align the size of the converted feature map, highlight the part with the maximum response value in the feature map and map it to the new feature map. At this time, the original three feature maps of different scales are unified into a tensor of size (512,8,8). These three tensors are then flattened and compressed in the spatial dimension for calculating the cosine similarity Sim between each pair. 1,2 ,Sim 1,3 ,Sim 2,3. After calculating the cosine similarity between the three tensors, three cosine similarity scalars will be obtained. After the softmax calculation, the similarity will be mapped to the (0,1) interval to obtain the final similarity between the feature maps at different levels. Through visualization methods, it can be observed that the feature areas focused on by the feature maps at different levels in the original YOLOV5 method are very similar, which is not conducive to capturing the subtle local differences of similar objects. Therefore, we will separate the feature maps at different levels in the feature space by reducing their similarity. In YOLOV5, low-level feature maps focus on more detailed features, high-level feature maps focus on more global features, and middle-level feature maps are in between. Here we choose to widen the similarity between the high-level and low-level feature maps, so that the low-level feature maps can focus on more details of the object rather than the overall focus of the high-level feature maps. In YOLOV5, the middle-level feature map is a bridge connecting the low-level and high-level feature maps. Its interaction with the low-level and high-level feature maps will help the model combine local features and overall features for prediction and judgment. If the similarity between all feature maps is widened, the interactivity between each feature map may be destroyed. Therefore, while widening the similarity between the low-level and high-level feature maps, it is necessary to narrow the similarity between the low-level feature map and the middle-level feature map, and between the high-level feature map and the middle-level feature map, so that the diversity between the feature maps is maintained while also retaining good interactivity. The final diversity loss function is expressed as:

[0105] L div =αSim 1,3 ―βSim 2,3 ―γSim 1,2

[0106] Where, L div is the diversity loss function value, Sim 1,3 is the similarity between the low-level feature map and the high-level feature map, Sim 2,3 is the similarity between the middle-level feature map and the high-level feature map, Sim 1,2 is the similarity between the low-level feature map and the middle-level feature map, and α, β, and γ are all hyperparameters greater than zero.

[0107] In this embodiment, according to experimental results, in this data set, the best effect is achieved when α=4, β=γ=0.5 in the diversity loss function.

[0108] Furthermore, the discriminative component of the fine-grained loss function Mutual Feature Layer Loss module is specifically implemented as follows: the images in the labeled boxes of the training set are mapped to the feature maps of the low, medium and high levels respectively. In each layer, the feature map of each box with the true value of the damaged mark in a batch is compared with the feature map of the box with the true value of the normal mark, and the closest pair of positive and negative samples is selected by calculating the cosine similarity. This pair of positive and negative sample feature maps is subjected to the same convolution transformation as the diversity component, the channel is expanded to 512 dimensions and the SE attention mechanism is used to obtain the positive and negative sample feature maps. Represents the feature maps of low, medium and high levels, 0 represents damage and 1 represents normal). Then a cross-channel maximum pooling (CCMP cross-channel maximum pooling) is performed to map the part with the largest response peak in the feature map to the new feature map. At this time, the number of channels is reduced to 1 dimension again. Then global average pooling is performed in the spatial domain. At this time, the positive and negative sample feature maps will be compressed into two scalars Its concentration contains the discriminative information of each. Then it is softmax transformed and cross entropy is calculated with the real label to form the discriminative loss function between the two. Since there is more than one pair of positive and negative samples in a batch, the number of damaged marks in each batch is recorded as N0, then the discriminative loss function of this layer is The final discriminative loss function is The core purpose of the discriminative loss is to ensure that the feature maps extracted and transformed from each layer contain sufficient discriminative information. In addition, since the operation of widening the similarity between high-level and low-level feature maps in the diversity component is random, it is possible that the stretched feature maps will lose key information. Therefore, the discriminative component also plays an important guiding role in the diversity component. The combination of the two can guide the model to focus on different local areas with discriminative information.

[0109] The calculation formula of the final fine-grained loss function Mutual Feature Layer Loss is:

[0110] L MFL =L dis +λL div

[0111] Among them, L dis is the discriminative loss function value, L div is the value of the diversity loss function, and λ is a hyperparameter greater than zero.

[0112] In this embodiment, according to experimental results, in this data set, the best effect is achieved when λ is 1.

[0113] Reference Figure 4 This application is based on the improved YOLOV5 model to detect damaged traffic signs. The improved YOLOV5 model can be called the MFL-YOLO model. The network structure of the improved YOLOV5 model is as follows: Figure 4 shown.

[0114] Among them, in order to improve the model's ability to extract key features, this application adds the SE channel attention mechanism to the backbone part of the traditional YOLOV5 model. By adding an attention mechanism module to the traditional YOLOV5 model, different weights are assigned to different features, enabling the network to extract key features more effectively.

[0115] Furthermore, the lightweight convolution GSConv module is used in the Neck section of the traditional YOLOV5 model to replace the standard convolution. Its computational cost is about 60%-70% of the standard convolution, but its contribution to the model's learning ability is comparable to the latter. GSConv is a new convolution operation that combines dense convolution and sparse convolution.

[0116] Based on the GSConv module, the GS bottleneck submodule is further introduced, and the one-shot aggregation method is used to design the cross-level partial network VoV-GSCSP submodule. The VoV-GSCSP submodule replaces the traditional CSP module in the original Neck part of YOLO5, which reduces the computational efficiency of the model by an average of 15.72% in FLOPS compared to the original one, completing the lightweight improvement of the Neck part of the traditional YOLOV5 model.

[0117] Among them, the network structures of GSConv module, Gsbottleneck submodule and VoV-GSCSP submodule are as follows: Figure 5 and Figure 6 shown.

[0118] Reference Figure 5 In addition to using L1 regularization to sparsify the convolution kernel, the GSConv module of this application also utilizes two technologies, namely Dynamic Receptive Field (DRF) and Half Dense Connection (HDC), to retain certain dense convolution characteristics.

[0119] DRF technology can adaptively adjust the receptive field size of the convolution kernel during the convolution process to adapt to different image features. In GSConv, each convolution kernel has a different receptive field size, which can improve the expressiveness and accuracy of the model.

[0120] HDC technology introduces a certain proportion of fully connected operations in each set of convolution operations to preserve some of the characteristics of dense connectivity. In each set of convolution kernels, some are fully connected, while others are sparsely connected. This reduces model parameters and computational complexity while retaining some of the characteristics of dense convolution, improving the model's expressiveness and accuracy.

[0121] Reference Figure 6 The GS bottleneck submodule and the VoV-GSCSP submodule replace some standard convolutions in the original bottleneck of yolov5 with GSConv. This improvement can reduce the number of parameters and computational complexity of the structure while improving the accuracy of the model.

[0122] Next, a lightweight YOLOV5 model is used to extract feature maps of each detection layer, including low-level feature maps, mid-level feature maps, and high-level feature maps.

[0123] Reference Figure 2 The training set is input into the improved MFL-YOLO model for training. Specifically, an RTX3060 graphics card is used and the SGD optimizer is used to train the model for 400 iterations. The performance is evaluated using the data in the validation set.

[0124] For example, the main evaluation indicator used in this application is mAP. mAP is a measure of the average accuracy of an object detector across all categories. For each category, mAP calculates the average (Average) of the AP under that category. In object detection, AP is used to evaluate the accuracy of the model for a single category. It is the area under the Precision and Recall curves (AUC). More specifically, for each category, we calculate Precision and Recall at different IOU thresholds, and then calculate an AP value. The average of the APs of all categories is mAP, which is used to comprehensively evaluate the performance of the model in multiple categories.

[0125] Until the model meets the preset conditions, the best model is selected and the trained model is output as the target detection model.

[0126] In this embodiment, the preset condition may be to select the weight of the training result with the highest mAP during the training process as the final trained model.

[0127] In order to verify the detection performance of the MFL-YOLO model of this application, the traditional YOLOV5s, the latest yolov8x, and yolov8s were trained respectively using the same data set and data partitioning training set, and the training results of each model were compared with the training results of the MFL-YOLO model of this application. The specific results are shown in Table 1 below.

[0128] Table 1

[0129]

[0130] As can be seen from the table, the mAP of the MFL-YOLO model of this application is 92.6%, which is much higher than the detection accuracy of YOLOv5s and the latest YOLOv8 series algorithms. In addition, the model has also been optimized to varying degrees in terms of computational complexity, computational speed, and volume.

[0131] This application can improve the detection accuracy while meeting the lightweight requirements, and can be well embedded in the smart maintenance mobile terminal to realize automatic detection of damaged traffic signs.

[0132] The results of GradCAM heat map visualization at each level during the damaged traffic sign detection process are shown in the figure below. Figure 7 As shown in the figure, we can see that compared with the traditional YOLOV5 model, the improved model is more able to focus on the local damage characteristics of traffic signs.

[0133] Damaged traffic signs are detected based on the target detection model. Specifically, in mobile terminal devices, the MFL-YOLO model can be embedded and deployed, and only the detection results of damaged signs can be displayed. The video stream collected by the terminal monitoring device is input into the MFL-YOLO model to obtain the detection results of damaged traffic signs, so as to realize automatic detection of damaged traffic signs in the video stream output by the terminal device. The detection example is as follows: Figure 8 shown.

[0134] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0135] In summary, a damaged traffic sign detection method is based on an improved YOLOV5 model, and an innovative fine-grained loss function is designed, so that the model can better capture the local subtle damage features of damaged traffic signs, thereby better distinguishing them from normal traffic signs, and can effectively detect damaged traffic signs. This method does not increase the number of parameters and computational complexity of the reasoning process. At the same time, this application combines the lightweight convolution design of GhostNet and streamlines the Neck part of the traditional YOLOV5 model, so that the model improves the detection accuracy while reducing the number of parameters and computational complexity. It can be better applied to intelligent maintenance and detection mobile terminals to realize intelligent detection of damaged traffic signs.

[0136] The present application also provides a damaged traffic sign detection device, which corresponds to the damaged traffic sign detection method in the above embodiment. The damaged traffic sign detection device includes:

[0137] The raw data module is used to collect data from several damaged traffic signs and normal traffic signs to construct a data set;

[0138] A training data module, used to label the data set to obtain a training set;

[0139] A fine-grained module is used to preset the YOLOV5 model and design a fine-grained loss function module for the YOLOV5 model. The fine-grained loss function module includes a discriminative component and a diversity component. The discriminative component is used to ensure that feature maps at different levels contain sufficient discriminative information, and the diversity component is used to maintain the diversity and interactivity of feature maps at different levels.

[0140] A training module is used to input the training set into the YOLOV5 model, perform model training in combination with the fine-grained loss function module until the preset conditions are met, and output the trained model as the target detection model;

[0141] A detection module is used to detect damaged traffic signs based on the target detection model.

[0142] The specific definitions of a damaged traffic sign detection device can be found in the definitions of a damaged traffic sign detection method described above and will not be further elaborated here. Each module in the damaged traffic sign detection device described above may be implemented in whole or in part through software, hardware, or a combination thereof. Each of these modules may be embedded in or independent of a processor within a computer device in hardware form, or may be stored in a computer device memory in software form, allowing the processor to call and execute the corresponding operations of each module.

[0143] In one embodiment, a computer device is provided, which may be a server. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When executed by the processor, the computer program implements any of the above-mentioned methods for detecting damaged traffic signs.

[0144] In one embodiment, a computer-readable storage medium is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0145] S1: Collect data of several damaged traffic signs and normal traffic signs to build a data set;

[0146] S2: labeling the data set to obtain a training set;

[0147] S3: Preset the YOLOV5 model and design a fine-grained loss function module of the YOLOV5 model, wherein the fine-grained loss function module includes a discriminative component and a diversity component, the discriminative component is used to make the feature maps of different levels contain sufficient discriminative information, and the diversity component is used to maintain the diversity and interactivity of the feature maps of different levels;

[0148] S4: Input the training set into the YOLOV5 model, perform model training in combination with the fine-grained loss function module until the preset conditions are met, and output the trained model as the target detection model;

[0149] S5: Detect damaged traffic signs based on the target detection model.

[0150] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0151] Those skilled in the art will clearly understand that for the sake of convenience and brevity in description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A method for detecting damaged traffic signs, characterized in that: The following steps are included: Collect data from several damaged traffic signs and normal traffic signs to build a data set; Annotate the data set to obtain a training set; Preset the YOLOV5 model and design a fine-grained loss function module for the YOLOV5 model, wherein the fine-grained loss function module includes a discriminative component and a diversity component, the discriminative component is used to ensure that feature maps at different levels contain sufficient discriminative information, and the diversity component is used to maintain the diversity and interactivity of feature maps at different levels; The diversity component includes a second SE channel attention mechanism layer, a maximum pooling layer, a flattening layer, a cosine similarity calculation layer, and a second Softmax layer. The second SE channel attention mechanism layer is used to reconstruct the weights of the expanded feature maps, increase the weights of key feature channels, and reduce the weights of redundant feature channels. The maximum pooling layer is used to align the size of the feature map of the expanded reconstruction weight, highlight the part with the maximum response value in the feature map, and obtain a tensor of uniform size for feature maps at different levels; The flattening layer is used to flatten and compress the tensor in the spatial dimension; The cosine similarity calculation layer is used to obtain the cosine similarity between the two tensors; The second Softmax layer is used to calculate the similarity between feature maps of different levels according to the scalar of the cosine similarity, and design a diversity loss function based on the similarity; Input the training set into the YOLOV5 model, perform model training in combination with the fine-grained loss function module until the preset conditions are met, and output the trained model as the target detection model; Damaged traffic signs are detected based on the target detection model.

2. The damaged traffic sign detection method according to claim 1, characterized in that: The discriminative component includes the first SE channel attention mechanism layer, the positive and negative sample pair selection layer, the cross-channel maximum pooling layer, the global average pooling layer and the first Softmax layer; The first SE channel attention mechanism layer is used to reconstruct the weights of the expanded feature map, increase the weights of key feature channels and reduce the weights of redundant feature channels; The positive-negative sample pair selection layer is used to determine the target-level negative sample feature map of the damaged mark and the target-level positive sample feature map of the normal mark; The cross-channel maximum pooling layer is used to perform a cross-channel maximum pooling on the negative sample feature map and the positive sample feature map to obtain a dimensionality reduction feature map; The global average pooling layer is used to perform global average pooling on the dimensionality reduction feature map in the spatial domain to obtain a compressed feature map containing discriminative information of positive and negative samples; The first Softmax layer is used to perform cross entropy calculation based on the compressed feature map in combination with the true label, and predict the discriminative loss value of any group of positive and negative sample feature maps.

3. The damaged traffic sign detection method according to claim 2, characterized in that: The calculation formula for the discriminative loss value of any set of positive and negative sample feature maps includes: Where, is the discriminative loss function value of any set of positive and negative sample feature maps, is the negative sample feature map, is the positive sample feature map, Represent low-level, medium-level, and high-level feature maps, respectively. is the cross entropy loss function, is the true label, is the dimension reduction and compression operation of the negative sample feature map, is the dimensionality reduction and compression operation of the positive sample feature map, is an exponential function.

4. The damaged traffic sign detection method according to claim 3, characterized in that: The discriminative component further includes a cross entropy calculation layer and an averaging layer, wherein the input end of the cross entropy calculation layer is connected to the output end of the first Softmax layer, and the output end of the cross entropy calculation layer is connected to the input end of the averaging layer; The cross entropy calculation layer is used to predict the discriminative loss value of the positive and negative sample feature maps at any level; The averaging layer is used to design a discriminative loss function based on the discriminative loss values of the positive and negative sample feature maps at any level.

5. The damaged traffic sign detection method according to claim 4, characterized in that: The expression of the discriminative loss function includes, Where, is the discriminative loss function value, is the discriminative loss value of the positive and negative sample feature maps at any level, is the number of damaged traffic signs, i is used to count the number of each damaged traffic sign, that is, from the first number to the Every damaged traffic sign is involved in the calculation.

6. The damaged traffic sign detection method according to claim 1, characterized in that: The expression of the diversity loss function includes: Where, is the diversity loss function value, is the similarity between the low-level feature map and the high-level feature map, is the similarity between the middle-level feature map and the high-level feature map, is the similarity between the low-level feature map and the middle-level feature map, 、 、 are all hyperparameters greater than zero.

7. A damaged traffic sign detection device, characterized in that: include, The raw data module is used to collect data from several damaged traffic signs and normal traffic signs to construct a data set; A training data module, used to label the data set to obtain a training set; A fine-grained module is used to preset the YOLOV5 model and design a fine-grained loss function module for the YOLOV5 model, wherein the fine-grained loss function module includes a discriminative component and a diversity component. The discriminative component is used to make the feature maps of different levels contain sufficient discriminative information, and the diversity component is used to maintain the diversity and interactivity of the feature maps of different levels; the diversity component includes a second SE channel attention mechanism layer, a maximum pooling layer, a flattening layer, a cosine similarity calculation layer, and a second Softmax layer. The second SE channel attention mechanism layer is used to reweight the expanded feature maps. structure, increasing the weight of the key feature channel and reducing the weight of the redundant feature channel; the maximum pooling layer is used to align the size of the feature map of the expanded reconstruction weight, highlighting the part of the maximum response value in the feature map, and obtaining a tensor of uniform size of the feature maps of different levels; the flattening layer is used to flatten and compress the tensor in the spatial dimension; the cosine similarity calculation layer is used to obtain the cosine similarity between the two tensors; the second Softmax layer is used to calculate the similarity between the two feature maps of different levels according to the scalar of the cosine similarity, and design a diversity loss function based on the similarity; A training module is used to input the training set into the YOLOV5 model, perform model training in combination with the fine-grained loss function module until the preset conditions are met, and output the trained model as the target detection model; A detection module is used to detect damaged traffic signs based on the target detection model.

8. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images