Night traffic sign recognition method and system based on deep learning, and storage medium

Abstract

The application discloses a night traffic sign recognition method and system based on deep learning and a storage medium. The application relates to the technical field of intelligent traffic, and is built based on an existing deep neural network to form a night traffic sign recognition model, so that the night traffic sign can be accurately and quickly recognized in a night environment. The night traffic sign recognition method based on deep learning proposed by the application enhances the adaptability of the model to the night environment through image preprocessing and improved backbone network, effectively alleviates the problems of false detection and missed detection of the model to the traffic sign in the night environment, improves the recognition accuracy of the model, and ensures the recognition speed.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, and more specifically, to a method, system, and storage medium for nighttime traffic sign recognition based on deep learning. Background Technology

[0002] The core of intelligent vehicles is the driver assistance system (ADAS). Traffic sign detection and recognition is a crucial component of ADAS. With the continuous advancement of science and technology in my country, automobiles have become a primary means of transportation and an indispensable daily necessity for every household. As the automotive industry continues to develop and the number of vehicles increases, urban traffic congestion is worsening, making road traffic management increasingly difficult. Road congestion and frequent accidents pose severe challenges to traffic control and place immense pressure on drivers. Road traffic signs serve to convey vital information to drivers, including directions, warnings, and restrictions, guiding safe driving, ensuring smooth traffic flow, regulating vehicle volume, and reducing traffic accidents. Therefore, under specific conditions, the detection and recognition of road traffic signs will become increasingly important in the future.

[0003] Traffic sign detection technology is a key technology for achieving automated traffic sign maintenance, assisted driving, and autonomous driving. In automated traffic sign maintenance, traffic sign detection systems can promptly detect damaged traffic signs and report them to relevant departments, avoiding the inconvenience caused by delayed manual maintenance. In assisted driving, traffic sign detection systems can accurately and promptly provide drivers with a wealth of traffic sign information, reducing driver stress. In autonomous driving, autonomous vehicles can autonomously determine their speed and trajectory based on traffic signs detected by the system, ensuring safe and orderly driving while adhering to traffic rules. However, achieving a perfect balance between reliability and real-time performance in practical applications of traffic sign detection is challenging, limiting its direct application to complex urban scenarios. This is a key area worthy of further research. With the continuous maturation of deep learning research, convolutional neural networks have become a hot topic for many researchers. Neural networks built using convolutional structures typically have deep layers and relatively complex structures. Therefore, convolutional neural networks can extract rich feature information from feature maps, which is crucial for object detection. In recent years, convolutional neural networks (CNNs) have demonstrated excellent performance in image classification, segmentation, and object detection. Therefore, more researchers are inclined to utilize CNNs to address problems in traffic sign detection.

[0004] Most current research focuses on traffic sign recognition in well-lit and low-noise environments. At night, insufficient light significantly increases the difficulty of traffic sign detection, leading to frequent traffic accidents. Therefore, a traffic sign recognition algorithm suitable for nighttime operation is needed. Summary of the Invention

[0005] In order to overcome the shortcomings of the above-mentioned technical problems, the present invention provides a method, system and storage medium for nighttime traffic sign recognition based on deep learning, so as to realize the recognition of nighttime traffic signs and avoid traffic accidents caused by insufficient light and other problems.

[0006] Therefore, the present invention provides the following technical solution:

[0007] This invention provides a deep learning-based method for nighttime traffic sign recognition, comprising:

[0008] Obtain a dataset of nighttime traffic signs;

[0009] Preprocess the image;

[0010] A nighttime traffic sign recognition model is established based on the YOLOv5 network. The model includes: a backbone network for feature extraction, an attention and feature enhancement module for global feature map learning, a Neck network for feature fusion, and a detection head for selecting prediction boxes. The backbone network employs depthwise separable convolution to perform convolution operations on each input image. The resulting feature maps at multiple scales are fed into the attention and feature enhancement module for global feature map learning. The feature pyramid in the Neck network module is improved by adding an adaptive attention module and a spatial pyramid pooling module. Finally, the Neck network module fuses the feature maps at multiple scales, and the fused result is fed into the detection head for prediction box selection to obtain the final prediction boxes.

[0011] The nighttime traffic sign recognition model is trained using the nighttime traffic sign dataset.

[0012] The trained nighttime traffic sign recognition model is used to recognize nighttime traffic signs, and the nighttime traffic sign recognition results are obtained.

[0013] Furthermore, the image is preprocessed, including: image preprocessing using an improved adaptive histogram equalization algorithm; wherein:

[0014] The improved adaptive histogram equalization algorithm includes:

[0015] Divide the input image into several square blocks and calculate the histogram of each block.

[0016] Solve for the constraint values ​​of the constrained small block histogram;

[0017] For each small block, pixels are reassigned. When the histogram of a small block exceeds the limit, the excess part is cropped and evenly distributed to other gray levels.

[0018] Furthermore, the backbone network performs feature extraction, including:

[0019] For the preprocessed image, an M-dimensional feature image is generated, and the simultaneous convolution of both spatial and channel dimensions is changed to a step-by-step convolution of channel-wise convolution and point-wise convolution; where:

[0020] The channel-wise convolution includes: calculating the input feature image using convolution kernels respectively. The input image has M channels. M convolution kernels are used to operate on each of its channels respectively, and the results are stacked to obtain a feature map of M channels.

[0021] The pointwise convolution includes: using pointwise convolution to fuse the results of the channelwise convolution, weighting and combining N 1×1M convolution kernels with the output of the channelwise convolution, and finally generating a new N-dimensional feature map.

[0022] Furthermore, the Neck network module includes: an improved feature pyramid AS-PAN structure, an adaptive attention module, and a spatial pyramid pooling module; wherein:

[0023] An Attention Adaptation (AAM) module is added to the very front of the Neck section to focus limited attention more on traffic sign targets;

[0024] The Spatial Pyramid Pooling-Fast SPPF module is added before the P3 and P4 outputs to reduce feature loss and enhance feature representation.

[0025] Furthermore, the prediction box filtering includes: using the detection box fusion algorithm DBF to generate the optimal prediction box by changing the selection of redundant prediction boxes to the discarding method;

[0026] The detection bounding box fusion algorithm DBF includes:

[0027] Add all the current predicted boxes to a list named List B, and sort them in descending order of confidence.

[0028] Initialize lists C, L, and F. Predicted boxes and their corresponding values ​​are placed in lists B and C respectively. Each element in list L is a collection of boxes. List F contains the boxes that are merged from the L boxes.

[0029] Start iterating through B, matching boxes in list F whose IoU value is greater than the threshold; F is initially empty.

[0030] If no match is found, the box is added to the end of both list L and list F, and then the next box is processed; if the i-th element in F is matched, the box is placed in list L at the position corresponding to the element in list F.

[0031] Merge the accumulated bounding boxes at each position in list L, and recalculate the box coordinates. The merging formula is as follows:

[0032]

[0033]

[0034]

[0035] After traversing list B, update the confidence score in list F. The formula for the confidence score is:

[0036]

[0037] In the formula, T is the number of detection boxes, C is the confidence score of the fused box, Ci is the confidence score of each detection box in list L, and Xa i 、Xb i Ya i Yb i Here, N represents the coordinates of the fused bounding boxes in the updated list L, and N is the number of models. Further, obtain the nighttime traffic sign dataset, including:

[0038] Capture images of nighttime environments using image acquisition equipment, extract images from dashcams, and / or crawl nighttime traffic sign images from the web.

[0039] Furthermore, before image preprocessing, the process includes selecting a preset number of traffic signs as annotation objects and labeling the acquired nighttime images with the corresponding annotation object types.

[0040] This invention also provides a deep learning-based nighttime traffic sign recognition system, comprising:

[0041] The dataset acquisition module is used to acquire a dataset of nighttime traffic signs;

[0042] The image preprocessing module is used to process the nighttime images obtained by the dataset acquisition module into images with higher visibility.

[0043] The model building module is used to build a nighttime traffic sign recognition model based on the YOLOv5 network. The nighttime traffic sign recognition model includes: a backbone network for feature extraction, an attention and feature enhancement module for global feature map learning, a Neck network for feature fusion, and a detection head for filtering prediction boxes. The backbone network uses depthwise separable convolution to perform convolution operations on each input image. The obtained feature maps at multiple scales are fed into the attention and feature enhancement module, which globally learns the feature maps. The feature pyramid in the Neck network module is improved by adding an adaptive attention module and a spatial pyramid pooling module. Finally, the Neck network module fuses the feature maps at multiple scales, and the fused result is fed into the detection head for prediction box filtering to obtain the final prediction boxes.

[0044] The model training module is used to train the nighttime traffic sign recognition model using the nighttime traffic sign dataset;

[0045] The traffic sign recognition module is used to perform nighttime traffic sign recognition using the nighttime traffic sign recognition model trained by the model training module, and to obtain the nighttime traffic sign recognition result.

[0046] The present invention also provides a computer-readable storage medium storing a computer instruction set, which, when executed by a processor, implements the deep learning-based nighttime traffic sign recognition method described above.

[0047] The beneficial effects of this invention are:

[0048] The nighttime traffic sign recognition method of this invention first annotates traffic sign images in a nighttime environment, labeling ten common traffic signs in the images; then, a network model is established, and an improved image preprocessing method is used to enhance image brightness and contrast; next, the backbone network extracts features from the image, adding an attention module and a feature enhancement module, and a Neck network is used for feature fusion. The Neck part improves the feature pyramid, adding an adaptive attention module and a spatial pyramid pooling module; after training, it can be used to obtain a recognition model for various types of traffic signs, providing an effective image processing method for driving assistance systems, with significant beneficial effects and suitable for application and promotion. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart of the nighttime traffic sign recognition method in an embodiment of the present invention;

[0051] Figure 2 This is an image preprocessing diagram in an embodiment of the present invention;

[0052] Figure 3 This is a diagram of the convolutional pattern of the backbone network in an embodiment of the present invention;

[0053] Figure 4 This is a fusion prediction block diagram in an embodiment of the present invention;

[0054] Figure 5 This is a network structure diagram of the Neck network module in an embodiment of the present invention;

[0055] Figure 6 This is a structural block diagram of the nighttime traffic sign recognition model in an embodiment of the present invention. Detailed Implementation

[0056] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0057] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0058] like Figure 1As shown, the nighttime traffic sign recognition method based on deep learning in this embodiment of the invention specifically includes the following steps:

[0059] S1. Obtain the nighttime traffic sign dataset;

[0060] Images of nighttime environments are captured using devices such as cameras and mobile phones, images are extracted from dashcams, and nighttime traffic sign images are crawled from the internet to form a traffic sign dataset for nighttime environments.

[0061] S2, Image annotation;

[0062] On the nighttime images acquired by S1, ten types of traffic signs are selected as annotation objects, and the corresponding categories of the corresponding images are labeled with LabelImg;

[0063] S3, Image Preprocessing;

[0064] The loss of detail or partial distortion caused by nighttime images can lead to poor recognition results. In order to enable the network to learn image features better and improve the visibility of nighttime traffic sign images, it is necessary to preprocess the images to improve the brightness and contrast of the images without affecting the image resolution.

[0065] S4. Backbone network feature extraction;

[0066] The preprocessed image in S3 is used to extract features through the backbone of a convolutional neural network. In this embodiment, the standard convolution method of the backbone network is improved to a depthwise separable convolution method. This reduces the number of model parameters without reducing the feature extraction capability of the backbone network, thereby improving the model inference speed. The resulting network feature map shows that the size of the feature map gradually decreases while the depth gradually increases with the increase of network depth.

[0067] S5 and Neck network feature fusion;

[0068] In the Neck section, the feature pyramid is improved by adding an adaptive attention module and a spatial pyramid pooling module. Different weights are assigned to different parts, making the network pay more attention to key location information. This reduces information loss during feature map generation and enhances representational ability. The acquired 52×52, 26×26, and 13×13 feature maps are fused in this stage. The fact that all three feature maps are odd-sized ensures that the network has a central location. Simultaneously, three scales are output to detect large, medium, and small targets respectively. Specifically, the image features extracted in the backbone network are fused in the Neck section. After adding attention and feature enhancement modules, feature maps of three scales (20*20, 40*40, and 80*80) are fused. The feature maps are then directly accumulated across channels using a concat operation.

[0069] S6, Prediction Filter Box;

[0070] By appropriately lowering the confidence threshold, the network generates multiple prediction boxes. The prediction boxes are then sorted according to their confidence levels. The coordinates of the prediction boxes in the same cluster are recalculated based on their confidence scores to obtain the final prediction boxes.

[0071] S7. Model building and optimization;

[0072] An improvement based on the YOLOv5 network was made on a self-built dataset. The CPU used was an Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz, with 16GB of memory and an NVIDIA GeForce RTX 2080Ti GPU. The training, validation, and test sets were divided in a 7:2:1 ratio. The number of epochs was set to 250, and the batch size was set to 16. The model was built and trained using the training set data, and the parameters were adjusted during the training process using the validation set data until the model's loss value and the number of iterations reached the set thresholds.

[0073] When building the network model, a loss value and the number of iterations are given. When the fluctuation of the model loss is less than a fluctuation threshold, the network is considered to have converged and stops learning; or when the model reaches the set number of iterations threshold, the network stops learning. The network training effect is then observed, and parameters are adjusted to achieve better convergence.

[0074] S8. Accuracy calculation;

[0075] After the model training is completed, the accuracy, recall and mAP values ​​of each category in the test set are calculated, and the average value of each category is used to obtain the final mAP value of the model.

[0076] The nighttime traffic sign recognition method in this embodiment of the invention first annotates traffic sign images in a nighttime environment, labeling ten common traffic signs in the images respectively; then, a network model is established, and an improved image preprocessing method is used to enhance image brightness and contrast; next, the backbone network extracts features from the image, adding an attention module and a feature enhancement module, and using a Neck network for feature fusion. After training, it can be used to obtain a recognition model for various types of traffic signs, providing an effective image processing method for driving assistance systems, with significant beneficial effects and suitable for application and promotion.

[0077] Currently, common image processing methods include HE and AHE algorithms. Histogram equalization (HE) adjusts image contrast, but this can lead to loss of detail. Adaptive histogram equalization (AHE) divides the image evenly, causing excessive local contrast and distortion. Limiting contrast histogram equalization (CLAHE) effectively improves image contrast, but its interpolation increases inference time, making it unsuitable for the end-to-end deep learning model of this invention. This invention combines the advantages of AHE and CLAHE algorithms, proposing an improved adaptive histogram equalization algorithm (IAHE) that improves contrast, suppresses noise, and effectively controls computation without increasing its complexity.

[0078] Preferably, in another embodiment, an improved adaptive histogram equalization (IAHE) algorithm is used for image preprocessing, which ensures improved contrast and noise suppression while effectively controlling the computational load. Figure 2 As shown, IAHE specifically includes the following steps:

[0079] S31. Divide the input image into n×n square blocks. The larger the value of n, the better the image enhancement effect, but the less detail is preserved. Simultaneously calculate the histogram of each block.

[0080] S32. Solve for the restricted value T.

[0081]

[0082] In the formula: c represents the constraint coefficient; n x Represented as the number of pixels in the x-direction of the small block; n y It represents the number of pixels in the y-direction of the small block; K is the gray level.

[0083] S33. Redistribute pixels for each small block. The histogram of a small block is denoted by h(x), and its value is constrained by T. When h(x) exceeds the constraint value represented by T, the excess portion is cropped and evenly distributed to other gray levels. If S pixels exceed the constraint value, the average distribution value for the small block is represented by A, as shown in the following formula:

[0084]

[0085]

[0086] After the pixels are redistributed, the histogram is represented by h′(x):

[0087]

[0088] As shown in equation (4), the constraint value T can limit the range of the average brightness, and selecting an appropriate T value is crucial for image preprocessing. The IAHE algorithm can preserve image details, suppress noise, and improve the brightness and contrast of the original image.

[0089] To facilitate understanding, the following provides a detailed explanation of the backbone network feature extraction in step S4, such as... Figure 3 As shown, it illustrates a schematic diagram of the backbone network in an embodiment of the present invention. The backbone network feature extraction specifically includes the following steps:

[0090] S41. After the input image is preprocessed by S3, an M-dimensional feature map is generated. The simultaneous convolution of the spatial and channel dimensions is changed to a step-by-step process of depthwise convolution and pointwise convolution.

[0091] S42. Channel-wise convolution. The input feature map is calculated using convolution kernels. Since the input image has M channels, M convolution kernels are used to operate on each of its channels, and the results are stacked to obtain an M-channel feature map. The computational complexity is shown in equation (5).

[0092] C d =H×W×M×K 2 (5)

[0093] Among them, C d The symbol represents the computational cost of channel-wise convolution; H and W represent the height and width of the feature map; K×K represents the height and width of the convolution.

[0094] S43. Pointwise convolution. The results obtained in the previous step are fused using pointwise convolution. The N 1×1M convolution kernels are weighted and combined with the previous output to generate a new N-dimensional feature map. The computational complexity is shown in Equation (6):

[0095] C p =H′×W′×M×N(6)

[0096] C2 = C d +C p (7)

[0097]

[0098] Where H′ and W′ represent the height and width calculated channel by channel; C p C1 represents the computational cost of channel-wise convolution; C2 represents the total computational cost of depthwise separable convolution. If the dimensions remain unchanged before and after convolution, then H′×W′=H×W, and equation (9) can be approximately expressed as:

[0099]

[0100] The model parameters and computational cost of depthwise separable convolution can be approximately reduced to 1 / K of the original standard convolution. 2 This can significantly improve the inference speed of the model.

[0101] This invention uses depthwise separable convolutions instead of traditional standard convolutions in the YOLOv5 backbone network module. This significantly reduces the number of model parameters, thereby improving speed, without sacrificing almost any model accuracy. Furthermore, the activation function of the convolution module is improved, which not only reduces computational load but also enhances computational efficiency in quantized mode. This makes the model more suitable for embedded and low-power environments.

[0102] Preferably, in another embodiment, to address the feature loss problem during model learning, the present invention also proposes an improved feature pyramid AS-PAN structure, incorporating an adaptive attention module and a spatial pyramid pooling module. For example... Figure 4 As shown, this illustrates the improvement to the Neck section in an embodiment of the present invention. The improved feature pyramid AS-PAN structure incorporates an adaptive attention module and a spatial pyramid pooling module. The specific improvements are as follows:

[0103] An Attention Adaptation Module (AAM) is added to the very front of the Neck section to direct limited attention more towards traffic sign targets;

[0104] The Spatial Pyramid Pooling-Fast (SPPF) module is added before the outputs of P3 and P4 to reduce feature loss and enhance feature representation.

[0105] Preferably, in another embodiment, such as Figure 5 As shown, this illustrates the fusion prediction block diagram in an embodiment of the present invention. The method of generating the optimal prediction box by changing the selection of redundant prediction boxes to discarding them reduces the false detection and false negative rates of the image. This embodiment employs the Detection Boxes Fusion (DBF) algorithm based on confidence weights. By changing the selection of redundant prediction boxes to discarding them, the optimal prediction box is generated, reducing the false detection and false negative rates of the image. The steps of the DBF algorithm are as follows:

[0106] S61. Add all the current boxes to a list named List B, and sort them in descending order of confidence.

[0107] S62. Initialize lists C, L and F. Predicted boxes and their corresponding boxes are placed in lists B and C respectively. Each element in list L is a collection of boxes. List F contains the boxes that are merged from L.

[0108] S63. Start iterating, traverse B, and match boxes in list F whose IoU value is greater than the threshold (F is initially empty);

[0109] S64. If no match is found, add the box to the end of both list L and list F, and then process the next box. If the i-th element in F is matched, place the box in list L at the position corresponding to the element in list F.

[0110] S65. Merge the accumulated bounding boxes at each position in list L, and recalculate the box coordinates. The merging formula is as follows:

[0111]

[0112]

[0113]

[0114] S66. After traversing list B, update the confidence scores in list F, as shown in formula (14):

[0115]

[0116] Where T is the number of detection boxes, C is the confidence score of the fused box, Ci is the confidence score of each detection box in list L, and Xa i 、Xb i Ya i Yb i Here are the coordinates of the fused bounding box in L after the update, and N is the number of models.

[0117] Traditional methods that use non-maximum suppression (NMS) and soft NMS to filter predicted bounding boxes remove redundant boxes, which can lead to selectively discarding targets. In contrast, this invention uses all predicted bounding boxes, effectively reducing false positives and improving the model's detection accuracy.

[0118] Through relevant improvements, this invention effectively alleviates the problems of false detection and missed detection of traffic signs by the model in nighttime environments, improves the model's detection accuracy, and ensures detection speed.

[0119] Corresponding to the deep learning-based nighttime traffic sign recognition method in this invention, this invention also provides a deep learning-based nighttime traffic sign recognition system, comprising:

[0120] The dataset acquisition module is used to acquire a dataset of nighttime traffic signs;

[0121] The image preprocessing module is used to process the nighttime images obtained by the dataset acquisition module into images with higher visibility.

[0122] The model building module is used to build a nighttime traffic sign recognition model based on the YOLOv5 network; the nighttime traffic sign recognition model includes: a backbone network module, an attention network module, a Neck network module, and a fusion prediction box module; for example... Figure 6 As shown, the image preprocessed by the image preprocessing module serves as the input to the backbone network module. The backbone network module uses depthwise separable convolution to perform convolution operations on each input image. Throughout the process, the feature maps obtained from the convolution are divided into three sizes: 20*20, 40*40, and 80*80. The obtained feature maps are then fed into the attention module, which globally learns the feature maps. The activation function of the convolution module is improved to enhance computational efficiency. The feature pyramid in the Neck network module is improved by adding an adaptive attention module and a spatial pyramid pooling module. Finally, the Neck network module fuses the three feature maps, and the fused result is fed into the detection head for recognition to determine the probability of each type of traffic sign.

[0123] The model training module is used to train the nighttime traffic sign recognition model using the nighttime traffic sign dataset;

[0124] The traffic sign recognition module is used to perform nighttime traffic sign recognition using the nighttime traffic sign recognition model trained by the model training module, and to obtain the nighttime traffic sign recognition result.

[0125] The description of the deep learning-based nighttime traffic sign recognition system in this embodiment is relatively simple because it corresponds to the deep learning-based nighttime traffic sign recognition method in the above embodiment. For related similarities, please refer to the description of the deep learning-based nighttime traffic sign recognition method in the above embodiment, which will not be described in detail here.

[0126] This invention also discloses a computer-readable storage medium storing a computer instruction set, which, when executed by a processor, implements the deep learning-based nighttime traffic sign recognition method provided in any of the above embodiments.

[0127] In the several embodiments provided by this invention, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection can be through some interfaces; the indirect coupling or communication connection of units or modules can be electrical or other forms.

[0128] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0130] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for nighttime traffic sign recognition based on deep learning, characterized in that, include: Obtain a dataset of nighttime traffic signs; Preprocess the image; A nighttime traffic sign recognition model was established based on the YOLOv5 network. The nighttime traffic sign recognition model includes: a backbone network for feature extraction, an attention and feature enhancement module for global feature map learning, a Neck network for feature fusion, and a detection head for filtering prediction boxes. The backbone network uses depthwise separable convolution to perform convolution operations on each input image. The obtained feature maps at multiple scales are fed into the attention and feature enhancement module, which globally learns the feature maps. The feature pyramid in the Neck network module is improved by adding an adaptive attention module and a spatial pyramid pooling module. Finally, the Neck network module fuses the feature maps at multiple scales, and the fused result is fed into the detection head for prediction box filtering to obtain the final prediction box. The Neck network module includes: an improved feature pyramid AS-PAN structure with an adaptive attention module and a spatial pyramid pooling module. Specifically, an attention adaptation module (AAM) is added at the front end of the Neck part to focus the limited attention more on the traffic sign target. A spatial pyramid pooling-fast SPPF module is added before the outputs of P3 and P4 to reduce feature loss and enhance feature representation. The nighttime traffic sign recognition model is trained using the nighttime traffic sign dataset. The trained nighttime traffic sign recognition model is used to recognize nighttime traffic signs, and the nighttime traffic sign recognition results are obtained.

2. The nighttime traffic sign recognition method based on deep learning according to claim 1, characterized in that, Image preprocessing includes: performing image preprocessing using an improved adaptive histogram equalization algorithm; wherein: The improved adaptive histogram equalization algorithm includes: Divide the input image into several square blocks and calculate the histogram of each block. Solve for the constraint values ​​of the constrained small block histogram; For each small block, pixels are reassigned. When the histogram of a small block exceeds the limit, the excess part is cropped and evenly distributed to other gray levels.

3. The nighttime traffic sign recognition method based on deep learning according to claim 1, characterized in that, The backbone network performs feature extraction, including: Generate from the preprocessed image For 3D feature images, the simultaneous convolution of both spatial and channel dimensions is changed to a step-by-step process of channel-wise convolution and point-wise convolution; where: The channel-wise convolution includes: calculating the input feature image using convolution kernels, where the input image has... Channel, use Each convolutional kernel is operated on individually with each of its channels, and the results are stacked to obtain... Feature maps of each channel; The pointwise convolution includes: performing channel fusion on the results obtained from the channel-wise convolution using pointwise convolution, and... indivual The convolution kernel is weighted and combined with the output of the channel-wise convolution to finally generate A new feature map of the dimension.

4. The nighttime traffic sign recognition method based on deep learning according to claim 1, characterized in that, Predicted bounding box filtering includes: using the detection bounding box fusion algorithm DBF to generate the optimal predicted bounding box by changing the selection of redundant predicted bounding boxes to the method of discarding them; The detection bounding box fusion algorithm DBF includes: Add all the current predicted boxes to a list named List B, and sort them in descending order of confidence. Initialize lists C, L, and F. Predicted boxes and their corresponding values ​​are placed in lists B and C respectively. Each element in list L is a collection of boxes. List F contains the boxes that are merged from the L boxes. Start iterating through B, matching boxes in list F whose IoU value is greater than the threshold; F is initially empty. If no match is found, the box is added to the end of both list L and list F, and then the next box is processed; if the i-th element in F is matched, the box is placed in list L at the position corresponding to that in list F. Merge the accumulated bounding boxes at each position in list L, and recalculate the box coordinates. The merging formula is as follows: ； , ； , ； After traversing list B, update the confidence score in list F. The formula for the confidence score is: ； In the formula, T represents the number of detection boxes, C represents the confidence level of the fused box, and Ci represents the confidence level of each detection box in list L. , , , The coordinates of the fusion box in the updated list L are given, and N is the number of models.

5. The nighttime traffic sign recognition method based on deep learning according to claim 1, characterized in that, Obtain the nighttime traffic sign dataset, including: Images can be captured using image acquisition equipment in nighttime environments, images can be extracted from dashcams, and / or images of nighttime traffic signs can be crawled from the web.

6. The nighttime traffic sign recognition method based on deep learning according to claim 1, characterized in that, Before image preprocessing, the process also includes: selecting a preset number of traffic signs as annotation objects, and labeling the acquired nighttime images with the corresponding annotation object types.

7. A nighttime traffic sign recognition system based on deep learning, characterized in that, include: The dataset acquisition module is used to acquire a dataset of nighttime traffic signs; The image preprocessing module is used to process the nighttime images obtained by the dataset acquisition module into images with higher visibility. The model building module is used to build a nighttime traffic sign recognition model based on the YOLOv5 network. The nighttime traffic sign recognition model includes: a backbone network for feature extraction, an attention and feature enhancement module for global feature map learning, a Neck network for feature fusion, and a detection head for filtering prediction boxes. The backbone network uses depthwise separable convolution to perform convolution operations on each input image. The obtained feature maps at multiple scales are fed into the attention and feature enhancement module, which globally learns the feature maps. The feature pyramid in the Neck network module is improved by adding an adaptive attention module and a spatial pyramid pooling module. Finally, the Neck network module fuses the feature maps at multiple scales, and the fused result is fed into the detection head for prediction box filtering to obtain the final prediction box. The Neck network module includes: an improved feature pyramid AS-PAN structure with an adaptive attention module and a spatial pyramid pooling module. Specifically, an attention adaptation module (AAM) is added at the front end of the Neck part to focus the limited attention more on the traffic sign target. A spatial pyramid pooling-fast SPPF module is added before the outputs of P3 and P4 to reduce feature loss and enhance feature representation. The model training module is used to train the nighttime traffic sign recognition model using the nighttime traffic sign dataset; The traffic sign recognition module is used to perform nighttime traffic sign recognition using the nighttime traffic sign recognition model trained by the model training module, and to obtain the nighttime traffic sign recognition result.

8. A computer-readable storage medium storing a computer instruction set, which, when executed by a processor, implements the deep learning-based nighttime traffic sign recognition method as described in any one of claims 1 to 6.

Images