Traffic sign recognition method, traffic sign model training method and device

Abstract

The application provides a traffic sign recognition method and a traffic sign model training method and device. The method comprises: acquiring a traffic image to be recognized; determining a model detection result according to the traffic image to be recognized and a pre-trained traffic sign model; determining geometric information features corresponding to the model detection result according to the model detection result; and determining a target detection result of the traffic image to be recognized according to the geometric information features and preset geometric features. The traffic sign model is trained in the embodiment of the application. The traffic sign model increases a slice downsampling unit and an interactive positioning module, can make up for the shortcoming that small target semantic information is prone to losing information in the process of neural network learning, effectively solves the problems of missing detection and false detection of small target objects, and improves the detection efficiency.

Description

Technical Field

[0001] This application relates to the field of detection technology, and more specifically, to a traffic sign recognition method, a training method for a traffic sign model, and an apparatus. Background Technology

[0002] With economic growth, the total number of cars is increasing year by year, leading to more severe traffic congestion and frequent traffic accidents. Road traffic safety and transportation efficiency have become increasingly prominent issues. Autonomous driving, as a new research field, aims to optimize a series of traffic problems caused by automobiles. Traffic sign detection is a crucial component of autonomous driving systems. Traffic signs, as important road safety facilities, play a vital role in regulating traffic behavior, indicating road conditions, ensuring road efficiency, guiding pedestrians, and ensuring safe driving.

[0003] In existing technologies, label detection is performed using machine learning. Traditional machine learning features rely on manual design, which requires researchers to have considerable experience, thus presenting certain limitations. In contrast, deep learning-based label detection avoids this limitation by employing a black-box learning method, automatically learning through model training.

[0004] However, the detection of small targets is not ideal. Good network performance generally requires increasing the number of neural network layers. However, the features of small targets are more easily lost as the network becomes deeper. How to improve the detection efficiency of small targets is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] The purpose of some embodiments of this application is to provide a traffic sign recognition method, a traffic sign model training method and apparatus. Through the technical solutions of the embodiments of this application, a traffic sign model is pre-trained. The traffic sign model is obtained by improving and training a YOLOv5 network model. The traffic sign model includes at least an encoding module, an interactive localization module, and an output module. The encoding module includes a slice downsampling unit, which is used to add refined semantic features to the downsampled semantic features. The interactive localization module is used to determine the semantic communication information at different stages using deep semantic features and shallow semantic features, and to determine the global decoding features and optimized solution based on high-level and low-level semantic information. The output module is used to calculate classification loss and regression loss; acquire the traffic image to be identified; determine the model detection result based on the traffic image to be identified and the pre-trained traffic sign model; determine the geometric information features corresponding to the model detection result based on the model detection result; determine the target detection result of the traffic image to be identified based on the geometric information features and the preset geometric features. In this embodiment, by training a traffic sign model, which adds a slice downsampling unit and an interactive localization module, the shortcomings of small target semantic information being easily lost during neural network learning can be made up for, effectively solving the problem of missed detection and false detection of small target objects, and improving detection efficiency.

[0006] Firstly, some embodiments of this application provide a traffic sign recognition method, including:

[0007] Acquire the traffic image to be identified;

[0008] Based on the traffic image to be identified and the pre-trained traffic sign model, the model detection result is determined; wherein, the pre-trained traffic sign model is obtained by improving and training the YOLOv5 network model;

[0009] The traffic sign model includes at least an encoding module, an interactive positioning module, and an output module.

[0010] The encoding module includes a slice downsampling unit, which is used to add refined semantic features to the downsampled semantic features.

[0011] The interactive positioning module is used to determine semantic communication information at different stages by using deep semantic features and shallow semantic features, and to determine global decoding features and optimized decoding features based on high-level semantic information and low-level semantic information.

[0012] The output module is used to calculate classification loss and regression loss.

[0013] Based on the model detection results, determine the geometric information features corresponding to the model detection results;

[0014] Based on the geometric information features and preset geometric features, the target detection result of the traffic image to be identified is determined.

[0015] Some embodiments of this application add a slice downsampling unit and an interactive positioning module to the traffic sign model. This compensates for the shortcoming that semantic information of small targets is easily lost during the neural network learning process, effectively solving the problem of missed detection and false detection of small target objects. Furthermore, the network detection results are subjected to geometric information feature extraction and comparison to remove false positives and obtain the final detection results, thereby improving the detection efficiency.

[0016] In some embodiments, the pre-trained traffic sign model is obtained in the following manner:

[0017] Obtain a dataset of sample images of traffic signs;

[0018] Based on the sample image dataset, determine the training set, test set, and validation set corresponding to the sample image dataset;

[0019] The receptive fields corresponding to different hole rates of feature maps in the sample identification images in the training set are obtained, and the receptive fields are filtered and fused using attention mechanism and self-attention mechanism to obtain feature information of different receptive fields, wherein the feature information includes at least refined semantic features.

[0020] By employing deep semantic features and shallow semantic features, semantic communication information at different stages is determined, and an attention mechanism is used to weight the deep semantic information layer by layer with the low-level semantic information to determine the global decoding features and the optimized decoding features.

[0021] Based on the global decoding features and the optimized decoding features, loss information is obtained by calculating classification loss and regression loss.

[0022] If the loss information satisfies the preset convergence condition, the resulting improved YOLOv5 network model is determined as the traffic sign model.

[0023] Some embodiments of this application add a slice downsampling unit and an interactive localization module to the traffic sign model, and use attention and self-attention mechanisms to filter and fuse multi-scale information. Adaptively learn the discrimination features of small target objects under multiple receptive fields and supplement them into the max pooling downsampling operation, thereby enhancing and enriching the detailed features of small target objects. The Concat module is replaced with an interactive localization module, which is used to enhance the semantic communication problem at different stages, and obtain global guidance and detailed optimized decoding features after information interaction between adjacent layers.

[0024] In some embodiments, determining the training set, test set, and validation set corresponding to the sample image dataset based on the sample image dataset includes:

[0025] The various traffic signs in the sample image dataset are statistically analyzed to obtain statistically analyzed sample sign images.

[0026] The statistically analyzed sample recognition images are processed using data augmentation methods to obtain a processed sample identification image set. The data augmentation methods include at least one or more of rotation, cropping, or scaling.

[0027] Based on the sample image set and the preset allocation ratio, determine the training set, test set, and validation set corresponding to the sample image dataset;

[0028] The sample identification images in the training set, test set, and validation set are selected and classified respectively to obtain the ground truth values ​​corresponding to the training set, test set, and validation set. The ground truth values ​​are used for neural network training and testing.

[0029] Some embodiments of this application improve the accuracy of model training by acquiring a large amount of sample image data and processing the sample image data to obtain inputs that meet the requirements of neural networks.

[0030] In some embodiments, the receptive fields corresponding to different hole rates of feature maps in the sample identifier images of the training set are obtained, and attention mechanisms and self-attention mechanisms are used to filter and fuse the receptive fields to obtain feature information of different receptive fields. The feature information includes at least refined semantic features, including:

[0031] According to the formula Obtain four slices with different receptive fields;

[0032] Where, f(X) i ) represents the feature of the i-th stage of the coding layer; DC i (·) indicates a dilated convolution operation, and 0, 1, 2, and 3 represent the dilation rate; These are the four slices of the i-th stage of the coding layer;

[0033] According to the formula and We obtain the local and global features for a large receptive field, and the local and global features for a small receptive field, respectively; where f′1 is the local feature and f′2 is the global feature; and concat(·) represents the tensor concatenation operation; f 1×1 (·) represents a 1×1 convolution operation;

[0034] Using formula Attention and self-attention learning are performed on the local features and the global features respectively to obtain the learned local features and global features; where R(·) represents the dimension reshape operation, transforming C×H×W into C×HW; DR(·) represents the dimension reshape operation, transforming C×HW into C×H×W; S(·) is the non-linear action function Sigmoid function, which normalizes the feature values ​​to the range [0,1] to obtain the attention mask; SA represents the spatial attention module; This represents matrix multiplication, used for weighted features;

[0035] Using formula The local features and the global features are refined to obtain the refined semantic features, where DSAT(·) represents dual spatial attention; CA represents channel attention module; and MaxPool(·) represents max pooling operation.

[0036] Using formula The refined semantic features are supplemented into the downsampled semantic features through matrix addition to obtain feature information of different receptive fields; where MaxPool(·) represents the max pooling operation. This indicates that the elements at the corresponding positions are added together.

[0037] In some embodiments of this application, a slice downsampling unit is added to the encoding stage of the traffic sign model. Attention and self-attention mechanisms are used to filter and fuse multi-scale information, adaptively learn the discrimination features of small target objects under multiple receptive fields and supplement them into the max pooling downsampling operation, thereby enhancing and enriching the detailed features of small target objects.

[0038] In some embodiments, the method employs deep semantic features and shallow semantic features to determine semantic communication information at different stages, and uses an attention mechanism to weight the deep semantic information layer by layer with the low-level semantic information to determine the global decoding features and the optimized decoding features, including:

[0039] Using formula and Feature extraction is performed from local deep networks and local shallow networks respectively to obtain shallow semantic features and deep semantic features;

[0040] Using formula Hybrid feature extraction is performed on the shallow semantic features and the deep semantic features to obtain high-level semantic information and low-level semantic information; wherein, Indicates shallow network characteristics; Indicates the characteristics of deep networks; This indicates that the elements at corresponding positions are added together; MCA (·) represents the nonlinear action function;

[0041] Using formula The high-level semantic information is weighted layer by layer into the low-level semantic information to obtain the global decoding feature and the optimized decoding feature.

[0042] In some embodiments of this application, the Concat module is replaced with an interactive positioning module. The interactive positioning module is used to enhance the semantic communication problem at different stages and obtain global guidance and detailed optimization decoding features after information interaction between adjacent layers.

[0043] In some embodiments, the step of calculating loss information based on the global decoding features and the optimized decoding features using classification loss and regression loss includes:

[0044] The center and width / height of the bounding box are corrected using regression loss to obtain the first loss information;

[0045] For the target category and confidence level, the improved YOLOv5 network model is trained using classification loss, and confidence loss is calculated for negative samples to obtain the second loss information;

[0046] The third loss information of the improved YOLOv5 network model is calculated using the adaptive gradient method.

[0047] One or more of the first loss information, the second loss information, and the third loss information are determined as the loss information.

[0048] Some embodiments of this application determine whether to stop model training by judging loss information, thereby improving the efficiency of model training.

[0049] In some embodiments, determining the geometric information features corresponding to the model detection results based on the model detection results includes:

[0050] The Canny operator is used to obtain the boundary information of the marker image in the model detection result;

[0051] The boundary information of the marker image is subjected to Hough transform to obtain geometric information features corresponding to the detection results of the model;

[0052] The step of determining the target detection result of the traffic image to be identified based on the geometric information features and preset geometric features includes:

[0053] The geometric information features corresponding to the detection results of the model are compared with the preset geometric features of the standard image;

[0054] If the difference between the geometric information feature and the preset geometric feature is less than a preset range, the image corresponding to the geometric information feature is determined as the target detection result.

[0055] Some embodiments of this application improve the accuracy of detection results by extracting and comparing geometric information features from the model detection results, removing false positives, and obtaining the final detection results.

[0056] Secondly, some embodiments of this application provide a method for training a traffic sign model, the method comprising:

[0057] Obtain a dataset of sample images of traffic signs;

[0058] Based on the sample image dataset, determine the training set, test set, and validation set corresponding to the sample image dataset;

[0059] The receptive fields corresponding to different hole rates of feature maps in the sample identification images in the training set are obtained, and the receptive fields are filtered and fused using attention mechanism and self-attention mechanism to obtain feature information of different receptive fields, wherein the feature information includes at least refined semantic features.

[0060] By employing deep semantic features and shallow semantic features, semantic communication information at different stages is determined, and an attention mechanism is used to weight the deep semantic information layer by layer with the low-level semantic information to determine the global decoding features and the optimized decoding features.

[0061] Based on the global decoding features and the optimized decoding features, loss information is obtained by calculating classification loss and regression loss.

[0062] If the loss information satisfies the preset convergence condition, the resulting improved YOLOv5 network model is determined as the traffic sign model.

[0063] Some embodiments of this application add a slice downsampling unit and an interactive localization module to the traffic sign model, and use attention and self-attention mechanisms to filter and fuse multi-scale information. Adaptively learn the discrimination features of small target objects under multiple receptive fields and supplement them into the max pooling downsampling operation, thereby enhancing and enriching the detailed features of small target objects. The Concat module is replaced with an interactive localization module, which is used to enhance the semantic communication problem at different stages, and obtain global guidance and detailed optimized decoding features after information interaction between adjacent layers.

[0064] Thirdly, some embodiments of this application provide a traffic sign recognition device, including:

[0065] The acquisition module is used to acquire traffic images to be identified;

[0066] A determination module is used to determine the model detection result based on the traffic image to be identified and a pre-trained traffic sign model. The pre-trained traffic sign model is obtained by improving and training a YOLOv5 network model. The traffic sign model includes at least an encoding module, an interaction localization module, and an output module. The encoding module includes a slice downsampling unit, which adds refined semantic features to the downsampled semantic features. The interaction localization module uses deep and shallow semantic features to determine semantic communication information at different stages, and determines global decoding features and optimized decoding features based on high-level and low-level semantic information. The output module is used to calculate classification loss and regression loss.

[0067] The detection module is used to determine the geometric information features corresponding to the model detection results based on the model detection results;

[0068] The comparison module is used to determine the target detection result of the traffic image to be identified based on the geometric information features and preset geometric features.

[0069] Optionally, the apparatus further includes a model building module, the model building module being used for:

[0070] Obtain a dataset of sample images of traffic signs;

[0071] Based on the sample image dataset, determine the training set, test set, and validation set corresponding to the sample image dataset;

[0072] The receptive fields corresponding to different hole rates of feature maps in the sample identification images in the training set are obtained, and the receptive fields are filtered and fused using attention mechanism and self-attention mechanism to obtain feature information of different receptive fields, wherein the feature information includes at least refined semantic features.

[0073] By employing deep semantic features and shallow semantic features, semantic communication information at different stages is determined, and an attention mechanism is used to weight the deep semantic information layer by layer with the low-level semantic information to determine the global decoding features and the optimized decoding features.

[0074] Based on the global decoding features and the optimized decoding features, loss information is obtained by calculating classification loss and regression loss.

[0075] If the loss information satisfies the preset convergence condition, the resulting improved YOLOv5 network model is determined as the traffic sign model.

[0076] Optionally, the model building module is used for:

[0077] The various traffic signs in the sample image dataset are statistically analyzed to obtain statistically analyzed sample sign images.

[0078] The statistically analyzed sample recognition images are processed using data augmentation methods to obtain a processed sample identification image set. The data augmentation methods include at least one or more of rotation, cropping, or scaling.

[0079] Based on the sample image set and the preset allocation ratio, determine the training set, test set, and validation set corresponding to the sample image dataset;

[0080] The sample identification images in the training set, test set, and validation set are selected and classified respectively to obtain the ground truth values ​​corresponding to the training set, test set, and validation set. The ground truth values ​​are used for neural network training and testing.

[0081] Optionally, the model building module is used for:

[0082] According to the formula Obtain four slices with different receptive fields;

[0083] Where, f(X) i ) represents the feature of the i-th stage of the coding layer; DC i (·) indicates a dilated convolution operation, and 0, 1, 2, and 3 represent the dilation rate; These are the four slices of the i-th stage of the coding layer;

[0084] According to the formula and We obtain the local and global features for a large receptive field, and the local and global features for a small receptive field, respectively; where f′1 is the local feature and f′2 is the global feature; and concat(·) represents the tensor concatenation operation; f 1×1 (·) represents a 1×1 convolution operation;

[0085] Using formula

[0086] Attention and self-attention learning are performed on the local features and the global features respectively to obtain the learned local features and global features; where R(·) represents the dimension reshape operation, transforming C×H×W into C×HW; DR(·) represents the dimension reshape operation, transforming C×HW into C×H×W; S(·) is the non-linear action function Sigmoid function, which normalizes the feature values ​​to the range [0,1] to obtain the attention mask; SA represents the spatial attention module; This represents matrix multiplication, used for weighted features;

[0087] Using formula The local features and the global features are refined to obtain the refined semantic features, where DSAT(·) represents dual spatial attention; CA represents channel attention module; and MaxPool(·) represents max pooling operation.

[0088] Using formula The refined semantic features are supplemented into the downsampled semantic features through matrix addition to obtain feature information of different receptive fields; where MaxPool(·) represents the max pooling operation. This indicates that the elements at the corresponding positions are added together.

[0089] Optionally, the model building module is used for:

[0090] Using formula and Feature extraction is performed from local deep networks and local shallow networks respectively to obtain shallow semantic features and deep semantic features;

[0091] Using formula Hybrid feature extraction is performed on the shallow semantic features and the deep semantic features to obtain high-level semantic information and low-level semantic information; wherein, Indicates shallow network characteristics; Indicates the characteristics of deep networks; This indicates that the elements at corresponding positions are added together; M CA (·) represents the nonlinear action function;

[0092] Using formula The high-level semantic information is weighted layer by layer into the low-level semantic information to obtain the global decoding feature and the optimized decoding feature.

[0093] Optionally, the model building module is used for:

[0094] The center and width / height of the bounding box are corrected using regression loss to obtain the first loss information;

[0095] For the target category and confidence level, the improved YOLOv5 network model is trained using classification loss, and confidence loss is calculated for negative samples to obtain the second loss information;

[0096] The third loss information of the improved YOLOv5 network model is calculated using the adaptive gradient method.

[0097] One or more of the first loss information, the second loss information, and the third loss information are determined as the loss information.

[0098] Optionally, the comparison module is used to:

[0099] The Canny operator is used to obtain the boundary information of the marker image in the model detection result;

[0100] The boundary information of the marker image is subjected to Hough transform to obtain geometric information features corresponding to the detection results of the model;

[0101] The step of determining the target detection result of the traffic image to be identified based on the geometric information features and preset geometric features includes:

[0102] The geometric information features corresponding to the detection results of the model are compared with the preset geometric features of the standard image;

[0103] If the difference between the geometric information feature and the preset geometric feature is less than a preset range, the image corresponding to the geometric information feature is determined as the target detection result.

[0104] Fourthly, some embodiments of this application provide a training apparatus for a traffic sign model, including:

[0105] The sample acquisition module is used to acquire a sample image dataset of traffic signs;

[0106] The sample set processing module is used to determine the training set, test set, and validation set corresponding to the sample image dataset based on the sample image dataset.

[0107] The encoding module is used to obtain the receptive fields corresponding to different hole rates of feature maps in the sample identification images of the training set, and to filter and fuse the receptive fields using attention mechanism and self-attention mechanism to obtain feature information of different receptive fields, wherein the feature information includes at least refined semantic features.

[0108] The interactive positioning module is used to determine semantic communication information at different stages by using deep semantic features and shallow semantic features, and to use an attention mechanism to weight the deep semantic information layer by layer with the low-level semantic information to determine the global decoding features and the optimized decoding features.

[0109] The output module is used to calculate loss information based on the global decoding features and the optimized decoding features, using classification loss and regression loss.

[0110] The judgment module is used to determine the improved YOLOv5 network model as the traffic sign model if the loss information meets the preset convergence condition.

[0111] Fifthly, some embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it can implement the traffic sign recognition method as described in any embodiment of the first aspect or the traffic sign model training method as described in the second aspect.

[0112] Sixthly, some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the traffic sign recognition method as described in any embodiment of the first aspect or the traffic sign model training method as described in the second aspect.

[0113] In a seventh aspect, some embodiments of this application provide a computer program product, the computer program product including a computer program, wherein when the computer program is executed by a processor, it can implement the traffic sign recognition method as described in any embodiment of the first aspect or the traffic sign model training method as described in the second aspect. Attached Figure Description

[0114] To more clearly illustrate the technical solutions of some embodiments of this application, the accompanying drawings used in some embodiments of this application will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0115] Figure 1 A flowchart illustrating a traffic sign recognition method provided in an embodiment of this application;

[0116] Figure 2 A schematic diagram illustrating the model training process provided in this application embodiment;

[0117] Figure 3 A schematic diagram illustrating model training provided in an embodiment of this application;

[0118] Figure 4 A schematic diagram of a slice downsampling unit provided in an embodiment of this application;

[0119] Figure 5 A schematic diagram of the interactive positioning module provided in an embodiment of this application;

[0120] Figure 6 This is a schematic diagram of the structure of a traffic sign recognition device provided in an embodiment of this application;

[0121] Figure 7 A schematic diagram of the structure of a training device for a traffic sign model provided in an embodiment of this application;

[0122] Figure 8 A schematic diagram of an electronic device provided in an embodiment of this application;

[0123] The attached figures are labeled as follows:

[0124] 1—Slice downsampling module; 2—Upsampling; 3—Interactive positioning module;

[0125] 4—Void rate -3; 5—Void rate -2; 6—Void rate -1; 7—Void rate -0;

[0126] 8—Feature fusion; 9—Convolution; 10—Size adjustment. Detailed Implementation

[0127] The technical solutions of some embodiments of this application will now be described with reference to the accompanying drawings.

[0128] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0129] With economic growth, the total number of cars in China has increased year by year, leading to increased traffic congestion, frequent traffic accidents, and increasingly prominent issues related to road traffic safety and transportation efficiency. Autonomous driving, as a new research field, aims to optimize a series of traffic problems caused by automobiles. Research on autonomous driving is complex and long-term. Traffic sign detection is a crucial component of autonomous driving systems, and a good traffic sign detection system largely determines the safety of autonomous vehicles. Traffic signs, as important road safety facilities, play a vital role in regulating traffic behavior, indicating road conditions, ensuring road efficiency, guiding pedestrians, and ensuring safe driving. To ensure the timely and accurate transmission of traffic sign information, Automatic Traffic Sign Recognition (TSR) systems have received significant attention from scholars worldwide. Traffic sign detection is a crucial branch of road information collection. However, due to long-term outdoor exposure, traffic signs themselves suffer from color fading, dirt, and deformation; during video and image acquisition, changes in lighting can cause color distortion, and tilted viewing angles can cause shape changes; in complex environments, traffic signs can be obscured by other objects, resulting in incomplete edges. All these factors pose challenges to traffic sign detection.

[0130] The current main research directions for traffic sign detection can be divided into three categories:

[0131] One approach is color-based sign detection. This algorithm typically uses color space to segment the image, and with minimal computation, it can remove a large number of non-interesting regions, thus offering good real-time performance. However, sign detection based solely on color information is significantly affected by lighting and weather conditions.

[0132] Second, sign detection based on sign shape can effectively extract traffic signs from complex backgrounds. Hough transform is usually used to detect circles and lines, which has a significant detection effect. However, its large computational load affects its feasibility, and the limited shape (such as triangle, rhombus, circle) restricts the detection.

[0133] Thirdly, there is learning-based sign detection. Traditional machine learning-based sign detection methods have the advantages of high detection accuracy and timely response to changes in road conditions. However, the features of traditional machine learning rely on manual design, which requires researchers to have considerable experience, and therefore has certain limitations. In contrast, deep learning-based sign detection breaks free from the limitation of relying on professionals and adopts a black-box learning method, learning automatically through model training.

[0134] Although there have been many studies and attempts on detection networks, and they have achieved good results in terms of accuracy and speed, they are not ideal for detecting small targets. Good network performance generally requires increasing the number of neural network layers, but the features of small targets are more easily lost as the network becomes deeper. How to improve the detection efficiency of small targets is a technical problem that urgently needs to be solved.

[0135] In view of this, some embodiments of this application provide a traffic sign recognition method. This method pre-trains a traffic sign model, which is obtained by improving and training a YOLOv5 network model. The traffic sign model includes at least an encoding module, an interaction localization module, and an output module. The encoding module includes a slice downsampling unit, which adds refined semantic features to the downsampled semantic features. The interaction localization module uses deep and shallow semantic features to determine semantic communication information at different stages, and determines global decoding features and optimized decoding features based on high-level and low-level semantic information. The output module is used for segmentation... The process involves calculating class loss and regression loss; acquiring the traffic image to be identified; determining the model detection result based on the traffic image to be identified and a pre-trained traffic sign model; determining the geometric information features corresponding to the model detection result based on the model detection result; and determining the target detection result of the traffic image to be identified based on the geometric information features and preset geometric features. This embodiment trains a traffic sign model, which adds a slice downsampling unit and an interactive localization module. This compensates for the weakness of small target semantic information being easily lost during neural network learning, effectively solving the problem of missed and false detections of small target objects and improving detection efficiency.

[0136] like Figure 1 As shown, an embodiment of this application provides a traffic sign recognition method, which includes:

[0137] S101. Obtain the traffic image to be identified;

[0138] Specifically, the terminal device acquires the traffic image to be identified, which may be a traffic image captured by a vehicle-mounted camera, and then sends it to the terminal device, which may be a computer, tablet computer, etc.

[0139] S102. Based on the traffic image to be identified and the pre-trained traffic sign model, determine the model detection result; wherein, the pre-trained traffic sign model is obtained by improving and training the YOLOv5 network model; the traffic sign model includes at least an encoding module, an interaction localization module, and an output module. The encoding module includes a slice downsampling unit, which is used to add refined semantic features to the downsampled semantic features. The interaction localization module is used to use deep semantic features and shallow semantic features to determine the semantic communication information at different stages, and to determine the global decoding features and optimized decoding features based on high-level semantic information and low-level semantic information; the output module is used to calculate the classification loss and regression loss.

[0140] Specifically, the terminal device pre-acquires a large image dataset to train an improved YOLOv5 network model. This YOLOv5 network model includes an encoding module, an interaction localization module, and an output module. A slice downsampling unit is added to the encoding module to refine semantic features in the downsampled semantic features. The interaction localization module uses deep and shallow semantic features to determine semantic communication information at different stages, and determines global decoding features and optimized decoding features based on high-level and low-level semantic information. The output module is used to calculate classification loss and regression loss. This traffic sign model is used to detect semantic information of small targets, overcoming the shortcoming that small target semantic information is easily lost during neural network learning.

[0141] The terminal device inputs the acquired traffic images into the pre-trained traffic recognition model, which outputs the model detection results, which include at least the outlines of each traffic sign.

[0142] S103. Based on the model detection results, determine the geometric information features corresponding to the model detection results;

[0143] Specifically, the terminal device performs image processing on the model detection results, such as using the Canny algorithm and Hough transform, to obtain the boundary information of each traffic sign in the model detection results, and obtains the corresponding geometric information features based on the boundary information.

[0144] S104. Based on the geometric information features and preset geometric features, determine the target detection result of the traffic image to be identified.

[0145] Specifically, after acquiring the geometric information features of each detected traffic sign, the terminal device compares the geometric information features with the preset geometric features of the standard image of the traffic sign. If the difference between the geometric information features and the preset geometric features is within the allowable range, the traffic sign is determined to be a target detection result. If the difference exceeds the allowable range, it indicates that the traffic sign is a false positive interference image.

[0146] This application provides a traffic sign recognition method. During the model training process, a slice downsampling unit and an interactive localization module are added in the encoding stage. These compensate for the shortcoming that semantic information of small targets is easily lost during neural network learning, and effectively solve the problems of missed detection and false detection of small target objects. Furthermore, the network detection results are subjected to geometric information feature extraction and comparison to remove false positives and obtain the final detection results.

[0147] Another embodiment of this application further supplements the description of the traffic sign recognition method provided in the above embodiments.

[0148] like Figure 2 The diagram shown is a schematic representation of the model training process provided in an embodiment of this application. The model training method includes:

[0149] Step S201: Obtain the images of highway traffic signs and divide them into training set, validation set and test set;

[0150] Specifically, it includes:

[0151] Step S2011: Obtain a sample image dataset of traffic signs;

[0152] This application embodiment uses a large number of highway traffic sign images, and cleans the acquired highway traffic sign image data, removing some images without highway traffic signs or blurry highway traffic sign images, to obtain a sample image dataset of the traffic sign.

[0153] Step S2012: Based on the sample image dataset, determine the training set, test set, and validation set corresponding to the sample image dataset, including:

[0154] Step A1: Perform statistical analysis on each type of traffic sign in the sample image dataset to obtain the statistically analyzed sample sign images;

[0155] Step A2: Process the statistically analyzed sample recognition images using data augmentation methods to obtain a processed sample identification image set. The data augmentation methods include at least one or more of rotation, cropping, or scaling.

[0156] Step A3: Determine the training set, test set, and validation set corresponding to the sample image dataset based on the sample identifier image set and the preset allocation ratio;

[0157] Step A4: Select and classify the sample label images in the training set, test set, and validation set respectively to obtain the ground truth values ​​corresponding to the training set, test set, and validation set. The ground truth values ​​are used for neural network training and testing.

[0158] Specifically, firstly, the acquired highway traffic sign images are statistically analyzed. For sign types with a small number of images, offline data augmentation methods are used, such as rotation, cropping, and scaling, to increase the number of images and mitigate the impact of imbalance on network performance. Secondly, for the data-augmented dataset, each type of highway traffic sign is placed into the training, validation, and test sets in a ratio of 0.8:0.1:0.1. Finally, for the divided dataset, the labeling tool LabelImg is used to select and classify all categories of highway signs to obtain the corresponding ground truth values ​​for subsequent neural network model training and testing.

[0159] SeeFigure 3 The CNN neural network model provided in this application embodiment is an improved YOLOv5 architecture, specifically:

[0160] At the input of the CNN neural network model, mean-variance standardization is performed on each sample (sample label image). The mean [0.485, 0.456, 0.406] and variance [0.229, 0.224, 0.225] calculated by sampling from the ImageNet dataset are used to perform mean-variance standardization, which is to subtract the mean from each channel and then divide by the variance.

[0161] Mosaic data augmentation is performed on the normalized data. First, the image size is obtained, and a center point is randomly generated within this size. Then, three images are randomly selected from the dataset, and the four images are placed at the top left, top right, bottom left, and bottom right corners of the center point, respectively. Any part exceeding the image size is cropped, while the part smaller than the image size is enlarged to the image size using padding. The labels of the four images are then changed accordingly and stitched together to obtain the labels after the Mosaic data augmentation.

[0162] Next, adaptive anchor box calculation is used. K-means clustering is performed on all samples to obtain n anchor boxes. The genetic algorithm randomly mutates the width and height of the anchor boxes, iterates 1000 times, and obtains the initial values ​​of n and the width and height of the anchor boxes.

[0163] Finally, the letterbox adaptive image scaling technology is used to ensure that both its width and height are divisible by the maximum downsampling resolution, so that its width and height are not rounded during the downsampling process.

[0164] This includes an encoding module, namely a stacked denoising autoencoder (SDA) S202; an improved YOLOv5 network model, which adds a slice downsampling unit 1 and an upsampling unit 2 to the encoding stage of the YOLOv5 network model; the Concat module in the Neck stage of the YOLOv5 network model is replaced with an interactive localization module 3; and the YOLOv5 Head output layer is decoupled into two branches, classification and regression, to calculate the loss separately, such as classification loss, localization loss, and intersection-union ratio loss.

[0165] Specifically, it includes:

[0166] Step S2021: Obtain the receptive fields corresponding to different hole rates of feature maps in the sample identification images in the training set, and use attention mechanism and self-attention mechanism to filter and fuse the receptive fields to obtain feature information of different receptive fields. The feature information includes at least refined semantic features.

[0167] like Figure 3 and Figure 4 As shown, an image preprocessing stage is set at the input of the CNN neural network model. The mean and variance are standardized for each sample. The normalized data is then subjected to Mosaic data augmentation, adaptive anchor box calculation, and adaptive image scaling to improve the training speed and accuracy of the network.

[0168] In the encoding part of the CNN neural network model, to mitigate the loss of information from small targets during downsampling, slice downsampling units are used instead of the downsampling process. This is achieved by applying dilated convolutions with different dilation rates to shallow features, with a stride of 2, for example... Figure 4 In the diagram, 4 represents a dilatation rate of 3, 5 represents a dilatation rate of 2, 6 represents a dilatation rate of 1, and 7 represents a dilatation rate of 0. Then, feature fusion is performed (8), followed by convolution (9), and the size is adjusted (10) to obtain different receptive fields. This changes the receptive field size of the output features, allowing each feature map to adaptively have different receptive fields, thus improving the recognition accuracy of multi-scale targets. Attention and self-attention mechanisms are used to filter and fuse multi-scale information, adaptively learning the discriminative features of small targets under multiple receptive fields and supplementing them into the max-pooling downsampling operation, thereby enhancing and enriching the detailed features of small targets.

[0169] The self-attention mechanism is mainly designed to capture more nuanced feature representations. It targets 0 and 1 with a smaller dilation rate in dilated convolution downsampling, in order to retain more detailed features from small samples.

[0170] Attention mechanisms are more about integrating information. For example, channel attention is applied to dilatational convolutions with dilatation rates of 3 and 4. Channel attention uses a shared multilayer perceptron for computation to obtain a larger receptive field. Furthermore, spatial attention is used to integrate the fine-grained (self-attention) and coarse-grained (channel convolution) features (the corresponding operations are B1, B2, B3, and B4).

[0171] Step S2022: Using deep semantic features and shallow semantic features, determine the semantic communication information at different stages, and use an attention mechanism to weight the deep semantic information with the low-level semantic information layer by layer to determine the global decoding features and optimized decoding features.

[0172] In the Neck network decoding part of the CNN neural network model, the Concat module is replaced with an interactive localization module. This fully utilizes the guiding role of deep semantic information on shallow semantic information and the supplementary role of shallow semantic information on deep semantic information to enhance semantic communication at different stages. At the same time, attention mechanism is used to give more attention to the target region, and high-level semantic information is weighted layer by layer into low-level feature information to generate globally guided and detail-optimized decoding features.

[0173] Step S2023: Based on the global decoding features and optimized decoding features, loss information is obtained by calculating classification loss and regression loss.

[0174] In the output prediction part of the CNN neural network model, the idea of ​​decoupling is adopted, and the detection head is decoupled into two branches: classification and regression. In each branch, additional 3×3 convolutional layers, ReLU function, and SiLU activation function are introduced to improve performance.

[0175] The preprocessed test set obtained in step S201 is input into the improved YOLOv5 structure CNN neural network model in step S202. The improved YOLOv5 structure CNN neural network model is trained and validated. When the number of iterations reaches the set value, the training process stops, and the improved YOLOv5 structure CNN neural network model with the best detection performance on the validation set is saved. The model is then used to detect the test set images, and the location information and category of the road traffic signs in each image are output, which is the coarse detection result of the road traffic sign images.

[0176] For example, each image outputs 5 values, i.e. 5 types, including prohibition signs, warning signs, instruction signs, guide signs, roundabout signs and other signs, with the corresponding value c∈(0-5). The location information of the road traffic signs in each image is output as 4 values, for example, the coordinates of the top left corner of the road traffic sign (i.e., traffic sign) in the coordinate system (X, Y), and the width and height of the frame (W, H), and finally output (C,X,Y,W,H).

[0177] S203. Using the training set, validation set, and test set, the improved YOLOv5 network model is evaluated to obtain the traffic sign model for highway signs.

[0178] If the loss information meets the preset convergence conditions, the resulting improved YOLOv5 network model will be determined as the traffic sign model.

[0179] The improved YOLOv5 neural network model is trained and validated using the processed training and validation sample sets obtained in step S201. Regression loss (IOU loss) is used to correct the center and width / height of the bounding boxes, while classification loss (binary cross-entropy loss) is used to learn the network for the target category and confidence level. Confidence loss is applied only to negative samples to accelerate convergence and improve network accuracy. For the total loss, the model parameters are optimized using an adaptive gradient method (AdamW optimizer) to continuously train the neural network model by minimizing the loss. The training process stops when the number of iterations reaches a set value, and the neural network model with the best detection performance on the validation sample set is saved. During training, step S2021 includes:

[0180] Step B1, according to the formula Obtain four slices with different receptive fields;

[0181] Where, f(X) i ) represents the feature of the i-th stage of the coding layer; DC i (·) indicates a dilated convolution operation, and 0, 1, 2, and 3 represent the dilation rate; These are the four slices of the i-th stage of the coding layer;

[0182] Specifically, four slices with different receptive fields are obtained to provide the network with richer semantic information and to retain as much semantic information as possible for small target objects;

[0183] Using formula By using four dilated convolutions with different dilation rates and a length of 2, four slices with different receptive fields are obtained. At this time, the resolution of each slice will be 1 / 4 of the original, providing different multi-scale information for deep networks.

[0184] Step B2

[0185] According to the formula and We obtain the local and global features for a large receptive field, and the local and global features for a small receptive field, respectively; where f′1 is the local feature and f′2 is the global feature; and concat(·) represents the tensor concatenation operation; f 1×1 (·) represents a 1×1 convolution operation;

[0186] Specifically, the two features with small receptive fields and the two features with large receptive fields are integrated and learned separately to avoid semantic discrepancies caused by excessive differences in receptive fields, thus forming more representative local features f′1 and global features f′2.

[0187] Step B3, using the formula

[0188] Attention and self-attention learning are performed on local features and global features respectively to obtain the learned local features and global features;

[0189] Where R(·) represents the dimension reshape operation, transforming C×H×W into C×HW; DR(·) represents the dimension reshape operation, transforming C×HW into C×H×W; S(·) is the non-linear action function Sigmoid function, which normalizes the feature values ​​to the range [0,1] to obtain the attention mask; SA represents the spatial attention module; This represents matrix multiplication, used for weighted features;

[0190] Specifically, attention mechanisms and self-attention mechanisms are used to learn local features f′1 and global features f′2, respectively. Finally, the spatial weight matrices obtained by different mechanisms are added together to further enhance the detailed expression of semantic information in the spatial dimension.

[0191] Step B4: Using the formula The local and global features are refined to obtain the refined semantic features. Here, DSAT(·) represents dual spatial attention; CA represents channel attention module; and MaxPool(·) represents max pooling operation.

[0192] Specifically, the refined features learned by the dual spatial attention module are further refined using channel attention.

[0193] The spatial attention module and channel attention module in the slice downsampling unit of the encoding part are as follows:

[0194] Using formula Obtain the weighted matrix for each pixel in the spatial location; where f 1×1 (·) represents a 1×1 convolution operation; Sigmoid(·) is a non-linear function. This represents matrix multiplication and is used for weighted features.

[0195] Using formula The channel-wise weighted scores are obtained; where Avg(·) represents the global average pooling operation; Linear(·) represents the fully connected operation; and S(·) is the non-linear Sigmoid activation function. This represents matrix multiplication and is used for weighted features.

[0196] Step B5: Using the formula The refined semantic features are supplemented into the downsampled semantic features through matrix addition to obtain feature information of different receptive fields; where MaxPool(·) represents the max pooling operation; This indicates that the elements at the corresponding positions are added together.

[0197] Specifically, the refined detailed semantics were supplemented into the downsampled semantic features through matrix addition, further preserving and enhancing the detailed features of small targets.

[0198] Based on the above embodiments, step S2022 may optionally include:

[0199] Step C1

[0200] Using formula and Feature extraction is performed from local deep networks and local shallow networks respectively to obtain shallow semantic features and deep semantic features;

[0201] Step C2

[0202] Using formula Hybrid feature extraction is performed on shallow and deep semantic features to obtain high-level and low-level semantic information; among them, Indicates shallow network characteristics; Indicates the characteristics of deep networks; This indicates that the elements at corresponding positions are added together; M CA (·) represents the nonlinear action function;

[0203] Step C3

[0204] Using formula By progressively weighting high-level semantic information into low-level semantic information, we obtain global decoding features and optimized decoding features.

[0205] Using formula By progressively weighting high-level semantic information into low-level feature information, the communication between contexts is enhanced, generating decoding features that are more conducive to global guidance and detailed optimization, thereby improving the recognition rate of small targets.

[0206] Specifically, such as Figure 5 As shown, the interactive localization module in the Neck part of the CNN neural network model.

[0207] Using formula By progressively weighting high-level semantic information into low-level feature information, the communication between contexts is enhanced, generating decoding features that are more conducive to global guidance and detailed optimization, thereby improving the recognition rate of small targets; among which, Indicates shallow network characteristics; Represents deep network features; C(·) represents the tensor concatenation operation; f 3×3 (·) represents a 3×3 convolution operation; f 1×1 (·) represents a 1×1 convolution operation; This indicates that the elements at corresponding positions are added together; M CA (·) is a non-linear action function that normalizes the eigenvalues ​​to the range [0,1] to obtain the attention mask; NCBL(·) represents N CBL(·) operations.

[0208] The attention masking operation in the interactive positioning module of the Neck section (M) CAThe NCBL operation, through the fusion of information from shallow and deep layers, enhances deep semantics, strengthens semantic communication between different adjacent layers, and mitigates the loss of semantic information for small targets. NCBL(X) consists of N stacked CBL modules, primarily used to construct a local deep network to acquire deep features. Specifically:

[0209] Using formula right In The mixed features are convolved to integrate information and improve the abstraction ability of the features. Furthermore, the convolved features are activated using the Sigmoid function to normalize the feature values ​​to the range [0,1], resulting in the attention mask M. CA ; where f 1×1 (·) represents a 1×1 convolution operation. Using the formula CBL(X)=LeaklyRelu(BN(f 3×3 (X))), first X, i.e. The middle layer represents a shallow network that further integrates information, then uses Batch Normalization (BN) for information normalization to accelerate network convergence, and finally uses LeaklyReLU to increase nonlinear activation capability. NCBL(X) is the superposition of N NCBLs; where f 3×3 (·) represents a 3×3 convolution operation; BN(·) represents a batch normalization operation; LeaklyRelu(·) represents a non-linear activation function.

[0210] Based on the above embodiments, step S2023 may optionally include:

[0211] Step D1: Correct the center and width / height of the bounding box using regression loss to obtain the first loss information;

[0212] Step D2: For the target category and confidence level, the classification loss is used to learn the improved YOLOv5 network model, and the confidence loss is calculated for negative samples to obtain the second loss information;

[0213] Step D3: Calculate the third loss information of the improved YOLOv5 network model using the adaptive gradient method;

[0214] Step D4: Determine one or more of the first loss information, the second loss information, and the third loss information as loss information.

[0215] In the output prediction part of the CNN neural network model, the idea of ​​decoupling is adopted. The decoded features obtained by the Neck part are sent to the detection head and decoupled into classification loss and regression loss. In addition, an additional 3×3 convolutional layer, ReLU function and SiLU activation function are introduced in each branch to improve performance.

[0216] The improved YOLOv5 neural network model was trained and validated using the obtained processed training and validation sets. Regression loss (IOU loss) was used to correct the center and height of the bounding boxes, while classification loss (binary cross-entropy loss) was used to learn the target category and confidence level. Confidence loss was applied only to negative samples to accelerate convergence and improve network accuracy. For the total loss, the model parameters were optimized using an adaptive gradient method (AdamW optimizer) to continuously train the neural network model by minimizing the loss. The training process was stopped when the number of iterations reached a set value, and the neural network model with the best detection performance on the validation set was saved.

[0217] The sample images in the processed test set are input one by one into the neural network model that detects the best performance on the validation set. The neural network model obtained at this time is determined as the traffic sign model.

[0218] Based on the above embodiments, step S103 includes:

[0219] Step E1: Use the Canny operator to obtain the boundary information of the marker image in the model detection result;

[0220] Specifically, the terminal device inputs the acquired traffic image to be identified into a pre-trained traffic sign model, and outputs the coarse detection result of the highway traffic sign, i.e., the model detection result. The Canny algorithm is used to process the model detection result to obtain the boundary information of the highway traffic sign, specifically:

[0221] The road traffic signs obtained in step S3 are saved as TXT files in the model detection results. The coordinate information and category of each road traffic sign in the model detection results are also saved, and the road traffic signs are cropped onto the corresponding original image.

[0222] For the cropped and saved highway traffic sign images, the image is converted using the formula Gray = R * 0.299 + G * 0.587 + B * 0.114, converting the original RGB image into a grayscale image to facilitate boundary extraction.

[0223] For the processed grayscale image, use the formula The image is processed to obtain a Gaussian kernel, and then the obtained Gaussian kernel is used to perform convolution filtering on the grayscale image, allowing signal components below a certain frequency to pass through, while greatly suppressing signal components above that frequency, thus eliminating the noise mixed in during image digitization.

[0224] For the denoised image, the Soble operator is used to calculate the gradient magnitude and direction, and non-maximum suppression is performed on the magnitude along the gradient direction to retain the gradient value of the point with the maximum local gradient, so as to obtain a refined edge.

[0225] Then, two specific thresholds are given for dual threshold detection: a high threshold and a low threshold, with a ratio of 2:1 or 3:1. Points less than the low threshold are discarded and assigned 0; points greater than the high threshold are immediately assigned 1 or 255; points less than the high threshold and greater than the low threshold are determined using 8-connected regions (if there are weak edge pixels in the 8-neighborhood of a strong edge, the weak edge pixel is transformed into a strong edge and assigned 1 or 255).

[0226] Step E2: Perform Hough transform on the boundary information of the marker image to obtain the geometric information features corresponding to the model detection results;

[0227] Based on the above embodiments, step S104 includes:

[0228] Step F1: Compare the geometric information features corresponding to the model detection results with the preset geometric features of the standard image;

[0229] Step F2: If the difference between the geometric information feature and the preset geometric feature is less than a preset range, the image corresponding to the geometric information feature is determined as the target detection result.

[0230] Specifically, Hough transform is used to extract boundary information for judgment and removal of false positive interference, as follows:

[0231] For each type of road traffic sign, the Canny operator and Hough transform are used to extract all boundary information of the sample image. The number of circles, straight lines, triangles and other shapes corresponding to each type of road traffic sign is counted under the optimal threshold and step size.

[0232] For each coarsely segmented highway traffic sign obtained through the Canny algorithm, the optimal parameters of the Hough transform are used to extract line segment information from the same type of sample highway traffic sign image. If the error of the detected line segment (circle, straight line, triangle, etc.) is within the allowable range, it is judged as a highway traffic sign; otherwise, it is judged as a false positive interference target. The final detection result is obtained after Hough transform filtering.

[0233] For example, different types of identifiers have different shapes. For instance, prohibition signs are usually octagonal, such as those for no parking and no left turn; warning signs are usually triangular, such as those for sharp turns, steep slopes, and construction zones. Geometric features can be used for post-screening to improve detection accuracy. The network portion retains as much image information as possible to reduce missed detections; then post-processing is used to reduce false positives (multiple detections).

[0234] This application's embodiments utilize a downsampling attention module creatively constructed based on self-attention and attention mechanisms. This effectively solves the problem of severe information loss in small targets during downsampling, preserving more important features of small targets through multi-scale dilated convolution. Secondly, this invention enhances semantic information exchange at different stages through an interactive localization module, progressively weighting high-level semantic information onto low-level features. This enhances local details, suppresses targets in irrelevant regions, and effectively identifies most false negatives and false positives. Furthermore, geometric information features are compared to further filter and remove false positives, optimizing the final detection results.

[0235] It should be noted that each of the implementable methods in this embodiment can be implemented individually or in any combination without conflict. This application does not limit this.

[0236] Another embodiment of this application provides a traffic sign recognition device for performing the traffic sign recognition method provided in the above embodiments.

[0237] like Figure 6 The diagram shown is a structural schematic of a traffic sign recognition device provided in an embodiment of this application. The traffic sign recognition device includes an acquisition module 601, a determination module 602, a detection module 603, and a comparison module 604, wherein:

[0238] The acquisition module 601 is used to acquire the traffic image to be identified;

[0239] The determination module 602 is used to determine the model detection result based on the traffic image to be identified and the pre-trained traffic sign model. The pre-trained traffic sign model is obtained by improving and training a YOLOv5 network model. The traffic sign model includes at least an encoding module, an interaction localization module, and an output module. The encoding module includes a slice downsampling unit, which adds refined semantic features to the downsampled semantic features. The interaction localization module uses deep and shallow semantic features to determine semantic communication information at different stages, and determines global decoding features and optimized decoding features based on high-level and low-level semantic information. The output module is used to calculate classification loss and regression loss.

[0240] The detection module 603 is used to determine the geometric information features corresponding to the model detection results based on the model detection results;

[0241] The comparison module 604 is used to determine the target detection result of the traffic image to be identified based on the geometric information features and the preset geometric features.

[0242] Regarding the apparatus in this embodiment, the specific manner in which each module performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0243] Another embodiment of this application further supplements the description of the traffic sign recognition device provided in the above embodiments.

[0244] Optionally, the device further includes a model building module, which is used for:

[0245] Obtain a dataset of sample images of traffic signs;

[0246] Based on the sample image dataset, determine the training set, test set, and validation set corresponding to the sample image dataset;

[0247] Obtain the receptive fields corresponding to different hole rates of feature maps in the sample labeled images in the training set, and use attention mechanism and self-attention mechanism to filter and fuse the receptive fields to obtain feature information of different receptive fields. The feature information includes at least refined semantic features.

[0248] By employing deep semantic features and shallow semantic features, semantic communication information at different stages is determined. Furthermore, an attention mechanism is used to weight the deep semantic information layer by layer with the low-level semantic information to determine the global decoding features and optimize the decoding features.

[0249] Based on the global decoding features and optimized decoding features, the loss information is obtained by using classification loss and regression loss.

[0250] If the loss information meets the preset convergence conditions, the resulting improved YOLOv5 network model will be determined as the traffic sign model.

[0251] Optionally, the model building module is used for:

[0252] The various traffic signs in the sample image dataset are statistically analyzed to obtain the statistically analyzed sample sign images.

[0253] Data augmentation methods are used to process the statistically analyzed sample recognition images to obtain a processed sample identification image set. The data augmentation methods include at least one or more of rotation, cropping, or scaling.

[0254] Based on the sample image set and the preset allocation ratio, determine the training set, test set, and validation set corresponding to the sample image dataset;

[0255] The sample labeled images in the training set, test set, and validation set are selected and classified respectively to obtain the ground truth values ​​corresponding to the training set, test set, and validation set. The ground truth values ​​are used for neural network training and testing.

[0256] Optionally, the model building module is used for:

[0257] According to the formula Obtain four slices with different receptive fields;

[0258] Where, f(X) i ) represents the feature of the i-th stage of the coding layer; DC i (·) indicates a dilated convolution operation, and 0, 1, 2, and 3 represent the dilation rate; These are the four slices of the i-th stage of the coding layer;

[0259] According to the formula and We obtain the local and global features for a large receptive field, and the local and global features for a small receptive field, respectively; where f′1 is the local feature and f′2 is the global feature; and concat(·) represents the tensor concatenation operation; f 1×1 (·) represents a 1×1 convolution operation;

[0260] Using formula Attention and self-attention learning are performed on local and global features respectively to obtain the learned local and global features; where R(·) represents the dimension reshape operation, transforming C×H×W into C×HW; DR(·) represents the dimension reshape operation, transforming C×HW into C×H×W; S(·) is the non-linear action function Sigmoid function, which normalizes the feature values ​​to the range [0,1] to obtain the attention mask; SA represents the spatial attention module; This represents matrix multiplication, used for weighted features;

[0261] Using formula The local and global features are refined to obtain the refined semantic features. Here, DSAT(·) represents dual spatial attention; CA represents channel attention module; and MaxPool(·) represents max pooling operation.

[0262] Using formula The refined semantic features are supplemented into the downsampled semantic features through matrix addition to obtain feature information of different receptive fields; where MaxPool(·) represents the max pooling operation; This indicates that the elements at the corresponding positions are added together.

[0263] Optionally, the model building module is used for:

[0264] Using formula and Feature extraction is performed from local deep networks and local shallow networks respectively to obtain shallow semantic features and deep semantic features;

[0265] Using formula Hybrid feature extraction is performed on shallow and deep semantic features to obtain high-level and low-level semantic information; among them, Indicates shallow network characteristics; Indicates the characteristics of deep networks; This indicates that the elements at corresponding positions are added together; M CA (·) represents the nonlinear action function;

[0266] Using formula By progressively weighting high-level semantic information into low-level semantic information, we obtain global decoding features and optimized decoding features.

[0267] Optionally, the model building module is used for:

[0268] The center and width / height of the bounding box are corrected using regression loss to obtain the first loss information;

[0269] For the target category and confidence level, classification loss is used to learn the improved YOLOv5 network model, and confidence loss is calculated for negative samples to obtain the second loss information;

[0270] The third loss information of the improved YOLOv5 network model is calculated using the adaptive gradient method.

[0271] One or more of the first loss information, the second loss information, and the third loss information are identified as loss information.

[0272] Optionally, the comparison module is used for:

[0273] The Canny operator is used to obtain the boundary information of the marker image in the model detection results;

[0274] The boundary information of the marker image is subjected to Hough transform to obtain geometric information features corresponding to the model detection results;

[0275] Based on geometric information features and preset geometric features, the target detection results of the traffic image to be identified are determined, including:

[0276] The geometric information features corresponding to the model detection results are compared with the preset geometric features of the standard image;

[0277] If the difference between the geometric information feature and the preset geometric feature is less than a preset range, the image corresponding to the geometric information feature will be determined as the target detection result.

[0278] Regarding the apparatus in this embodiment, the specific manner in which each module performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0279] It should be noted that each of the implementable methods in this embodiment can be implemented individually or in any combination without conflict. This application does not limit this.

[0280] like Figure 7 The diagram shown is a structural schematic of a traffic sign recognition device provided in an embodiment of this application. The traffic sign recognition device includes:

[0281] The sample acquisition module 701 is used to acquire a sample image dataset of traffic signs;

[0282] The sample set processing module 702 is used to determine the training set, test set, and validation set corresponding to the sample image dataset based on the sample image dataset;

[0283] The encoding module 703 is used to obtain the receptive fields corresponding to different hole rates of feature maps in the sample identification images of the training set, and to filter and fuse the receptive fields using attention mechanism and self-attention mechanism to obtain feature information of different receptive fields, wherein the feature information includes at least refined semantic features.

[0284] The interactive positioning module 704 is used to determine semantic communication information at different stages by using deep semantic features and shallow semantic features, and to use an attention mechanism to weight the deep semantic information layer by layer with the low-level semantic information to determine the global decoding features and the optimized decoding features.

[0285] The output module 705 is used to calculate loss information based on the global decoding features and the optimized decoding features, using classification loss and regression loss.

[0286] The judgment module 706 is used to determine the obtained improved YOLOv5 network model as the traffic sign model if the loss information meets the preset convergence condition.

[0287] This application also provides a computer-readable storage medium storing a computer program thereon. When the program is executed by a processor, it can perform the operation of any of the methods corresponding to the traffic sign recognition methods provided in the above embodiments.

[0288] This application also provides a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operation of any of the methods corresponding to the traffic sign recognition methods provided in the above embodiments.

[0289] like Figure 8As shown, some embodiments of this application provide an electronic device 800, which includes a memory 810, a processor 820, and a computer program stored in the memory 810 and executable on the processor 820. When the processor 820 reads the program from the memory 810 via a bus 830 and executes the program, it can implement any of the methods included in the above-described traffic sign recognition method.

[0290] Processor 820 can process digital signals and can include various computing architectures. Examples include complex instruction set computer architectures, reduced instruction set computer architectures, or architectures that implement multiple instruction set combinations. In some examples, processor 820 can be a microprocessor.

[0291] The memory 810 can be used to store instructions executed by the processor 820 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all of the functions of one or more modules described in the embodiments of this application. The processor 820 of this disclosure embodiment can be used to execute the instructions in the memory 810 to implement the methods shown above. The memory 810 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0292] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0293] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0294] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A traffic sign recognition method, characterized in that, The method includes: Acquire the traffic image to be identified; Based on the traffic image to be identified and the pre-trained traffic sign model, the model detection result is determined. The pre-trained traffic sign model is obtained by improving and training a YOLOv5 network model. The traffic sign model includes at least an encoding module, an interactive localization module, and an output module. The encoding module includes a slice downsampling unit. This unit is added during the encoding stage of the traffic sign model to slice the receptive field. It uses attention and self-attention mechanisms to filter and fuse multi-scale information, adaptively learning the discriminative features of small objects in multiple receptive fields and supplementing them into the max-pooling downsampling operation. This strengthens and enriches the detailed features of small objects. The slice downsampling unit is used to add refined semantic features to the downsampled semantic features. The interactive positioning module is used to determine semantic communication information at different stages by using deep semantic features and shallow semantic features, to perform mixed feature extraction on the shallow semantic features and the deep semantic features to obtain high-level semantic information and low-level semantic information, and to weight the high-level semantic information layer by layer into the low-level semantic information to obtain global decoding features and optimized decoding features. The output module is used to calculate classification loss and regression loss. Based on the model detection results, determine the geometric information features corresponding to the model detection results; Based on the geometric information features and preset geometric features, the target detection result of the traffic image to be identified is determined; in: The pre-trained traffic sign model is obtained in the following way: Obtain a dataset of sample images of traffic signs; Based on the sample image dataset, determine the training set, test set, and validation set corresponding to the sample image dataset; The receptive fields corresponding to different hole rates of feature maps in the sample identification images in the training set are obtained, and the receptive fields are filtered and fused using attention mechanism and self-attention mechanism to obtain feature information of different receptive fields, wherein the feature information includes at least refined semantic features. Deep semantic features and shallow semantic features are used to determine semantic communication information at different stages, and an attention mechanism is used to weight high-level semantic information with low-level semantic information layer by layer to determine the global decoding features and the optimized decoding features. Based on the global decoding features and the optimized decoding features, loss information is obtained by calculating classification loss and regression loss. If the loss information satisfies the preset convergence condition, the resulting improved YOLOv5 network model is determined as the traffic sign model. The method employs deep and shallow semantic features to determine semantic communication information at different stages, and uses an attention mechanism to weight the high-level semantic information layer by layer with the low-level semantic information to determine the global decoding features and the optimized decoding features, including: Using formula Feature extraction is performed from local deep networks and local shallow networks respectively to obtain shallow semantic features and deep semantic features; Using formula The shallow semantic features and the deep semantic features are combined for feature extraction to obtain high-level semantic information and low-level semantic information; wherein, Indicates shallow network characteristics; Represents the characteristics of deep networks; This indicates that the elements at corresponding positions are added together; It is a nonlinear action function; Using formula The high-level semantic information is weighted layer by layer into the low-level semantic information to obtain the global decoding feature and the optimized decoding feature.

2. The traffic sign recognition method according to claim 1, characterized in that, The step of determining the training set, test set, and validation set corresponding to the sample image dataset based on the sample image dataset includes: The various traffic signs in the sample image dataset are statistically analyzed to obtain statistically analyzed sample sign images. The statistically analyzed sample recognition images are processed using data augmentation methods to obtain a processed sample identification image set. The data augmentation methods include at least one or more of rotation, cropping, or scaling. Based on the sample image set and the preset allocation ratio, determine the training set, test set, and validation set corresponding to the sample image dataset; The sample identification images in the training set, test set, and validation set are selected and classified respectively to obtain the ground truth values ​​corresponding to the training set, test set, and validation set. The ground truth values ​​are used for neural network training and testing.

3. The traffic sign recognition method according to claim 2, characterized in that, The receptive fields corresponding to different hole rates in the feature maps of the sample identification images in the training set are obtained, and the receptive fields are filtered and fused using attention and self-attention mechanisms to obtain feature information of different receptive fields. The feature information includes at least refined semantic features, including: According to the formula Four slices with different receptive fields were obtained; in, For the coding layer Characteristics of the stage; This indicates a dilated convolution operation, where 0, 1, 2, and 3 represent the dilation rate. This is the coding layer. The four slices of the stage; According to the formula and Local and global features with large receptive fields, and local and global features with small receptive fields, are obtained respectively; among them, For local features and For global features; among them, This indicates a tensor concatenation operation; This represents a 1×1 convolution operation; Using formula Attention and self-attention learning are performed on the local features and the global features respectively to obtain the learned local features and global features; wherein, The Reshape operation represents the dimension, transforming C×H×W into C×HW; The Reshape operation represents the dimension, transforming C×HW into C×H×W; The sigmoid function is used as the non-linear interaction function to normalize the eigenvalues ​​to the range [0,1], thus obtaining the attention mask; SA represents the spatial attention module; This represents matrix multiplication, used for weighted features; Using formula The local features and the global features are refined to obtain refined semantic features, wherein... Represents dual spatial attention; CA represents channel attention module; This represents the max pooling operation; Using formula The refined semantic features are then added to the downsampled semantic features using matrix addition to obtain feature information for different receptive fields; wherein... This represents the max pooling operation; This indicates that the elements at the corresponding positions are added together.

4. The traffic sign recognition method according to claim 3, characterized in that, The loss information is obtained by calculating classification loss and regression loss based on the global decoding features and the optimized decoding features, including: The center and width / height of the bounding box are corrected using regression loss to obtain the first loss information; For the target category and confidence level, the improved YOLOv5 network model is trained using classification loss, and confidence loss is calculated for negative samples to obtain the second loss information; The third loss information of the improved YOLOv5 network model is calculated using the adaptive gradient method. One or more of the first loss information, the second loss information, and the third loss information are determined as the loss information.

5. The traffic sign recognition method according to claim 1, characterized in that, The step of determining the geometric information features corresponding to the model detection results based on the model detection results includes: The Canny operator is used to obtain the boundary information of the marker image in the model detection result; The boundary information of the marker image is subjected to Hough transform to obtain geometric information features corresponding to the detection results of the model; The step of determining the target detection result of the traffic image to be identified based on the geometric information features and preset geometric features includes: The geometric information features corresponding to the detection results of the model are compared with the preset geometric features of the standard image; If the difference between the geometric information feature and the preset geometric feature is less than a preset range, the image corresponding to the geometric information feature is determined as the target detection result.

6. A training method for a traffic sign model, characterized in that, The method includes: Obtain a dataset of sample images of traffic signs; Based on the sample image dataset, determine the training set, test set, and validation set corresponding to the sample image dataset; The receptive fields corresponding to different hole rates of feature maps in the sample identification images in the training set are obtained, and the receptive fields are filtered and fused using attention mechanism and self-attention mechanism to obtain feature information of different receptive fields, wherein the feature information includes at least refined semantic features. By employing deep and shallow semantic features, semantic communication information at different stages is determined. Furthermore, an attention mechanism is used to progressively weight high-level semantic information with low-level semantic information to determine global decoding features and optimize decoding features. Based on the global decoding features and the optimized decoding features, loss information is obtained by calculating classification loss and regression loss. If the loss information satisfies the preset convergence condition, the resulting improved YOLOv5 network model is determined as the traffic sign model. Wherein: the use of deep semantic features and shallow semantic features to determine semantic communication information at different stages, and the use of an attention mechanism to weight the high-level semantic information layer by layer with the low-level semantic information to determine the global decoding features and the optimized decoding features, including: Using formula Feature extraction is performed from local deep networks and local shallow networks respectively to obtain shallow semantic features and deep semantic features; Using formula The shallow semantic features and the deep semantic features are combined for feature extraction to obtain high-level semantic information and low-level semantic information; wherein, Indicates shallow network characteristics; Represents the characteristics of deep networks; This indicates that the elements at corresponding positions are added together; It is a nonlinear action function; Using formula The high-level semantic information is weighted layer by layer into the low-level semantic information to obtain the global decoding feature and the optimized decoding feature.

7. A traffic sign recognition device, characterized in that, The device includes: The acquisition module is used to acquire traffic images to be identified; A determination module is used to determine the model detection result based on the traffic image to be identified and a pre-trained traffic sign model. The pre-trained traffic sign model is obtained by improving and training a YOLOv5 network model. The traffic sign model includes at least an encoding module, an interaction localization module, and an output module. The encoding module includes a slice downsampling unit. By adding a slice downsampling unit during the encoding stage of the traffic sign model, the receptive field slices are processed. Attention and self-attention mechanisms are used to filter and fuse multi-scale information, adaptively learning the discriminative features of small objects in multiple receptive fields. This is added to the max pooling downsampling operation to enhance and enrich the detailed features of small target objects. The slice downsampling unit is used to add refined semantic features to the downsampling semantic features. The interactive localization module is used to use deep semantic features and shallow semantic features to determine the semantic communication information at different stages. The shallow semantic features and the deep semantic features are mixed to extract high-level semantic information and low-level semantic information. The high-level semantic information is then weighted layer by layer into the low-level semantic information to obtain global decoding features and optimized decoding features. The output module is used to calculate classification loss and regression loss. The detection module is used to determine the geometric information features corresponding to the model detection results based on the model detection results; The comparison module is used to determine the target detection result of the traffic image to be identified based on the geometric information features and preset geometric features; in: The pre-trained traffic sign model is obtained in the following way: Obtain a dataset of sample images of traffic signs; Based on the sample image dataset, determine the training set, test set, and validation set corresponding to the sample image dataset; The receptive fields corresponding to different hole rates of feature maps in the sample identification images in the training set are obtained, and the receptive fields are filtered and fused using attention mechanism and self-attention mechanism to obtain feature information of different receptive fields, wherein the feature information includes at least refined semantic features. Deep semantic features and shallow semantic features are used to determine semantic communication information at different stages, and an attention mechanism is used to weight the high-level semantic information with the low-level semantic information layer by layer to determine the global decoding features and the optimized decoding features. Based on the global decoding features and the optimized decoding features, loss information is obtained by calculating classification loss and regression loss. If the loss information satisfies the preset convergence condition, the resulting improved YOLOv5 network model is determined as the traffic sign model. The method employs deep and shallow semantic features to determine semantic communication information at different stages, and uses an attention mechanism to weight the high-level semantic information layer by layer with the low-level semantic information to determine the global decoding features and the optimized decoding features, including: Using formula Feature extraction is performed from local deep networks and local shallow networks respectively to obtain shallow semantic features and deep semantic features; Using formula The shallow semantic features and the deep semantic features are combined for feature extraction to obtain high-level semantic information and low-level semantic information; wherein, Indicates shallow network characteristics; Represents the characteristics of deep networks; This indicates that the elements at corresponding positions are added together; It is a nonlinear action function; Using formula The high-level semantic information is weighted layer by layer into the low-level semantic information to obtain the global decoding feature and the optimized decoding feature.

8. A training device for a traffic sign model, characterized in that, The device includes: The sample acquisition module is used to acquire a sample image dataset of traffic signs; The sample set processing module is used to determine the training set, test set, and validation set corresponding to the sample image dataset based on the sample image dataset. The encoding module is used to obtain the receptive fields corresponding to different hole rates of feature maps in the sample identification images of the training set, and to filter and fuse the receptive fields using attention mechanism and self-attention mechanism to obtain feature information of different receptive fields, wherein the feature information includes at least refined semantic features. The interactive localization module is used to determine semantic communication information at different stages by using deep semantic features and shallow semantic features. It also uses an attention mechanism to weight high-level semantic information with low-level semantic information layer by layer to determine global decoding features and optimize decoding features. The output module is used to calculate loss information based on the global decoding features and the optimized decoding features, using classification loss and regression loss. The judgment module is used to determine the improved YOLOv5 network model as the traffic sign model if the loss information meets the preset convergence condition. The interactive positioning module is used for: Using formula Feature extraction is performed from local deep networks and local shallow networks respectively to obtain shallow semantic features and deep semantic features; Using formula The shallow semantic features and the deep semantic features are combined for feature extraction to obtain high-level semantic information and low-level semantic information; wherein, Indicates shallow network characteristics; Represents the characteristics of deep networks; This indicates that the elements at corresponding positions are added together; It is a nonlinear action function; Using formula The high-level semantic information is weighted layer by layer into the low-level semantic information to obtain the global decoding feature and the optimized decoding feature.

9. An electronic device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it can implement the traffic sign recognition method according to any one of claims 1-5 or the training method for the traffic sign model according to claim 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, characterized in that, when the program is executed by a processor, it can implement the traffic sign recognition method according to any one of claims 1-5 or the training method for the traffic sign model according to claim 6.

Images