Method and system for unmanned vehicle to recognize traffic signs under extreme weather

Abstract

The application discloses a method and system for unmanned vehicle to identify traffic signs under extreme weather, wherein the method comprises the following steps: acquiring a traffic sign image to be identified under extreme weather; the extreme weather refers to rainy and snowy weather or fog and haze weather; for the rainy and snowy weather, the traffic sign image to be identified is subjected to rainy and snowy preprocessing; for the fog and haze weather, the traffic sign image to be identified is subjected to fog and haze preprocessing; and the preprocessed image is subjected to identification by using a trained traffic sign identification model to obtain an identified traffic sign.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method and system for unmanned vehicles to recognize traffic signs under extreme weather conditions. Background Technology

[0002] The statements in this section merely refer to the background art related to this invention and do not necessarily constitute prior art.

[0003] With the intelligent operation of urban transportation, autonomous vehicles are gradually developing and becoming a target for research and development. The advancement of autonomous vehicles inevitably requires the rapid development of traffic sign detection and recognition, leading to the widespread deployment of intelligent vehicles in the near future. Traffic sign recognition for autonomous driving refers to using onboard cameras to acquire road scene images and identify road signs and semantics within those images. This is a crucial factor in autonomous driving's ability to determine current road directions. How to enable cars to automatically and accurately identify road signs is of significant research importance. In image processing, image data quality is a key influencing factor. However, during the acquisition process, sensing devices inevitably encounter varying degrees of obstruction due to extreme weather conditions, often resulting in substandard image quality, which causes considerable inconvenience for subsequent data analysis.

[0004] Chinese patent document CN114140766A discloses an improved method and apparatus for traffic sign recognition using YOLOv3. This invention utilizes a trained YOLOv3 network model to recognize traffic signs. By performing data augmentation on the images, overfitting can be effectively prevented while increasing the model's generalization ability. Introducing channel attention enhancement into the deep network model can improve the accuracy and efficiency of traffic sign recognition; however, the detection efficiency is relatively low in real-world road environments.

[0005] Chinese patent document CN114998866A discloses a traffic sign recognition method based on an improved YOLOv4 network. This invention's improved YOLOv4 network can increase the detection accuracy of small target traffic signs while maintaining detection speed, but its recognition rate is relatively low under extreme weather interference.

[0006] Chinese patent document CN114821519A discloses a traffic sign recognition method and system based on coordinate attention. This invention balances the performance of detection speed and recognition accuracy, improves the feature extraction capability of the network and improves the detection effect of occluded targets and small targets, but its compatibility with complex road scenes is poor.

[0007] The methods described in the aforementioned patent documents all require high-quality experimental data and share a common problem: there is still significant room for improvement in handling traffic signs under extreme weather conditions. This is because natural weather conditions, lighting angles, and the complex background of actual roads can all affect the accuracy of small-target traffic sign recognition, leading to lower recognition accuracy.

[0008] The inventors discovered that when recognizing traffic signs in extreme weather conditions, current mainstream feature extractors, while reducing spatial redundancy and learning high-dimensional traffic sign features during the downsampling process of feature maps captured in real-time on-vehicle operation, eliminate the representation of small traffic sign targets, resulting in low recognition rates for these targets. Because the features of small traffic sign targets are easily affected by extreme weather, the detection network struggles to acquire crucial semantic information, leading to low recognition accuracy. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention provides a method and system for unmanned vehicles to identify traffic signs under extreme weather conditions. It solves the problems in existing technologies where traffic sign recognition accuracy is low due to the complexity of real-world road environments, issues such as lighting, target occlusion, and difficulty in detecting small targets, and the inability to quickly identify signs due to high vehicle speeds in real-world road scenarios. The invention achieves the goal of accurately identifying small traffic sign targets through a traffic sign recognition model, balancing detection speed and recognition accuracy, improving the network's feature extraction capabilities, and enhancing the detection effect on occluded and small targets, enabling real-time detection of road traffic signs in real-world scenarios.

[0010] In a first aspect, the present invention provides a method for unmanned vehicles to identify traffic signs under extreme weather conditions;

[0011] Methods for autonomous vehicles to recognize traffic signs in extreme weather conditions include:

[0012] Under extreme weather conditions, acquire traffic sign images to be identified; extreme weather refers to rainy, snowy, or foggy weather.

[0013] For rainy or snowy weather, the traffic sign image to be identified is preprocessed to remove rain and snow; for foggy or hazy weather, the traffic sign image to be identified is preprocessed to remove fog.

[0014] The preprocessed image is then used to identify the traffic sign using a trained traffic sign recognition model, resulting in the identified traffic sign.

[0015] Secondly, this invention provides a system for unmanned vehicles to recognize traffic signs under extreme weather conditions;

[0016] Systems for autonomous vehicles to recognize traffic signs in extreme weather conditions include:

[0017] The acquisition module is configured to acquire a traffic sign image to be identified under extreme weather conditions; the extreme weather conditions refer to rainy, snowy, or foggy weather.

[0018] The preprocessing module is configured to: perform rain and snow removal preprocessing on the traffic sign image to be identified for rainy or snowy weather; and perform fog removal preprocessing on the traffic sign image to be identified for foggy or hazy weather.

[0019] The recognition module is configured to: recognize the preprocessed image using a trained traffic sign recognition model to obtain the recognized traffic signs.

[0020] Thirdly, the present invention also provides an electronic device, comprising:

[0021] Memory, used for non-transitory storage of computer-readable instructions; and

[0022] Processor, for executing the computer-readable instructions,

[0023] When the computer-readable instructions are executed by the processor, they perform the method described in the first aspect above.

[0024] Fourthly, the present invention also provides a storage medium for non-transitory storage of computer-readable instructions, wherein, when the non-transitory computer-readable instructions are executed by a computer, the instructions for executing the method described in the first aspect are executed.

[0025] Fifthly, the present invention also provides a computer program product, including a computer program that, when run on one or more processors, is used to implement the method described in the first aspect above.

[0026] Compared with the prior art, the beneficial effects of the present invention are:

[0027] This invention employs a coordinate attention mechanism, which can effectively capture the relationship between location information and channels, more accurately obtain the region of interest, and reduce the interference of extreme weather conditions, thereby improving the feature weight of positive samples.

[0028] This invention employs SPD-Conv, incorporating this new convolutional neural network module, which effectively solves the problems of loss of fine-grained information and low-efficiency feature representation learning caused by convolution stride and each pooling layer.

[0029] This invention employs a weighted bidirectional feature pyramid network, which integrates more features without increasing costs, thereby improving the network's feature extraction capabilities and enhancing the detection performance of small traffic signs under extreme weather conditions.

[0030] This invention uses a dual decoupling head to accelerate network convergence and improve the accuracy of small traffic sign targets under extreme weather conditions.

[0031] This invention employs the loss function SIoU, which improves the training speed and inference accuracy of traffic sign recognition networks suitable for extreme weather conditions.

[0032] This invention replaces the C3 module with a lightweight C2f module, ensuring lightweight design while obtaining richer gradient flow information, making the traffic sign recognition model more lightweight and easier to deploy on autonomous vehicles. Attached Figure Description

[0033] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0034] Figure 1 This is a flowchart of the method in Example 1;

[0035] Figure 2 This is the traffic sign recognition model for Example 1;

[0036] Figure 3 The diagram shows the SPD-Conv when the scale is 2 in Example 1.

[0037] Figure 4 This is a schematic diagram of the internal structure of the CBS module and C2f module in Example 1;

[0038] Figure 5 This is a schematic diagram of the internal structure of the SPPF spatial pyramid pooling feature module in Example 1;

[0039] Figure 6 This is a schematic diagram of the internal structure of the Coordinate Attention mechanism in Example 1;

[0040] Figure 7 This is a schematic diagram of the internal structure of the Decoupled Head detection head in Example 1. Detailed Implementation

[0041] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0042] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0043] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0044] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.

[0045] Example 1

[0046] This embodiment provides a method for unmanned vehicles to recognize traffic signs under extreme weather conditions;

[0047] like Figure 1 As shown, the method for autonomous vehicles to recognize traffic signs under extreme weather conditions includes:

[0048] S101: Under extreme weather conditions, acquire the traffic sign image to be identified; the extreme weather refers to rainy, snowy, or foggy weather.

[0049] S102: For rainy or snowy weather, perform de-raining and de-snowing preprocessing on the traffic sign image to be identified; for foggy or hazy weather, perform de-foging preprocessing on the traffic sign image to be identified.

[0050] S103: The preprocessed image is then identified using the trained traffic sign recognition model to obtain the identified traffic signs.

[0051] Further, in step S101: the traffic sign image to be identified is acquired by using a high-definition camera of the autonomous vehicle for image acquisition.

[0052] Furthermore, in step S102: the traffic sign image to be identified is preprocessed for rain and snow removal using a multi-stage progressive image restoration network (MPR-Net).

[0053] It should be understood that using a multi-stage progressive image restoration network for preprocessing images of rain and snow can achieve a balance between spatial detail and high-level contextual information during the restoration of rain and snow images.

[0054] Furthermore, in step S102: the traffic sign image to be identified is preprocessed for dehazing by using a feature fusion attention network (FFA-Net).

[0055] It should be understood that the use of the FusionFeature Attention Network (FFA-Net) for preprocessing foggy images is an end-to-end approach to image dehazing.

[0056] Further, in step S103: the preprocessed image is identified using a trained traffic sign recognition model to obtain the identified traffic signs. The network structure of the trained traffic sign recognition model includes:

[0057] The backbone layer, neck layer, attention layer, and prediction layer are connected in sequence.

[0058] The backbone layer is used to extract features from the preprocessed image; the neck layer is responsible for multi-scale feature fusion of the feature map; the attention layer is used to focus on local information; and the prediction layer is used to perform the final regression prediction.

[0059] Furthermore, the backbone layer includes: a Focus convolutional neural network layer, a CBS1 layer, a convolutional layer SPD-Conv1, a C2f1 layer, a CBS2 layer, a convolutional layer SPD-Conv2, a C2f2 layer, a CBS3 layer, a convolutional layer SPD-Conv3, a C2f3 layer, a CBS4 layer, a convolutional layer SPD-Conv4, a C2f4 layer, and an SPPF layer, connected in sequence.

[0060] The Focus convolutional neural network layer is used to shrink large input feature maps into smaller ones, improving the model's speed and efficiency. The Focus layer achieves feature map scaling through strided convolutions and dilated convolutions.

[0061] The internal structure of the Focus convolutional neural network layer includes two strided convolutional layers and one dilated convolutional layer connected in sequence. First, the large input feature map is reduced to half its original size by a strided convolutional layer, and then further reduced to one-quarter of its original size by another strided convolutional layer. Finally, a dilated convolutional layer is used to extract features from the reduced feature map.

[0062] The Focus convolutional neural network layer works by reducing the size of the input feature map to a smaller size, thereby reducing the number of parameters and computational cost in the model, thus improving its speed and efficiency. Simultaneously, by using strided convolutions and dilated convolutions, the receptive field and feature extraction capability of the reduced feature map are preserved, ensuring that the model's detection accuracy is not affected. The Focus convolutional neural network layer is designed to adapt to various input and target sizes, thereby improving the model's versatility and robustness.

[0063] CBS layers are a novel type of convolutional operation consisting of compressed convolution and dilated convolution. They can compress and expand the feature maps of a model, thereby improving the model's receptive field and feature representation ability.

[0064] Furthermore, the internal structures of the convolutional layers SPD-Conv1, SPD-Conv2, SPD-Conv3, and SPD-Conv4 are the same, and they are all implemented using the convolutional layer SPD-conv.

[0065] Furthermore, such as Figure 3 As shown, the convolutional layer SPD-Conv1 includes: a space-to-depth (SPD) layer and a non-strided convolution (Conv) layer.

[0066] In a space-to-depth (SPD) layer, given any (original) feature map x, the slice of the sub-feature map sequence is fx. x,y f x,y The feature map X(i,j) is composed of all feature maps. i+x and j+y are divisible by a scale factor. Therefore, each submap is downsampled by a scale factor x. Figure 3 An example is given where, when scale = 2, we obtain four sub-maps f. 0,0 ,f 0,1 ,f 1,0 ,f 1,1 Their shapes are Then, x is downsampled by a factor of 2. Next, these sub-feature maps are connected along the channel dimension to obtain a feature map x′, which has a reduced spatial dimension by a scaling factor and a increased channel dimension by a scaling factor of 2.

[0067] Furthermore, the working process of the convolutional layer SPD-Conv1 includes: first using a space-to-depth (SPD) layer to perform adaptive convolution operation on the input feature map, and then using a convolution-free stride layer to perform convolution operation on the output feature map.

[0068] Furthermore, the internal structures of the C2f1, C2f2, C2f3 and C2f4 layers are identical.

[0069] Furthermore, such as Figure 4 As shown, the C2f1 layer includes:

[0070] The CBS5 layer, the Split network layer, the first Bottleneck residual block structure layer, the serial splicing layer L1, and the CBS6 layer are connected in sequence.

[0071] The output of the first Bottleneck residual block structure layer is connected to the input of the second Bottleneck residual block structure layer.

[0072] The output of the second Bottleneck residual block structure layer is connected to the input of the third Bottleneck residual block structure layer.

[0073] The outputs of the second and third Bottleneck residual block structures are both connected to the input of the serial splicing layer L1; the output of the Split network layer is connected to the input of the serial splicing layer L1.

[0074] Furthermore, the Split network layer divides the input feature map into two branches. One branch is output directly without any operation, while the other branch is transformed before being output. This method can improve the diversity and robustness of the model and reduce the gradient vanishing problem, thereby improving the training effect of the model. At the same time, the Split network layer can also achieve cross-layer feature fusion, thus better capturing the contextual information of the object.

[0075] Furthermore, the internal structures of the first Bottleneck residual block structure layer, the second Bottleneck residual block structure layer, and the third Bottleneck residual block structure layer are consistent. The first Bottleneck residual block structure layer includes a 1*1 convolutional layer, a 3*3 convolutional layer, and a 1*1 convolutional layer connected in sequence.

[0076] Furthermore, such as Figure 5 As shown, the SPPF spatial pyramid pooling feature layer includes:

[0077] The CBS7 layer, the first max pooling layer, the second max pooling layer, the third max pooling layer, the cascaded splicing layer L2, and the CBS8 layer are connected in sequence; the CBS7 layer serves as the input of the SPPF layer, and the CBS8 layer serves as the output of the SPPF spatial pyramid pooling feature layer.

[0078] The output terminals of the CBS7 layer, the first max pooling layer, and the second max pooling layer are all connected to the input terminal of the serial splicing layer.

[0079] Furthermore, the internal structures of CBS1, CBS2, CBS3, CBS4, CBS5, CBS6, CBS7, and CBS8 are identical.

[0080] Furthermore, such as Figure 4 As shown, the CBS1 layer includes: a convolutional layer, a batch normalization layer, and an activation function layer connected in sequence.

[0081] Furthermore, the working process of the SPPF spatial pyramid pooling feature layer includes: dividing the input feature map into sub-maps of different sizes and performing max pooling on each sub-map to obtain a feature vector of a set size.

[0082] In the SPPF spatial pyramid pooling feature module, different sizes are selected to adapt to target objects of different sizes, increasing the model's receptive field and thus better detecting objects of different sizes. Simultaneously, the SPPF spatial pyramid pooling feature module can reduce feature map compression, thereby avoiding information loss. The structure of the SPPF spatial pyramid pooling feature layer involves passing the input in parallel through multiple max pooling operations of different sizes, followed by further fusion, which can address the multi-scale target detection problem to some extent.

[0083] It should be understood that the backbone network's role is to extract features from the image. The original C3 module is replaced by a lighter C2f module, which combines the C3 module with the efficient layer aggregation network concept of ELAN. This design ensures lightweight operation while obtaining richer gradient flow information, making the traffic sign recognition model more lightweight and easier to deploy on autonomous vehicles.

[0084] It should be understood that the backbone network introduces SPD-Conv to better capture the feature information of small target traffic signs. This involves adding a Spatial-to-Depth (SPD) layer followed by a stride-free convolutional layer. The SPD layer downsamples the feature map X but preserves all information in the channel dimension, thus preventing information loss. Image transformation techniques are used to scale the original image before inputting it into the neural network, generalizing it to the downsampled feature maps throughout the network. Furthermore, a stride-free convolutional operation is added after each SPD to reduce the number of channels in the increased convolutional layers using learnable parameters.

[0085] Furthermore, such as Figure 2 As shown, the neck layer includes: CBS9 layer, first upsampling layer, tandem splicing layer L3, C2f5 layer, CBS10 layer, second upsampling layer and tandem splicing layer L4 connected sequentially from bottom to top;

[0086] The neck layer further includes: C2f6 layer, CBS11 layer, series splicing layer L5, C2f7 layer, CBS12 layer, series splicing layer L6 and C2f7 layer connected sequentially from top to bottom;

[0087] The input of the CBS9 layer is connected to the output of the SPPF spatial pyramid pooling feature layer.

[0088] The input terminal of the serial splicing layer L3 is connected to the output terminal of the C2f3 layer;

[0089] The input terminal of the serial splicing layer L4 is connected to the output terminal of the C2f2 layer;

[0090] The output of the CBS9 layer is connected to the input of the serial splicing layer L6;

[0091] The output of the CBS10 layer is connected to the input of the serial splicing layer L5;

[0092] The output terminal of the serial splicing layer L4 is connected to the input terminal of the C2f6 layer.

[0093] It should be understood that, in the neck layer, when the feature source image is decomposed using average pooling, to achieve a better balance between network accuracy and efficiency, the original path aggregation network in the YOLOv5 network is replaced with Bi-FPN (Weighted Bidirectional Feature Pyramid Network). This network employs efficient bidirectional cross-scale connections and uses Bi-FPN for weighted feature fusion, forming a top-down and bottom-up feature pyramid network. When deeper information is needed, additional edges can be added to the existing network, thus fusing more traffic sign features without increasing cost. Because the resolution of input features varies, their impact on the output will also differ. By adding more weights to improve system efficiency, the network can better understand and recognize the criticality of various input features.

[0094] Bi-FPN (Weighted Bidirectional Feature Pyramid Network) assigns different weights to each layer, allowing the network to focus more on key layers while reducing unnecessary inter-layer connections.

[0095] To maximize efficiency, when the original input and output nodes reach the same level, we will add new edges to integrate more information, thereby reducing operating costs. By constructing multiple top-down and bottom-up feature network layers and iterating through these layers, the efficiency of feature fusion can be effectively improved.

[0096] Furthermore, such as Figure 2 As shown, the attention layer includes: three parallel CoordinateAttention modules: a first CoordinateAttention module, a second CoordinateAttention module, and a third CoordinateAttention module;

[0097] The input of the first Coordinate Attention module is connected to the output of the C2f6 layer;

[0098] The input of the second Coordinate Attention module is connected to the output of the C2f7 layer;

[0099] The input of the third Coordinate Attention module is connected to the output of the C2f8 layer.

[0100] Furthermore, the internal structures of the first Coordinate Attention module, the second Coordinate Attention module, and the third Coordinate Attention module are identical.

[0101] Furthermore, such as Figure 6 As shown, the first Coordinate Attention module includes:

[0102] The residual module has its input end serving as the input end of the first Coordinate Attention module; the output end of the residual module is connected to the input end of the weight update module.

[0103] The output of the residual module is also connected to the input of the horizontal average pooling layer and the input of the vertical average pooling layer, respectively; the horizontal average pooling layer and the vertical average pooling layer are both implemented through the average pooling layer.

[0104] The outputs of the horizontal and vertical average pooling layers are connected to the input of the cascaded splicing layer L7. The output of the cascaded splicing layer L8 is connected to the input of the 2D convolutional layer E1. The output of the 2D convolutional layer is connected to the input of the normalization layer. The output of the normalization layer is connected to the input of the nonlinear activation function layer. The output of the nonlinear activation function layer is connected to the inputs of the 2D convolutional layers E2 and E3, respectively. The output of the 2D convolutional layer E2 is connected to the input of the first activation function layer. The output of the 2D convolutional layer E3 is connected to the input of the second activation function layer. The outputs of both the first and second activation function layers are connected to the input of the weight update module. The output of the weight update module serves as the output of the first Coordinate Attention module.

[0105] Furthermore, such as Figure 6 As shown, the first Coordinate Attention module includes:

[0106] Two one-dimensional global pooling operations are used to aggregate the input feature maps along the vertical and horizontal directions into two feature maps that embed directional information, respectively.

[0107] Then, the two feature maps with embedded orientation information are encoded into two attention maps, each of which captures the long-range dependencies of the input feature map along a spatial direction;

[0108] Location information is stored in the generated attention map, and then both attention maps are applied to the input feature map by multiplication to emphasize the representation of the attention region.

[0109] Furthermore, such as Figure 6 As shown, the first Coordinate Attention module's operation includes:

[0110] (11): Input feature map;

[0111] (12): Coordinate extraction: By extracting the coordinate position of the feature map, two coordinate vectors are obtained, which represent the horizontal and vertical coordinates in the feature map, respectively;

[0112] (13): Coordinate embedding: embed the two coordinate vectors respectively to obtain two new vectors, which represent the embedding vector of the horizontal coordinate and the embedding vector of the vertical coordinate respectively.

[0113] (14): Feature embedding: The feature map is embedded through a convolutional layer to obtain a new feature vector;

[0114] (15): Coordinate attention calculation: Use the dot product to calculate the attention weight between the embedding vector and the feature vector of the horizontal coordinate; use the dot product to calculate the attention weight between the embedding vector and the feature vector of the vertical coordinate; thus obtaining two coordinate attention vectors;

[0115] (16): Coordinate attention weighting: The coordinate attention vector and the feature vector are weighted and added together to obtain the final weighted feature vector;

[0116] (17): Output feature map: Output the weighted feature vector.

[0117] It should be understood that the attention layer operates by weighting information from different locations in the feature map to improve the model's accuracy and robustness in detecting targets. By using coordinate extraction and embedding techniques, coordinate position information in the feature map can be fused with the feature vector, resulting in a more expressive feature vector. Furthermore, using the coordinate attention mechanism maintains the same number of model parameters and computational complexity, thereby improving model performance and efficiency.

[0118] It should be understood that the acquisition of feature map information for small target traffic signs often results in the loss of some feature information location. To simultaneously and accurately capture both channel and location information, and to better acquire the feature information of small traffic signs under extreme weather conditions, the location information of each traffic sign feature image block is embedded into channel attention. The information embedding stage first uses pooling kernels of sizes (H,1) and (1,W) on the feature layer input to the coordinate attention mechanism module, pooling each channel in the two-dimensional direction respectively. This achieves better capture of location and channel information with a small increase in computation, thus facilitating the network's localization of targets of interest.

[0119] Furthermore, the working principle of the first Coordinate Attention module is as follows:

[0120] The pooled output of the c-th channel with height H and width W is as follows:

[0121]

[0122] The output of the c-th channel with width w is shown in formula (2).

[0123]

[0124] Where, x c (h,i) and x c (j, w) represent the features respectively. Figure 2 Features of the dimensional direction.

[0125] Furthermore, the prediction layer includes: three parallel first decoupled head detection head modules, second decoupled head detection head modules, and third decoupled head detection head modules;

[0126] The input of the first Decoupled Head detection module is connected to the output of the first Coordinate Attention module;

[0127] The input of the second decoupled head detection module is connected to the output of the second coordinate attention module;

[0128] The input of the third Decoupled Head module is connected to the output of the third Coordinate Attention module.

[0129] Furthermore, the internal structures of the first Decoupled Head detection module, the second Decoupled Head detection module, and the third Decoupled Head detection module are identical.

[0130] Furthermore, such as Figure 7 As shown, the first Decoupled Head detection module includes: a CBS13 layer;

[0131] The input terminal of CBS13 layer serves as the input terminal of the first Decoupled Head dual decoupled detection head module;

[0132] The output of CBS13 layer is connected to the input of CBS14 layer and the input of CBS15 layer, respectively.

[0133] The output of the CBS14 layer is connected to the Cls class layer through the convolutional layer J1 and the third activation function layer in sequence;

[0134] The output of CBS15 layer is connected to the input of convolutional layer J2 and the input of convolutional layer J3, respectively.

[0135] The output of convolutional layer J2 is connected to the Reg coordinate layer;

[0136] The output of convolutional layer J3 is connected to the fourth activation function layer, and the fourth activation function layer is connected to the Obj target layer.

[0137] The Cls category layer refers to a vector of size C representing the probability that the predicted box belongs to each target category, where C is the number of target categories. For each predicted box, a cls category vector is predicted, representing the probability that the box belongs to each target category.

[0138] The Reg coordinate layer refers to the position and size information of the predicted box. It is a vector of size 4, which represents the center coordinates, width and height of the predicted box. For each predicted box, a reg coordinate vector is predicted to represent the position and size information of the box.

[0139] The Obj target layer refers to the confidence level that the predicted box contains a target. It is a scalar. For each predicted box, an obj target value is predicted, indicating whether the box contains a target.

[0140] Furthermore, the first Decoupled Head dual decoupled detection head module's operation includes:

[0141] (21): Target box regression: The feature map is fed into the target box regression branch, and the predicted value of the target box is obtained through convolutional layers and fully connected layers; The output of the target box regression branch usually includes the center coordinates, width and height information of the target box;

[0142] (22): Target classification: The feature map is fed into the target classification branch, and the predicted value of the target category is obtained through convolutional layers and fully connected layers; the output of the target classification branch usually includes the probability value of the target belonging to each category;

[0143] (23): Target box decoding: Convert the output of the target box regression branch into the actual target box coordinates; usually, the anchor box method is used for decoding, that is, the offset is predicted for each anchor box to obtain the actual target box coordinates;

[0144] (24): Target box filtering: Target boxes are filtered and selected based on their confidence score and class probability to obtain the final target detection results.

[0145] Furthermore, the prediction layer's operation includes: dividing the feature map into multiple grids and predicting a set of bounding boxes for each grid; for each bounding box, the prediction layer predicts its position, size, and a first confidence score, which indicates whether the bounding box contains a target; for each target category, the prediction layer predicts a second confidence score, which indicates whether the bounding box contains a target of the current category. These predictions are calculated through convolutional and fully connected layers.

[0146] It should be understood that in the prediction layer, there is a conflict between classification and regression tasks. Therefore, to avoid this conflict, a dual decoupling head, widely used in most first- and second-level detectors for classification and localization, is employed. Figure 7 As shown, the dual decoupling head further decouples the classification and regression subnetworks, designing two independent subnetworks: a detection subnetwork and a classification subnetwork. The detection subnetwork is only used to predict the position and size of objects, while the classification subnetwork is specifically used to predict the category of objects. This improves the convergence speed of the network, makes the network more stable, and improves the accuracy of detecting small traffic signs under extreme weather conditions.

[0147] Further, in step S103: the preprocessed image is identified using a trained traffic sign recognition model to obtain the identified traffic signs. The training process of the trained traffic sign recognition model includes:

[0148] Construct a training set and a test set; both the training set and the test set include: a set of traffic sign images with known traffic sign recognition results;

[0149] The training set is input into the traffic sign recognition model to train the model. When the total loss function value of the model no longer decreases, the training is stopped, and a preliminary traffic sign recognition model is obtained.

[0150] The preliminary traffic sign recognition model is tested using a test set. When the test accuracy exceeds a set threshold, the current preliminary traffic sign recognition model is the trained traffic sign recognition model.

[0151] Furthermore, the construction of the training set specifically includes:

[0152] Collect images of traffic signs from real-world road conditions;

[0153] The collected images are subjected to image enhancement and image filling processing.

[0154] Use data annotation tools to annotate the collected traffic sign images;

[0155] Anchor frames are obtained by clustering the traffic sign dataset using a clustering algorithm.

[0156] It should be understood that the K-means++ clustering algorithm is used to re-cluster the traffic sign dataset to obtain anchor boxes, making the anchor boxes more suitable for traffic sign scenarios in real-world environments, and the number and size of anchor boxes in the YOLOv5 algorithm configuration file are adjusted.

[0157] Furthermore, the total loss function of the model includes:

[0158] The total loss function L is:

[0159] L=W box L box +W cls L cls

[0160] Among them, W box and W cls These are the bounding box and the classification loss weights, L. box It is the regression loss function, L cls It is focal loss, a loss function used to solve the class imbalance problem.

[0161] The regression loss function is:

[0162]

[0163] Where Δ is the distance cost, Ω is the shape cost, and IoU is a metric used to evaluate the degree of overlap between predicted and ground truth bounding boxes in tasks such as object detection and semantic segmentation. IoU is the area of ​​the intersection of the predicted and ground truth bounding boxes divided by the area of ​​their union.

[0164] First, the autonomous driving perception device triggers a target event. The triggered event is when the autonomous vehicle is driving on the road and receives image data frames of the road ahead in real time. Then, the traffic sign recognition model identifies and obtains the traffic sign feature information of the target feature image in the image data frame.

[0165] Secondly, preprocessing is performed on the acquired images. The acquired images under extreme weather conditions are roughly divided into rain / snow and fog / haze. For foggy images, a Feature Fusion Attention Network (FFA-Net) is used for preprocessing, employing an end-to-end approach for image dehazing. On the other hand, for rain / snow images, a multi-stage progressive image restoration network (MPR-Net) is used for preprocessing, enabling a balance between spatial detail and high-level contextual information during image restoration. The preprocessed images are then fed into a model adapted for recognizing minute traffic signs for detection.

[0166] Secondly, after receiving the preprocessed image data frame, the traffic sign recognition model continuously downsamples the received data, then performs two-dimensional pooling decomposition to obtain several decomposed feature image blocks; based on the same dimensional information in the several feature image blocks, it performs splicing and fusion at each stage, and obtains the target feature image after two-dimensional convolution processing.

[0167] To address the challenges of traffic sign recognition under extreme weather conditions and better adapt to interference from such conditions, the traffic sign recognition model is adaptively improved. The traffic sign recognition model in this invention is trained using a lightweight convolutional neural network structure, and the overall process is as follows: Figure 2 As shown.

[0168] The specific implementation process of the traffic sign recognition model in the method of this invention is as follows: First, the acquired image data frames are continuously subjected to pooling processing. An image decomposition network structure is set in the backbone network. After receiving the image data frames, the image decomposition network structure extracts feature source images of different sizes in a loop according to the preset downsampling rules. The feature images are decomposed by average pooling in the two-dimensional direction to obtain several small traffic sign target feature image blocks.

[0169] Before downsampling, the method of this invention includes preprocessing the received image data frames, adjusting the resolution of the input image data frames to a predetermined size, and then performing the downsampling operation. This invention increases the input resolution of the image data frames from 640×640 to 1280×1280 without changing the video memory size, batch size, or number of threads, enabling more accurate acquisition of the feature information of tiny traffic signs, thereby mitigating the loss of traffic sign feature information during downsampling. The K-means++ clustering algorithm is used to re-cluster the traffic label dataset to obtain anchor boxes, making the anchor boxes more suitable for real-world traffic sign scenarios. The number and size of anchor boxes in the YOLOv5 algorithm configuration file are also adjusted. The C3 module is improved by adopting a more lightweight C2f module, such as... Figure 4 As shown, the C2f module is designed with reference to the C3 module and the idea of ​​ELAN's efficient layer aggregation network. It ensures lightweight design while obtaining richer gradient flow information, making the traffic sign recognition model more lightweight and easier to deploy on autonomous vehicles.

[0170] To better capture the feature information of small traffic signs, SPD-Conv is introduced. It involves adding a Spatial-to-Depth (SPD) layer followed by a no-strut convolutional layer. The SPD layer downsamples the feature map X but preserves all information in the channel dimension, thus avoiding information loss. Image transformation techniques are used to scale the original image before feeding it into the neural network, generalizing it to the downsampled feature maps throughout the network. Furthermore, a no-strut convolutional operation is added after each SPD to reduce the number of channels using learnable parameters in the increased convolutional layers. Using SPD-Conv increases the network's sensitivity to small traffic signs, enabling it to capture traffic sign feature information more comprehensively.

[0171] In the practice of traffic sign recognition under extreme weather interference, considering the two important factors of speed and accuracy, when performing average pooling decomposition on the traffic sign feature source images, in order to obtain traffic sign feature information under extreme weather interference more quickly and accurately, the original path aggregation network in the YOLOv5 network was replaced with a weighted bidirectional feature pyramid network. After continuously downsampling the feature points, a set of feature layers with high semantic content were obtained. Then, upsampling was performed again, making the length and width of the feature layers larger. Using large-size feature maps to detect small traffic sign targets is equivalent to assigning different weights to each layer for fusion, making the network pay more attention to important feature layers and semantic information of small traffic sign targets. Moreover, it reduces the node connections of redundant layers without the need to add extra weights, making the model more lightweight.

[0172] Furthermore, in the acquisition of feature map information of minute traffic signs under extreme weather interference, the location of some feature information is often lost. In order to simultaneously and accurately capture channel information and location information, and to better obtain the feature information of minute traffic signs under the influence of extreme weather, the location information of each traffic sign feature image block is embedded into the channel attention, such as... Figure 6 As shown, the information embedding stage first applies pooling kernels of size (H,1) and (1,W) to the feature layer input to the coordinate attention mechanism module, pooling each channel in the two-dimensional direction. This better captures positional and channel information with a small increase in computation, thus facilitating the network's localization of the target of interest.

[0173] After completing the 2D convolution processing, activation functions are used to classify the features after 2D convolution. IoU loss can more accurately predict the localization of bounding box regression. However, YOLOv5's CIoU relies on the aggregation of bounding box regression metrics without considering the mismatch direction between the expected ground box and the predicted "experimental" box. This results in lower training speed and prediction accuracy compared to SIoU. SIoU reduces the loss weight for large and medium-sized targets, alleviating the uneven distribution of feature image patch sizes during recognition, and achieving more accurate loss calculation between predicted and ground truth boxes in traffic sign recognition tasks. In this invention, when classifying target feature images using activation functions, a loss function is also used to determine the true information of traffic sign feature images.

[0174] In the prediction layer, the conflict between classification and regression tasks is a well-known problem. Therefore, dual decoupled heads for classification and localization are widely used in most first- and second-level detectors. The recoupling shown here, replacing the native detector head with a dual decoupled detector head, improves the network's convergence speed.

[0175] Therefore, in response to the problems that arise in traffic sign recognition under extreme weather conditions, the method of this invention balances the performance of detection speed and recognition accuracy, improves the feature extraction capability of the network, and enhances the detection effect of targets and small targets under extreme weather interference, enabling real-time detection of road traffic signs in real extreme weather scenarios.

[0176] Compared with existing technologies, this invention introduces SPD-Conv to replace each convolution stride and each pooling layer, effectively solving the problems of fine-grained information loss and low-efficiency feature representation learning caused by convolution strides and each pooling layer, and better capturing the feature information of small traffic sign targets. A coordinate attention mechanism is employed, such as... Figure 6As shown, embedding location information into channel attention enables the lightweight convolutional neural network structure to acquire information on small traffic signs over a larger area under extreme weather conditions and to more accurately obtain important traffic sign features. A cross-layer connection is added to the feature fusion network structure, i.e., an extra path is added to fuse more features without increasing cost, improving the network's ability to extract features from small targets and enhancing the detection performance of traffic signs under extreme weather interference. This invention employs a dual decoupling head. Because the features for classification and regression are inconsistent, a dual decoupling head is used to accelerate network convergence and improve the detection accuracy of small traffic signs. This invention improves the C3 module by using a lightweight C2f module, such as... Figure 4 As shown, the C2f module is designed with reference to the C3 module and the efficient layer aggregation network concept of ELAN. It ensures lightweight design while obtaining richer gradient flow information, making the traffic sign recognition model more lightweight and easier to deploy on autonomous vehicles. This invention employs SIoU loss to reduce the loss weight of small targets, alleviating the uneven distribution of feature sample sizes during traffic sign recognition. This achieves more accurate loss calculation between predicted and real information in traffic sign recognition tasks, improving the recognition accuracy of small traffic sign targets.

[0177] Comparative data implementation:

[0178] To verify the stability of the proposed algorithm and optimize network parameters, the existing CCTSDB traffic sign dataset was selected as the sample set, and 3000 custom occlusion interference images and a large amount of foggy data were added to fully test the model's generalization ability. The dataset was divided into training and test sets in an 8:2 ratio. The initial learning rate was set to 0.003, and the batch size was set to 16. The experimental hardware configuration is shown in Table 1.

[0179] Table 1: List of Experimental Hardware Components

[0180]

[0181] Table 2 Comparative Experiments (Extended CCTSDB Traffic Sign Dataset):

[0182]

[0183] Experimental results show that the improved YOLOv5 algorithm model of the present invention achieves an mAP@0.5 of 89.1% on the expanded CCTSDB traffic sign dataset, which is 2.3 percentage points higher than YOLOv5. This significantly improves the detection accuracy of traffic signs in practical applications and can more accurately obtain traffic sign information in extreme weather conditions, thereby correctly judging target information.

[0184] Example 2

[0185] This embodiment provides a system for autonomous vehicles to recognize traffic signs under extreme weather conditions;

[0186] Systems for autonomous vehicles to recognize traffic signs in extreme weather conditions include:

[0187] The acquisition module is configured to acquire a traffic sign image to be identified under extreme weather conditions; the extreme weather conditions refer to rainy, snowy, or foggy weather.

[0188] The preprocessing module is configured to: perform rain and snow removal preprocessing on the traffic sign image to be identified for rainy or snowy weather; and perform fog removal preprocessing on the traffic sign image to be identified for foggy or hazy weather.

[0189] The recognition module is configured to: recognize the preprocessed image using a trained traffic sign recognition model to obtain the recognized traffic signs.

[0190] It should be understood that the acquisition module's task is to acquire image data from a camera or other sensors. Choosing appropriate sensors and cameras is crucial to ensuring high-quality image acquisition. When selecting a camera, factors such as resolution, frame rate, and field of view need to be considered.

[0191] Understandably, for rainy and snowy weather, the traffic sign image to be identified undergoes desnow / rain preprocessing. The MPR-Net multi-stage progressive image restoration network is used, which balances spatial detail and high-level contextual information during image restoration. Feature extraction is performed using an encoder-decoder approach, and information is preserved by fusing local information. Each layer employs pixel-adaptive adjustment, and local attention is adjusted through an attention supervision mechanism. The advantage of this approach is seamless information exchange between different layers. Therefore, when information is not passed sequentially from front to back, lateral connections are established between feature processing modules to prevent information loss. For hazy / foggy weather, the traffic sign image to be identified undergoes dehazing preprocessing. The Feature Fusion Attention Network FFA-Net is a deep learning network used in an end-to-end manner to restore haze-free images. In the Feature Fusion Attention Network (FFA-Net), the input is a foggy image. It first enters a shallow feature extractor, then enters N groups of cluster structures with multiple skip connections. These feature maps are then fused together with the attention module, transferring this feature to the learning structure of the reconstructed region and the global residual, thereby obtaining a fog-free output.

[0192] Understandably, the preprocessed image is then identified using a trained traffic sign recognition model. The specific process is as follows: First, the image to be detected is input into an improved YOLOv5 model. Next, the input image passes through a Backbone network consisting of convolutional and pooling layers to extract image features. Based on the features extracted by the Backbone network, a Neck network is introduced to further extract and optimize these features. Finally, a Head network is introduced onto the output feature map of the Neck network to detect target objects. The Head network contains multiple output layers, each outputting a feature map of a specific size. Each feature map is used to detect targets of different sizes.

[0193] It should be noted that the acquisition module, preprocessing module, and identification module described above correspond to steps S101 to S103 in Embodiment 1. The examples and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. It should also be noted that these modules, as part of the system, can be executed in a computer system, such as a set of computer-executable instructions.

[0194] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0195] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0196] Example 3

[0197] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in Embodiment 1.

[0198] Example 4

[0199] This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0200] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for autonomous vehicles to recognize traffic signs under extreme weather conditions, characterized by: include: Under extreme weather conditions, acquire traffic sign images to be identified; extreme weather refers to rainy, snowy, or foggy weather. For rainy or snowy weather, the traffic sign image to be identified is preprocessed to remove rain and snow; for foggy or hazy weather, the traffic sign image to be identified is preprocessed to remove fog. The preprocessed image is then used to identify the traffic sign using the trained traffic sign recognition model, resulting in the identified traffic sign. The trained traffic sign recognition model has the following network structure: The backbone layer, neck layer, attention layer, and prediction layer are connected in sequence. The backbone layer is used to extract features from the preprocessed image; the neck layer is responsible for multi-scale feature fusion of the feature map; the attention layer is used to focus on local information; and the prediction layer is used to perform the final regression prediction. The training process of the trained traffic sign recognition model includes: Construct a training set and a test set; both the training set and the test set include: a set of traffic sign images with known traffic sign recognition results; The training set is input into the traffic sign recognition model to train the model. When the total loss function value of the model no longer decreases, the training is stopped, and a preliminary traffic sign recognition model is obtained. The preliminary traffic sign recognition model is tested using a test set. When the test accuracy exceeds a set threshold, the current preliminary traffic sign recognition model is the trained traffic sign recognition model. The total loss function of the model includes: The total loss function L is: in, and These are the bounding box and the classification loss weights, It is a regression loss function. It is focal loss, a loss function used to solve the class imbalance problem; The regression loss function is: in, It is a distance loss. It's an angle loss. It is a metric used to evaluate the degree of overlap between predicted bounding boxes and ground truth bounding boxes in object detection and semantic segmentation tasks; It is the area of ​​the intersection of the predicted box and the ground truth box divided by the area of ​​their union.

2. The method for unmanned vehicles to recognize traffic signs under extreme weather conditions as described in claim 1, characterized in that, The backbone layer includes: a Focus convolutional neural network layer, a CBS1 layer, an SPD-Conv1 layer, a C2f1 layer, a CBS2 layer, an SPD-Conv2 layer, a C2f2 layer, a CBS3 layer, an SPD-Conv3 layer, a C2f3 layer, a CBS4 layer, an SPD-Conv4 layer, a C2f4 layer, and an SPPF spatial pyramid pooling feature layer, connected in sequence. The working process of the SPD-Conv1 layer includes: first, using a spatial-to-depth layer to perform adaptive convolution operation on the input feature map, and then using a non-convolutional stride layer to perform convolution operation on the output feature map. The C2f1 layer includes: The CBS5 layer, the Split network layer, the first Bottleneck residual block structure layer, the serial splicing layer L1, and the CBS6 layer are connected in sequence. The output of the first Bottleneck residual block structure layer is connected to the input of the second Bottleneck residual block structure layer. The output of the second Bottleneck residual block structure layer is connected to the input of the third Bottleneck residual block structure layer. The outputs of the second and third Bottleneck residual block structure layers are both connected to the input of the serial splicing layer L1; the output of the Split network layer is connected to the input of the serial splicing layer L1. The SPPF spatial pyramid pooling feature layer includes: The CBS7 layer, the first max pooling layer, the second max pooling layer, the third max pooling layer, the cascaded splicing layer L2, and the CBS8 layer are connected in sequence; the CBS7 layer serves as the input of the SPPF spatial pyramid pooling feature layer, and the CBS8 layer serves as the output of the SPPF spatial pyramid pooling feature layer. Among them, the output terminals of the CBS7 layer, the first max pooling layer, and the second max pooling layer are all connected to the input terminal of the serial splicing layer; The working process of the SPPF spatial pyramid pooling feature layer includes: dividing the input feature map into sub-maps of different sizes and performing max pooling on each sub-map to obtain a feature vector of a set size.

3. The method for unmanned vehicles to recognize traffic signs under extreme weather conditions as described in claim 1, characterized in that, The neck layer includes, from bottom to top, the CBS9 layer, the first upsampling layer, the cascaded splicing layer L3, the C2f5 layer, the CBS10 layer, the second upsampling layer, and the cascaded splicing layer L4; The neck layer further includes: C2f6 layer, CBS11 layer, series splicing layer L5, C2f7 layer, CBS12 layer, series splicing layer L6 and C2f7 layer connected sequentially from top to bottom; The input of the CBS9 layer is connected to the output of the SPPF spatial pyramid pooling feature layer; the input of the cascaded splicing layer L3 is connected to the output of the C2f3 layer; the input of the cascaded splicing layer L4 is connected to the output of the C2f2 layer; the output of the CBS9 layer is connected to the input of the cascaded splicing layer L6; the output of the CBS10 layer is connected to the input of the cascaded splicing layer L5; and the output of the cascaded splicing layer L4 is connected to the input of the C2f6 layer.

4. The method for unmanned vehicles to recognize traffic signs under extreme weather conditions as described in claim 1, characterized in that, The attention layer includes: three parallel Coordinate Attention modules: a first Coordinate Attention module, a second Coordinate Attention module, and a third Coordinate Attention module; The input of the first Coordinate Attention module is connected to the output of the C2f6 layer; The input of the second Coordinate Attention module is connected to the output of the C2f7 layer; The input of the third Coordinate Attention module is connected to the output of the C2f8 layer; The first Coordinate Attention module includes: The residual module has its input end serving as the input end of the first Coordinate Attention module; the output end of the residual module is connected to the input end of the weight update module. The output of the residual module is also connected to the input of the horizontal average pooling layer and the input of the vertical average pooling layer, respectively; the horizontal average pooling layer and the vertical average pooling layer are both implemented through the average pooling layer. The outputs of the horizontal and vertical average pooling layers are connected to the input of the cascaded splicing layer L7. The output of the cascaded splicing layer L8 is connected to the input of the 2D convolutional layer E1. The output of the 2D convolutional layer is connected to the input of the normalization layer. The output of the normalization layer is connected to the input of the nonlinear activation function layer. The output of the nonlinear activation function layer is connected to the inputs of the 2D convolutional layers E2 and E3, respectively. The output of the 2D convolutional layer E2 is connected to the input of the first activation function layer. The output of the 2D convolutional layer E3 is connected to the input of the second activation function layer. The outputs of both the first and second activation function layers are connected to the input of the weight update module. The output of the weight update module serves as the output of the first Coordinate Attention module.

5. The method for unmanned vehicles to recognize traffic signs under extreme weather conditions as described in claim 1, characterized in that, The prediction layer includes: three parallel first decoupled head detection head modules, second decoupled head detection head modules, and third decoupled head detection head modules; The input of the first Decoupled Head detection module is connected to the output of the first Coordinate Attention module; The input of the second decoupled head detection module is connected to the output of the second coordinate attention module; The input of the third Decoupled Head module is connected to the output of the third Coordinate Attention module; The first decoupled head detection module includes: a CBS13 layer; The input terminal of CBS13 layer serves as the input terminal of the first Decoupled Head dual decoupled detection head module; The output of CBS13 layer is connected to the input of CBS14 layer and the input of CBS15 layer, respectively. The output of the CBS14 layer is connected to the Cls class layer through the convolutional layer J1 and the third activation function layer in sequence; The output of CBS15 layer is connected to the input of convolutional layer J2 and the input of convolutional layer J3, respectively. The output of convolutional layer J2 is connected to the Reg coordinate layer; The output of convolutional layer J3 is connected to the fourth activation function layer, and the fourth activation function layer is connected to the Obj target layer. The Cls category layer refers to a vector of size C representing the probability that the predicted box belongs to each target category, where C is the number of target categories. For each predicted box, a cls category vector is predicted, representing the probability that the box belongs to each target category. The Reg coordinate layer refers to the position and size information of the predicted box. It is a vector of size 4, which represents the center coordinates, width and height of the predicted box. For each predicted box, a reg coordinate vector is predicted to represent the position and size information of the box. The Obj target layer refers to the confidence level that the predicted box contains a target. It is a scalar. For each predicted box, an obj target value is predicted, indicating whether the box contains a target. The prediction layer operates by: dividing the feature map into multiple grids and predicting a set of bounding boxes for each grid; for each bounding box, the prediction layer predicts its position, size, and a first confidence score, which indicates whether the bounding box contains a target; for each target category, the prediction layer predicts a second confidence score, which indicates whether the bounding box contains a target of the current category.

6. A system for autonomous vehicles to recognize traffic signs under extreme weather conditions, characterized by: include: The acquisition module is configured to acquire a traffic sign image to be identified under extreme weather conditions; the extreme weather conditions refer to rainy, snowy, or foggy weather. The preprocessing module is configured to: perform rain and snow removal preprocessing on the traffic sign image to be identified for rainy or snowy weather; and perform fog removal preprocessing on the traffic sign image to be identified for foggy or hazy weather. The recognition module is configured to: recognize the preprocessed image using a trained traffic sign recognition model to obtain the recognized traffic sign; The trained traffic sign recognition model has the following network structure: The backbone layer, neck layer, attention layer, and prediction layer are connected in sequence. The backbone layer is used to extract features from the preprocessed image; the neck layer is responsible for multi-scale feature fusion of the feature map; the attention layer is used to focus on local information; and the prediction layer is used to perform the final regression prediction. The training process of the trained traffic sign recognition model includes: Construct a training set and a test set; both the training set and the test set include: a set of traffic sign images with known traffic sign recognition results; The training set is input into the traffic sign recognition model to train the model. When the total loss function value of the model no longer decreases, the training is stopped, and a preliminary traffic sign recognition model is obtained. The preliminary traffic sign recognition model is tested using a test set. When the test accuracy exceeds a set threshold, the current preliminary traffic sign recognition model is the trained traffic sign recognition model. The total loss function of the model includes: The total loss function L is: in, and These are the bounding box and the classification loss weights, It is a regression loss function. It is focal loss, a loss function used to solve the class imbalance problem; The regression loss function is: in, It is a distance loss. It's an angle loss. It is a metric used to evaluate the degree of overlap between predicted bounding boxes and ground truth bounding boxes in object detection and semantic segmentation tasks; It is the area of ​​the intersection of the predicted box and the ground truth box divided by the area of ​​their union.

7. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-5.

8. A storage medium, characterized in that, The computer-readable instructions are stored non-transitory, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1-5 are executed.

Images