Traffic sign identification method and apparatus, and electronic device

Abstract

The invention discloses a traffic sign identification method and apparatus, and an electronic device. The method comprises the following steps: acquiring a traffic sign image in a traffic driving scene; performing super-resolution reconstruction processing on the traffic sign image to obtain a high-resolution image of the traffic sign image; a traffic sign recognition model is adopted to analyze the high-resolution image to obtain a recognition result corresponding to the traffic sign image, the traffic sign recognition model comprises a backbone network, a neck network and a head network, the backbone network is used for carrying out multi-level feature extraction on the high-resolution image through a target converter to obtain a multi-level feature map, and the head network is used for carrying out multi-level feature extraction on the high-resolution image; the neck network is used for fusing the multi-level feature maps through the feature pyramid network and the path aggregation network, and the head network is used for identifying the fused feature maps through a target loss function to obtain an identification result. According to the invention, the technical problems of high omission ratio, poor detection precision and slow identification speed caused by unclear traffic sign images are solved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and more specifically, to a method, device and electronic device for recognizing traffic signs. Background Art

[0002] With the rapid development of intelligent transportation systems and autonomous driving technology, accurate detection and recognition of traffic signs has become particularly important, which not only enhances driving safety, but also significantly improves traffic efficiency and helps reduce the incidence of traffic accidents. The intelligent systems of autonomous vehicles rely on deep learning models to recognize traffic signs. These models can recognize traffic signs even in bad weather, insufficient lighting or from complex viewing angles, thereby ensuring that vehicles can comply with traffic rules and drive safely.

[0003] However, despite the significant progress made in this field by deep learning, traffic sign recognition methods in related technologies still have some challenges and limitations, especially when the image resolution is low, visibility is poor, or the traffic signs are severely worn. For example, deep learning models require a large amount of labeled data for training, and the process of obtaining this data is time-consuming and costly. For uncommon traffic signs or those with a small number of samples, it may be difficult for the model to learn enough features, resulting in low recognition rates. When traffic signs are obscured, or shot from a variety of angles, or under extreme conditions that have never been seen (such as extreme weather, changes in lighting, or severely worn signs), the model's generalization ability is insufficient, resulting in reduced recognition accuracy. And when traffic rules change or new traffic signs are introduced, the deep learning models in related technologies cannot adapt immediately, and the model needs to be retrained or fine-tuned to effectively recognize the newly added traffic signs.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a method, device and electronic device for recognizing traffic signs, so as to at least solve the technical problems of high missed detection rate, poor detection accuracy and slow recognition speed caused by unclear traffic sign images.

[0006] According to one aspect of an embodiment of the present application, a method for recognizing traffic signs is provided, including: acquiring a traffic sign image in a traffic driving scene; performing super-resolution reconstruction processing on the traffic sign image to obtain a high-resolution image of the traffic sign image; using a traffic sign recognition model to analyze and process the high-resolution image to obtain a recognition result corresponding to the traffic sign image, wherein the traffic sign recognition model includes a backbone network, a neck network and a head network, the backbone network is used to perform multi-level feature extraction on the high-resolution image through a target transformer to obtain a multi-level feature map, the neck network is used to fuse the multi-level feature maps through a feature pyramid network and a path aggregation network to obtain a fused feature map, and the head network is used to perform position recognition on the fused feature map through a target loss function to obtain a recognition result.

[0007] Optionally, the traffic sign image is subjected to super-resolution reconstruction processing to obtain a high-resolution image of the traffic sign image, including: performing feature extraction on the traffic sign image through a convolutional layer in an image super-resolution reconstruction module to obtain a first feature map, wherein the first feature map is used to represent shallow feature information of the traffic sign image; performing feature extraction on the first feature map through a residual dense block in the image super-resolution reconstruction module to obtain a second feature map, wherein the residual dense block performs feature extraction and feature fusion on the first feature map through a dense connection layer and a residual learning mechanism, and the second feature map is used to represent deep feature information of the traffic sign image; and performing an upsampling operation on the second feature map to obtain a high-resolution image.

[0008] Optionally, an upsampling operation is performed on the second feature map to obtain a high-resolution image, including: performing an upsampling operation on the second feature map through a convolution layer, an activation function layer, and a sub-pixel convolution layer in an image super-resolution reconstruction module to obtain a first resolution image of the traffic sign image; enlarging the second feature map through a bicubic interpolation method to obtain a second resolution image of the traffic sign image, wherein the bicubic interpolation method is used to reduce a jagged effect of the second feature map during the enlargement process; and fusing the first resolution image and the second resolution image to obtain a high-resolution image.

[0009] Optionally, the backbone network is used to perform multi-level feature extraction on the high-resolution image through a target transformer to obtain a multi-level feature map, including: extracting features from the high-resolution image through the first module in the backbone network to obtain a third feature map, wherein the first module is used to perform dimensionality reduction and downsampling operations on the high-resolution image; extracting features from the third feature map through the second module in the backbone network to obtain a fourth feature map, wherein the second module is used to perform convolution, normalization and nonlinear activation processing on the third feature map; extracting features from the fourth feature map through the third module in the backbone network to obtain a multi-level feature map, wherein the third module includes a target transformer, and the third module is used to perform image segmentation and image fusion on the fourth feature map through the target transformer.

[0010] Optionally, the neck network is used to fuse multiple levels of feature maps through a feature pyramid network and a path aggregation network to obtain a fused feature map, including: performing feature fusion on multiple levels of feature maps through a feature pyramid network to obtain a fifth feature map, wherein the feature pyramid network fuses multiple levels of feature maps through a top-down path; performing feature fusion on multiple levels of feature maps through a path aggregation network to obtain a sixth feature map, wherein the path aggregation network fuses multiple levels of feature maps through a bottom-up path; fusing the fifth feature map and the sixth feature map to obtain a fused feature map.

[0011] Optionally, the target loss function is determined in the following manner: determining the prediction box and prediction category of the historical traffic sign image through a convolution operation, wherein the prediction box is used to represent the predicted position and size of the traffic sign in the historical traffic sign image, and the prediction category is used to represent the predicted traffic sign type to which the traffic sign in the historical traffic sign image belongs; determining the position information, category information and confidence of the historical traffic sign image through an initial loss function, the position information is used to represent the positioning loss between the prediction box and the label box, the category information is used to represent the classification loss between the prediction category and the label category, the confidence is used to represent the degree of matching between the prediction box and the label box, the label box is used to represent the actual position and size of the traffic sign in the historical traffic sign image, and the label category is used to represent the actual traffic sign type to which the traffic sign in the historical traffic sign image belongs; determining the target loss function based on the position information, category information and confidence.

[0012] Optionally, the traffic sign recognition model is trained in the following manner: dividing historical traffic sign images into a training set and a test set; using the training set to train the initial traffic sign recognition model, and when the total loss function value of the initial traffic sign recognition model no longer decreases, stopping the training to obtain a first traffic sign recognition model; using the test set to test the first traffic sign recognition model to obtain a test result, and when the test result exceeds a preset threshold, stopping the test to obtain a traffic sign recognition model.

[0013] According to another aspect of an embodiment of the present application, a traffic sign recognition device is also provided, including: an acquisition module, used to acquire a traffic sign image in a traffic driving scene; a reconstruction module, used to perform super-resolution reconstruction processing on the traffic sign image to obtain a high-resolution image of the traffic sign image; and a recognition module, used to analyze and process the high-resolution image using a traffic sign recognition model to obtain a recognition result corresponding to the traffic sign image, wherein the traffic sign recognition model includes a backbone network, a neck network and a head network, the backbone network is used to perform multi-level feature extraction on the high-resolution image through a target transformer to obtain a multi-level feature map, the neck network is used to fuse the multi-level feature maps through a feature pyramid network and a path aggregation network to obtain a fused feature map, and the head network is used to perform position recognition on the fused feature map through a target loss function to obtain a recognition result.

[0014] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned traffic sign recognition method.

[0015] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned traffic sign recognition method by running the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, which implement the above-mentioned traffic sign recognition method when executed by a processor.

[0017] In an embodiment of the present application, a traffic sign image in a traffic driving scene is acquired; a super-resolution reconstruction process is performed on the traffic sign image to obtain a high-resolution image of the traffic sign image; a traffic sign recognition model is used to analyze and process the high-resolution image to obtain a recognition result corresponding to the traffic sign image, wherein the traffic sign recognition model includes a backbone network, a neck network and a head network, the backbone network is used to perform multi-level feature extraction on the high-resolution image through a target transformer to obtain a multi-level feature map, the neck network is used to fuse the multi-level feature maps through a feature pyramid network and a path aggregation network to obtain a fused feature map, and the head network is used to perform position recognition on the fused feature map through a target loss function to obtain a recognition result, thereby achieving the purpose of significantly improving the detection accuracy and recognition speed of traffic signs in low-resolution images based on super-resolution image reconstruction technology and an improved RCS-YOLOv5 model, thereby achieving efficient detection of traffic signs in complex driving environments, especially under conditions of poor image quality, and maintaining a high detection performance. The technical effect, thereby solving the technical problems of high missed detection rate, poor detection accuracy and slow recognition speed caused by unclear traffic sign images. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a hardware structure diagram of a computer terminal for implementing a method for recognizing a traffic sign according to an embodiment of the present application;

[0020] Figure 2 is a flow chart of a method for identifying a traffic sign according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of a traffic sign recognition process according to an embodiment of the present application;

[0022] Figure 4 is a structural diagram of a CP-Transformer module according to an embodiment of the present application;

[0023] Figure 5 is a structural schematic diagram of a traffic sign model according to an embodiment of the present application;

[0024] Figure 6 It is a structural diagram of a traffic sign recognition device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] First, some nouns or terms that appear in the process of explaining the embodiments of the present application are subject to the following explanations:

[0028] CNN (Convolutional Neural Network): CNN is a deep learning model that is particularly suitable for processing data with grid structures, such as images. It learns the hierarchical feature representation of input data through a combination of convolutional layers, pooling layers, and fully connected layers.

[0029] CSP (Cross Stage Partial): The CSP module is a design for optimizing the computational efficiency and information flow of neural networks. By subdividing the intermediate stages of the network into multiple computational paths, the amount of parameters can be reduced while maintaining the diversity and depth of information.

[0030] CBL (Conv-BN-SiLU, Convolution-Batch Normalization-Linear Rectification Unit): CBL is a common combination unit in convolutional neural networks, used for feature extraction and normalization processing. Among them, Conv represents the convolution layer, which is responsible for feature extraction; BN represents batch normalization, which is used to accelerate training and improve the stability of the model; SiLU represents a variant of the Swish activation function, which is a nonlinear activation function that can increase the network's expressive power.

[0031] CA (Coordinate Attention): CA is an attention mechanism that independently learns attention weights in the width and height dimensions of the input feature map, thereby achieving selective amplification of important information in the feature map.

[0032] DPCNB (Dual-channel Pixel Channel Note Block): DPCNB is a special attention block that performs attention operations in both pixel and channel dimensions to enhance the model’s sensitivity to image details and its ability to distinguish features.

[0033] FPN (Feature Pyramid Network): FPN is a network structure used to generate multi-scale feature maps to improve the model's ability to detect objects of different sizes. It forms a multi-level feature representation by fusing feature maps of different depths, thereby enhancing the model's multi-scale perception.

[0034] BCE (Binary Cross-Entropy Loss): BCE is a loss function commonly used in binary classification problems. It is used to quantify the difference between the model's predicted probability and the actual label. It is the key to adjusting parameters in model training to minimize losses.

[0035] HCFWM (Hybrid Cavity Feeling Wild Module): HCFWM is a module designed for specific applications. It combines the concepts of dilated convolution and receptive field to expand the model's receptive field of the input image, thereby capturing more contextual information.

[0036] GAP (Global Average Pooling): GAP is a pooling operation used to reduce the size of the feature map to 1×1, thereby reducing the dimension to a vector of a fixed size. It is usually used at the tail of the network as a step before classification.

[0037] GMP (Global Max Pooling): GMP is similar to GAP, but uses maximum pooling instead of average pooling to extract the most prominent feature information in the feature map.

[0038] GIoU (Generalized IoU Loss): GIoU is an extension of IoU (Intersection over Union) and is used to calculate the degree of overlap between the predicted box and the true box in object detection. It improves the limitation of IoU that only considers the overlapping area, making the model more stable and accurate in box regression.

[0039] IoU (Intersection over Union): IoU is a commonly used evaluation metric in object detection, which is used to measure the degree of overlap between the predicted box and the true box. The closer the IoU value is to 1, the higher the degree of overlap and the more accurate the detection.

[0040] mAP (Mean Average Precision): mAP is an important indicator for evaluating the overall performance of the model in target detection. It comprehensively considers the average precision of targets of different categories and reflects the detection accuracy and recall rate of the model under different confidence thresholds.

[0041] PANet (Path Aggregation Networks): PANet supplements the top-down path of FPN by adding a bottom-up information flow path to strengthen the fusion of shallow and deep features, which helps improve the detection effect of small targets.

[0042] PAB (Pixelated Attention Block): PAB is an attention mechanism used to enhance the model's attention to image details at the pixel level, optimize the feature map by learning the importance of each pixel, and thus improve the detection performance.

[0043] RNN (Recurrent Neural Network): A neural network architecture for processing sequential data that handles the temporal order dependencies of input data by maintaining internal states.

[0044] SPP (Spatial Pyramid Pooling): SPP is used to generate a fixed-size output vector, which can maintain the performance of the model even if the size of the input image changes. It pools the feature maps at different scales and then concatenates the results to form the final output.

[0045] DNFPM (Densely Networked Features Pyramid Module): DNFPM is a module that enhances the FPN structure. It increases the connection density between feature maps to form a more complex feature representation, thereby enhancing the detection capability of the model.

[0046] RCS-YOLOv5: An improved model based on YOLOv5 for traffic sign detection, which aims to improve detection accuracy and model robustness by introducing super-resolution image reconstruction and special network structures (such as CP-Transformer modules).

[0047] SE (Squeeze-and-Excitation): SE is an attention mechanism used to improve the information expression ability of feature maps at the channel level. It compresses spatial information into channel weights through global average pooling, and then redistributes the weights through excitation operations to enhance key features and suppress irrelevant features.

[0048] CIoU_loss (Complete IoU Loss): CIoU_loss is an extended version of IoU loss. It also considers the center point distance between the predicted box and the true box, as well as the aspect ratio of the box while calculating IoU, providing a more comprehensive loss metric that helps with faster and more accurate box regression.

[0049] ResNet (Residual Network): ResNet is a deep convolutional neural network model. By introducing the residual learning framework, it solves the gradient vanishing problem in deep networks, allowing the network to be trained deeper, thereby improving model performance.

[0050] RDB (Residual Dense Block): RDB is a network module for image super-resolution reconstruction, which combines the residual learning of ResNet and the dense connection of DenseNet, allowing the network to learn the details and contextual information of the image more effectively.

[0051] In order to solve the problem of poor detection accuracy of deep learning models in related technologies, the present invention provides a method for identifying traffic signs, which can be run on Figure 1 In the computer terminal shown, the computer terminal is described below.

[0052] The traffic sign recognition method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a method for recognizing traffic signs. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0053] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0054] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the traffic sign recognition method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned traffic sign recognition method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0055] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0056] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0057] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0058] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a method for recognizing a traffic sign. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0059] Figure 2 is a flow chart of a method for identifying a traffic sign according to an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0060] Step S202: Acquire a traffic sign image in a traffic driving scene.

[0061] In the above step S202, it is first necessary to collect image data containing traffic signs from the traffic driving environment. This is usually done by a road monitoring camera or a front camera of a vehicle. The collected images may have different qualities due to various factors, such as weather conditions, light, distance and camera resolution.

[0062] Step S204: perform super-resolution reconstruction processing on the traffic sign image to obtain a high-resolution image of the traffic sign image.

[0063] In the above step S204, after the traffic sign image is collected, the image can be preprocessed using super-resolution reconstruction technology. This process aims to convert a low-resolution image into a high-resolution image, thereby improving the visibility and clarity of details in the image. Specifically, super-resolution reconstruction can be achieved through three steps: shallow feature extraction, deep feature extraction (using residual dense blocks RDB) and upsampling. Among them, the shallow feature extraction part uses the convolution layer to extract low-level feature information; the deep feature extraction uses the newly designed RDB to extract the deep features of the image, and at the same time fuses the feature maps at different scales; upsampling uses the convolution layer, the activation function layer and the sub-pixel convolution layer to upsample the feature map, and then uses the convolution layer and the activation function layer to reconstruct the super-resolution image, and then adds the feature map amplified by the bicubic interpolation method to the reconstructed super-resolution feature map to generate the final output image.

[0064] The generator network adopts a deep residual network (ResNet) structure. Its main purpose is to recover a high-resolution image from a low-resolution input image. It is mainly composed of an input layer, a pre-sampling layer, a residual module, a post-sampling layer, and an output layer. The input layer inputs the low-resolution image, and the pre-sampling layer enlarges the image size to the target size through nearest neighbor interpolation to prepare for subsequent processing; the residual module contains multiple residual blocks, each of which includes two convolutional layers and jump connections. The residual module is used to learn the feature information of the image and enhance the feature details; the post-sampling layer further enlarges the image size through a sub-pixel convolutional layer; and the output layer outputs the final high-resolution image.

[0065] Step S206, using the traffic sign recognition model to analyze and process the high-resolution image to obtain a recognition result corresponding to the traffic sign image, wherein the traffic sign recognition model includes a backbone network, a neck network and a head network, the backbone network is used to perform multi-level feature extraction on the high-resolution image through a target transformer to obtain a multi-level feature map, the neck network is used to fuse the multi-level feature map through a feature pyramid network and a path aggregation network to obtain a fused feature map, and the head network is used to perform position recognition on the fused feature map through a target loss function to obtain a recognition result.

[0066] In the above step S206, the traffic sign recognition model (RCS-YOLOv5 model) is used to analyze and process the high-resolution image after super-resolution reconstruction. The RCS-YOLOv5 model includes three main parts: the backbone network, the neck network and the head network. The details are as follows:

[0067] The backbone network is responsible for multi-level feature extraction of high-resolution images to capture the detailed features of traffic signs. It should be noted that the backbone network integrates the CP-Transformer (i.e. the target transformer mentioned above) module to enhance the extraction of traffic sign information. Through parallel processing and global feature extraction, the information dimension of the feature map is improved, avoiding the interference of background information during target detection.

[0068] Specifically, in the backbone network, due to the complex traffic driving scene, traffic signs occupy a small proportion of the screen. In the process of downsampling, the multiple iterations of the background will cause a large amount of redundant information. It is easy to be affected by noise when extracting features, which interferes with target detection and reduces the accuracy. Therefore, a parallel Transformer module (CP-Transformer) is added to the backbone network to enhance the feature extraction network's ability to extract information from traffic signs. Relying on its excellent global feature extraction ability, it obtains the global distribution characteristics of the target in the feature map and improves the dimension of feature information. Secondly, directly integrating Transformer into YOLOv5 will cause a surge in the number of calculation parameters, so that the model is too large, and the feature map will destroy the integrity of the feature when passing through the Transformer module. The CP-Transformer can first group the feature map to reduce the tokens parameter, and then connect the Transformer encoder in parallel to improve the reuse rate of the feature. Finally, the grouped feature map is divided to obtain feature blocks of different sizes to avoid the probability of the complete target being segmented.

[0069] The neck network mainly fuses the multi-level feature maps output by the backbone network through the feature pyramid network (FPN) and the path aggregation network (PANet) to generate higher quality feature maps. FPN and PANet can process multi-scale features, which is crucial for traffic sign detection of different sizes, especially for small target detection, which can provide rich semantic information and enhance detection capabilities.

[0070] Specifically, in the neck network, the main function is to fuse the feature maps of different levels extracted by the backbone network so as to detect targets of different scales more efficiently. In order to solve the problem of inaccurate feature recognition of small targets such as traffic signs in the feature extraction and feature fusion stages, we can learn from the idea of ​​feature FPN and combine the characteristics of path PANet to form an efficient feature fusion mechanism. Among them, FPN can extract features of image features at each scale and generate multi-scale feature representations. The feature map of each scale also has strong semantic information. The shallow feature map has rich semantics. The main function of FPN is to obtain feature maps from different stages of the backbone network and fuse these feature maps together by upsampling and element-by-element addition to obtain a series of feature maps of different resolutions. Each feature map corresponds to a different scale, thereby helping the model to detect targets of different sizes. PANet introduces a bottom-up path to supplement the information flow in FPN. This structure allows information to flow not only from high-level to low-level (the so-called "top-down" path), but also from low-level to high-level (the "bottom-up" path), which helps to enhance the information content of shallow features and enable small targets to be better detected.

[0071] The head network is mainly responsible for predicting the target position and category on the fused feature map. The CIoU_loss loss function is used. This target loss function improves the calculation method of the position loss and adds the aspect ratio and center point distance loss of the target box, making the model more accurate and stable in position regression. In addition, the SE attention mechanism is introduced to weight the channels to suppress background noise, highlight the key features of traffic signs, and further improve detection accuracy.

[0072] Specifically, in the head network, due to the complexity of traffic driving scenes, traffic signs occupy a small proportion of the screen. In the process of downsampling, the multiple iterations of the background will cause a large amount of redundant information, which is easily affected by noise when extracting features, resulting in reduced accuracy and difficulty in detecting traffic signs at different scales. Therefore, the SE attention mechanism is added to the head network to enhance the precise positioning of traffic signs. Secondly, the target loss function of RCS-YOLOv5 includes three categories: confidence, category, and position. RCS-YOLOv5 uses CIoU_loss to predict the position loss of the detected target. Since CIoU has the characteristics of making the target box regression more stable, faster convergence, and higher accuracy, CIoU can be used to effectively optimize the loss function.

[0073] Through the above steps S202 to S206, the purpose of significantly improving the detection accuracy and recognition speed of traffic signs in low-resolution images based on super-resolution image reconstruction technology and the improved RCS-YOLOv5 model is achieved, thereby achieving efficient detection of traffic signs in complex driving environments, especially under conditions of poor image quality, and maintaining a high detection performance. The technical problem of high missed detection rate, poor detection accuracy and slow recognition speed caused by unclear traffic sign images is solved.

[0074] Figure 3 is a schematic diagram of a traffic sign recognition process according to an embodiment of the present application, such as Figure 3 As shown, the recognition process first receives a traffic scene image (i.e., the traffic sign image mentioned above), improves the image quality through an image super-resolution reconstruction module, obtains a super-resolution image (i.e., the high-resolution image mentioned above), and then uses the RCS-YOLOv5 model to detect and recognize traffic signs. Among them, the RCS-YOLOv5 model uses a scene dataset (such as historical traffic sign images) during the training phase. The dataset has been annotated and contains the location and category information of traffic signs. After the model training is completed, it can accurately locate and identify traffic signs in super-resolution images and output the results. The entire process involves three main stages: image preprocessing, model training, and detection and recognition, which ensures the accuracy and efficiency of traffic sign detection. The following is combined with Figure 2 and Figure 3 The above steps are explained in detail.

[0075] In the above step S204, the traffic sign image is subjected to super-resolution reconstruction processing to obtain a high-resolution image of the traffic sign image, including: extracting features of the traffic sign image through a convolutional layer in an image super-resolution reconstruction module to obtain a first feature map, wherein the first feature map is used to represent shallow feature information of the traffic sign image; extracting features of the first feature map through a residual dense block in the image super-resolution reconstruction module to obtain a second feature map, wherein the residual dense block extracts and fuses features of the first feature map through a dense connection layer and a residual learning mechanism, and the second feature map is used to represent deep feature information of the traffic sign image; and performing an upsampling operation on the second feature map to obtain a high-resolution image.

[0076] Optionally, an upsampling operation is performed on the second feature map to obtain a high-resolution image, including: performing an upsampling operation on the second feature map through a convolution layer, an activation function layer, and a sub-pixel convolution layer in an image super-resolution reconstruction module to obtain a first resolution image of the traffic sign image; enlarging the second feature map through a bicubic interpolation method to obtain a second resolution image of the traffic sign image, wherein the bicubic interpolation method is used to reduce a jagged effect of the second feature map during the enlargement process; and fusing the first resolution image and the second resolution image to obtain a high-resolution image.

[0077] In the embodiment of the present application, by performing the above-mentioned super-resolution reconstruction step on the traffic sign image, the quality of the image can be effectively improved, especially when encountering low-resolution or blurred traffic sign images, the details of the image can be significantly enhanced, and the input quality of the traffic sign detection model can be improved, thereby improving the accuracy and efficiency of detection. The convolution layer extracts shallow features, the RDB module deeply extracts and fuses features, and then uses upsampling operations to improve the image resolution. This series of operations work together to make the final high-resolution image an ideal input for traffic sign detection, significantly enhancing the overall performance of the system. The specific process can be as follows:

[0078] First, the traffic sign image is processed by a 3×3 convolution layer in the image super-resolution reconstruction module to generate the first feature map. The convolution layer here is responsible for extracting shallow features of the image, such as edges, textures, etc. These features are the basic components of the image and provide raw information for subsequent feature processing. Since traffic sign images may be affected by many factors during the acquisition process (such as lighting, weather, resolution, etc.), the features extracted by the convolution layer can capture these basic details and lay the foundation for subsequent deep feature extraction.

[0079] Subsequently, the first feature map enters m RDBs for in-depth feature extraction and fusion to generate the second feature map. The RDB part consists of three parts: dense connection layer, local feature fusion, and local residual learning. The dense connection mechanism enables the state information of the previous RDB to be passed to the next RDB through skip connections. m-1 and W m Represents the input and output of the mth RDB. The output of the nth convolution of the mth RDB is expressed as:

[0080] W m,n =σ(L m,n [w m-1 ,W m,1 ,...,W m,n-1 ])

[0081] Among them, W m,nrepresents the output of n convolutional layers in RDB, σ represents the ReLU activation function, L m,n represents the weight of the nth convolutional layer, [w m-1 ,W m,1 ,...,W m,n-1 ] represents the concatenation of the output feature of the m-1th RDB and the output feature map of the mth convolutional layer from the 1st to the n-1th layer. The feature map is fused by local feature fusion and adaptive method, so as to ensure that the output map of each RDB is fixed in the C0 dimension. It is implemented by 1×1 convolution through the following formula:

[0082]

[0083] Among them, W m,LF It represents the output of the mth RDB after 1×1 convolution. represents the mth RDB to achieve 1×1 convolution, W m,N It represents the output of the last convolutional layer in the RDB dense connection layer. Finally, the output features are optimized using local residual learning. The output of the mth RDB is expressed as:

[0084] W m =W m-1 +W m,LF

[0085] The number of channels obtained after extraction from the shallow feature extraction part is 64, and then three RDB modules, depthwise separable convolution and ReLU activation function constitute the RDB module. The depthwise separable convolution performs independent depthwise convolution on each input channel and performs channel weighting to achieve feature fusion. The depthwise convolution operation is to equip each input channel with a convolution kernel, and then splice the convolution kernels of all channels according to the channel dimension to form the final output. The specific operation is:

[0086]

[0087] Among them, Y i,j,k represents the output feature map of the deep convolution, X i+m,j+n,k represents the input feature map of the deep convolution, K m,n,k Represents the convolution kernel.

[0088] Then, the feature fusion of the deep convolution feature map is performed. The specific operation is as follows:

[0089]

[0090] Among them, Z i,j,c represents the fused feature map, P k,cIt represents the point-by-point convolution kernel, and the description of the depth-separable convolution is as follows:

[0091]

[0092] The final output expression of the RDB module is as follows:

[0093] r l =Add(Conv(Concat(SRDB1,SRDB2,SRDB3)),r l-1 )

[0094] Among them, r l Represents the output of the current RDB, that is, the second feature map mentioned above, r l-1 It represents the output of the previous RDB module, SRDB1, SRDB2, and SRDB3 represent the feature map output of the residual block, Add represents the point-by-point addition operation, Conv represents the convolution operation, and Concat represents the cascade operation.

[0095] Finally, the second feature map is converted into a high-resolution image through an upsampling operation. Upsampling usually involves using convolutional layers, activation function layers, and sub-pixel convolutional layers to increase the resolution of the image so that it contains more pixel information. This process is implemented through the post-sampling layer of the generator network, using the sub-pixel convolutional layer to enlarge the feature map, and the output layer generates the final first-resolution image. The specific process can be as follows:

[0096] First, the second feature map after feature fusion is sent to a 3×3 convolution layer and an LReLU activation function layer, and then the feature map is upsampled using a sub-pixel convolution layer, and then a 3×3 convolution layer and an LReLU activation function layer are used to obtain the first resolution image.

[0097] Secondly, the second feature map is enlarged by bicubic interpolation to obtain a second resolution image. Bicubic interpolation is a high-quality image enlargement technology that can provide smooth transition effects and good detail preservation. It is suitable for further improving image resolution after preliminary upsampling to generate clearer and more detailed images.

[0098] Finally, the preliminary high-resolution feature map is fused with the second-resolution image obtained by bicubic interpolation to obtain the final high-resolution image. The fusion step usually involves adding or concat two images or feature maps of different resolutions to generate a high-resolution image that combines the advantages of both processing methods. Specifically, in RCS-YOLOv5, the preliminary high-resolution feature map obtained by the sub-pixel convolution layer is added to the feature map enlarged by the bicubic interpolation method to generate the final output image. This step makes full use of the optimization of feature extraction by the sub-pixel convolution layer and the advantages of bicubic interpolation in image quality to obtain the best super-resolution reconstruction effect.

[0099] It should be noted that the first loss term used in this application is SSIM loss, which uses the SSIM index to measure whether images are similar. For two given images x and y, SSIM can be defined as:

[0100]

[0101] Among them, μ x , μ y Represent the high-resolution image reconstructed by resolution and the original image grayscale mean, Represent the grayscale variance between the high-resolution image and the original image, σ xy represents the grayscale covariance between the high-resolution image and the original image, C1 and C2 are constants, and the SSIM loss term is defined as:

[0102] L SSIM =1-SSIM(x SR ,x HR )

[0103] in, x SR Representing the above high-resolution image, x HR is the original image.

[0104] In the above step S204, the backbone network is used to perform multi-level feature extraction on the high-resolution image through the target transformer to obtain a multi-level feature map, including: extracting features from the high-resolution image through the first module in the backbone network to obtain a third feature map, wherein the first module is used to perform dimensionality reduction and downsampling operations on the high-resolution image; extracting features from the third feature map through the second module in the backbone network to obtain a fourth feature map, wherein the second module is used to perform convolution, normalization and nonlinear activation processing on the third feature map; extracting features from the fourth feature map through the third module in the backbone network to obtain a multi-level feature map, wherein the third module includes a target transformer, and the third module is used to perform image segmentation and image fusion on the fourth feature map through the target transformer.

[0105] In the embodiment of the present application, the backbone network performs multi-level feature extraction on the high-resolution image, which includes the processing stages of three key modules (Focus module, CBL module, CP-Transformer module). The specific process can be as follows:

[0106] First, at the input end, the Mosaic algorithm is used to enhance the data of high-resolution images. Photos are selected from the data set, and the data is scaled, reversed, and the blank parts are filled with color. The pictures are combined into one photo from top to bottom and from left to right and sent to the neural network. The input image size is changed from 608×608 to 640×640 to increase the image resolution information.

[0107] Secondly, the high-resolution image after data enhancement is sent to the Focus module (the first module mentioned above) in the backbone network for processing, whose main function is dimensionality reduction and downsampling to reduce the amount of calculation and improve processing efficiency. Among them, dimensionality reduction processing is usually implemented through a convolutional layer, which takes values ​​for every other pixel in the input image and reduces the high-dimensional feature map to a low-dimensional feature map; the downsampling operation can further reduce the spatial dimension of the image, improve computing efficiency by reducing the number of pixels, and at the same time maintain the key information of the image.

[0108] Therefore, the third feature map output by the first module is a feature representation after dimensionality reduction and downsampling. It can not only reduce the original high-dimensional feature map to a low-dimensional feature map, achieving two-fold downsampling while ensuring no information loss, but also increase the number of channels to four times the original, improving the performance of feature extraction, thereby reducing the consumption of computing resources and adapting to the input requirements of subsequent processing modules.

[0109] Furthermore, after the third feature map enters the second module, a fourth feature map is generated through a series of processing processes. The second module consists of three CSP_CBL modules, including a convolutional layer (used to extract features), a batch normalization layer (used to standardize data distribution and improve the training stability and speed of the model) and a SiLU activation function (a nonlinear activation function that can introduce nonlinear changes and enhance the expressiveness of the model). The convolutional layer is responsible for extracting more complex and abstract features from the third feature map; the batch normalization layer normalizes the output of the convolutional layer to make the distribution of the feature map more stable, which helps to accelerate model convergence; the SiLU activation function performs nonlinear processing on the normalized feature map, stimulates and filters out important features, suppresses irrelevant information, and generates a fourth feature map, providing a more optimized feature representation for subsequent deep processing.

[0110] Finally, the fourth feature map is fed into the third module, which contains the target transformer (CP-Transformer) for image segmentation and feature fusion. The target transformer is designed to enhance the model's ability to extract global features of traffic signs. By processing and grouping feature maps in parallel, the number of tokens parameters is reduced and the reuse rate of features is improved. During processing, the feature map is divided into feature blocks of different sizes to avoid the integrity of the target being destroyed and to extract more detailed global features. Through the CP-Transformer module, multi-scale features are fused, and the model can obtain the global distribution characteristics of the target in the feature map and enrich the dimension of feature information, which is crucial for the accurate detection of traffic signs. In complex traffic scenes, traffic signs may be small or partially occluded. Through the processing of CP-Transformer, the model can better identify and locate these signs, thereby improving the accuracy and efficiency of detection.

[0111] Figure 4 is a schematic diagram of the structure of a CP-Transformer module according to an embodiment of the present application, such as Figure 4 As shown in the figure, the fourth feature map X is first decomposed into multiple feature blocks, which are then fed into three parallel multi-head attention calculation units through parallel assignment operations. Each unit contains a feedforward layer for processing the results of the attention calculation. These parallel multi-head attention calculation units can simultaneously capture the correlation of input features in different subspaces. Furthermore, after completing the multi-head attention calculation, the outputs of each unit are spliced ​​together through the Concat layer to form a new feature representation. This spliced ​​feature is further processed through a feedforward layer, and the final output result is X', which is the above-mentioned multi-level feature map. This structure allows the model to learn different feature representations in different attention heads, enhancing the model's understanding and processing capabilities of input data.

[0112] Specifically, we first need to convert the fourth feature map output by CSP_CBL in the backbone network into It is sent to the parallel Transformer module, and the fourth feature map is disassembled according to the number of channels C to obtain three branch feature maps x1, x2, and x3, where the parameters of each feature map are Then set different block sizes to segment the feature sub-graphs. For x1, For x2, For x3, three feature subgraphs are segmented using three different sizes to avoid target integrity damage and extract more detailed global features. The three feature subgraphs convert the segmented features into tokens. After the feature graph x1 completes the multi-head attention calculation, the parameters are obtained again from the output of the feedforward neural network layer: The feature map x1' is fused with x2 through the add superposition operation. Similarly, the feature map x3 is fused with x2' through the add superposition operation and then multi-head attention calculation is performed. In the pre-processed feature block position encoding link, the relative position encoding form is used, and the context encoding is used to associate the position encoding with the query vector, the key value vector and the value vector. i With other tokenx j When performing self-attention calculation, the calculation formula is as follows:

[0113]

[0114] Among them, Att i represents the self-attention value of the i-th token; n represents the total number of tokens; x i and x j Represents the feature vector of the i-th and j-th token; W Q , W K , W V Represents the weight matrices of query vector, key vector and value vector respectively, d represents the scaling factor, r ij represents the relative position, b ij It represents the relative position offset term, and its calculation formula is as follows:

[0115] b ij =(x i W Q )(r ij ) T +(x i W K )(r ij ) T

[0116] Furthermore, for the relative position calculation between feature blocks, the segment distance calculation formula can be used to represent the distance position relationship through different weights, as follows:

[0117]

[0118] Among them, x represents the distance between two feature blocks, α represents the distance judgment parameter, β represents the upper limit of the distance, γ is the harmonic number, and the relative position r between the two feature blocks is ij The calculation formula is:

[0119]

[0120] in, Indicates that the i-th and j-th feature blocks are in The relative position in the dimension, Indicates that the i-th and j-th feature blocks are in The relative position in one dimension.

[0121]

[0122] in, and Respectively represent the position of the i-th feature block in the x and y directions; g(·) represents a piecewise function, which is used to calculate the relative position between feature blocks; respectively represent the positions in the x and y directions, and They respectively represent the relative positions between the ith and jth feature blocks after being processed by the piecewise function g(·).

[0123] Finally, the three global attention sub-feature maps x1', x2', and x3' are concat fused to obtain the same feature map X'∈R as the input. H×W×C , X′ covers the long-distance dependency features of multi-scale feature blocks in the global features, and has rich feature levels.

[0124] Optionally, the neck network is used to fuse multiple levels of feature maps through a feature pyramid network and a path aggregation network to obtain a fused feature map, including: performing feature fusion on multiple levels of feature maps through a feature pyramid network to obtain a fifth feature map, wherein the feature pyramid network fuses multiple levels of feature maps through a top-down path; performing feature fusion on multiple levels of feature maps through a path aggregation network to obtain a sixth feature map, wherein the path aggregation network fuses multiple levels of feature maps through a bottom-up path; fusing the fifth feature map and the sixth feature map to obtain a fused feature map.

[0125] In the embodiment of the present application, for the detection of traffic signs, the neck network plays a key role in fusing multi-level feature maps, which helps the model recognize traffic signs of different scales. The specific process can be as follows:

[0126] First, the FPN module uses a top-down path to fuse multi-level feature maps to generate the fifth feature map. In this process, the backbone network outputs multi-level feature maps, and each layer of the feature map represents information of different scales. In order to unify this information, FPN uses upsampling and element-by-element addition to fuse high-level feature maps (smaller in size and rich in semantic information) with low-level feature maps (larger in size and more detailed information). Specifically, the high-level feature map is aligned with the low-level feature map through an upsampling operation, and then element-by-element addition is performed. In this way, a series of feature maps of different resolutions can be obtained, each of which fuses information at different levels, thereby enhancing the model's ability to detect traffic signs of various scales. The fifth feature map is generated through this process, which contains the fusion results of multi-level information from high to low layers.

[0127] At the same time, the PANet module fuses multiple levels of feature maps through a bottom-up path to obtain the sixth feature map. Unlike FPN, PANet allows information to flow not only from high-level to low-level, but also from low-level to high-level. This two-way information flow can further enhance the amount of information in shallow features. Especially in traffic sign detection, the detection of small targets often relies on the rich information of shallow features, so the bottom-up path of PANet is particularly helpful in enhancing this information, so that small targets can also be detected more accurately. The sixth feature map reflects the other side of the multi-level information contained in the feature map after fusion through the bottom-up path of PANet.

[0128] Finally, the neck network fuses the fifth feature map and the sixth feature map to generate a fused feature map. This fusion can be a simple element-by-element addition or a more complex operation, such as weighted fusion, concatenation, etc. The purpose is to integrate the strengths of the top-down and bottom-up paths, unify the feature information of different levels and scales into a feature map, and form a comprehensive representation containing rich semantics and detail information. The fused feature map will be used as the input of the head network for the final traffic sign detection and recognition.

[0129] For example, the neck network densely connects the original FPN network, marks the feature maps in the backbone network as {B1, B2, B3, B4, B5}, and the sizes of the feature maps are {320×320, 160×160, 80×80, 40×40, 20×20}. B'5 represents the feature map of B5 after processing, and uses a 1×1 convolution layer to transform the above feature map into {B'2, B'3, B'4, B"5} according to the number of channels. Since B"5 is No other networks are fused, so B″5 is equal to F5, and F5 is superimposed with B′4 by two times upsampling to obtain F3; the feature map obtained by two times upsampling of F3, the feature map obtained by four times upsampling of F4, and the feature map obtained by eight times upsampling of F5 are superimposed to obtain F2. Improvements are made on the original FPN network, and shallow features are densely connected with high-level information with rich semantic information, so that compared with the original features, the existing feature map will have accurate and rich semantic information.

[0130] Through the above process, the neck network effectively fuses the multi-level feature maps output by the backbone network to generate a feature representation containing multi-level information. This fusion strategy not only improves the model's detection performance for traffic signs of different scales, but also enhances the detection capability for small targets, making up for the limitations of single-scale feature maps in small target detection. Through the combined use of FPN and PANet, the present invention can more comprehensively utilize the feature information of the image, providing strong support for the accurate detection of traffic signs.

[0131] Optionally, in the head network, the target loss function is determined in the following manner: determining the prediction box and prediction category of the historical traffic sign image through a convolution operation, wherein the prediction box is used to indicate the predicted position and size of the traffic sign in the historical traffic sign image, and the prediction category is used to indicate the predicted traffic sign type to which the traffic sign in the historical traffic sign image belongs; determining the position information, category information and confidence of the historical traffic sign image through an initial loss function, the position information is used to indicate the positioning loss between the prediction box and the label box, the category information is used to indicate the classification loss between the prediction category and the label category, the confidence is used to indicate the degree of matching between the prediction box and the label box, the label box is used to indicate the actual position and size of the traffic sign in the historical traffic sign image, and the label category is used to indicate the actual traffic sign type to which the traffic sign in the historical traffic sign image belongs; determining the target loss function based on the position information, category information and confidence.

[0132] In the embodiment of the present application, in order to detect traffic signs of different scales more accurately, the SE attention mechanism is added before the detection head in the head network. SE weights different channels, automatically obtains the weight of each feature channel through learning, and uses the weight to screen key features. The SE module is mainly composed of a compression module and an excitation module. The details are as follows:

[0133] Suppose the input convolution feature map is X = [x 1 ,x 2 ,…,x C′ ],X∈R H′×W′×C′ , where H′, W′ and C′ represent the length, width and number of feature channels of the input feature map respectively. The convolution kernel is defined as: V = [v l ,v2,…,v c ], the Cth convolution kernel is The output after convolution is Y = [y1, y2, ..., y c ],y∈R H×W×C , the output calculation formula is:

[0134]

[0135] Among them, y c represents the cth channel of the output feature map after convolution, is the convolution operation, x s is the sth input, is the c-th two-dimensional convolution kernel.

[0136] The compression module converts a single two-dimensional feature channel into a real number, turning it into a global feature to expand the receptive field, and outputs a one-dimensional matrix Z = [z l ,z2,…,z c ],Z∈R C , where z c It is expressed as:

[0137]

[0138] Among them, z c represents the global average value of the cth channel, F sq (y c ) represents the global average pooling operation on the feature map, H is the height of a single feature channel map, W is the width of a single feature channel map, and y c (i, j) is the value of each point in the feature map channel. By calculating the average value of each feature map, a real number is obtained and finally a one-dimensional vector is output. Global average pooling reduces the number of parameters and avoids overfitting.

[0139] After the compression module is operated, the weight of each channel needs to be calculated. Let Z be the one-dimensional matrix vector obtained after compression, W l ,W2 are two fully connected layers, and The calculation formula for the excitation module operation is:

[0140] s=F ex (z,W)=σ(g(z,W))=σ(W2δ(W1z))

[0141] Among them, F ex (z,W) represents the calculation process of the excitation module, s is the weight of each channel, r is the scaling parameter, δ is the activation function ReLU, and σ is the Sigmoid activation function. The excitation module is conducive to extracting key features, suppressing invalid features, and improving the running speed. The excitation module operation obtains the weight of each channel and weights the original output. The calculation expression is as follows:

[0142]

[0143] Among them, F scale (y c ,s c ) represents the operation of weighting the original output features, s c represents the weight value of the cth channel. is the output feature of each channel after weighting.

[0144] At the same time, the determination of the target loss function is the core of the training process, which ensures that the model can simultaneously optimize positioning, classification, and confidence prediction, providing more reliable and accurate traffic sign detection capabilities for autonomous driving and intelligent transportation systems. The specific process can be as follows:

[0145] First, during the training phase, the model analyzes and processes historical traffic sign images through convolution operations to determine the predicted box positions and sizes of traffic signs in the images, as well as their possible predicted categories. The predicted box is the model's prediction of the position and size of the traffic sign, usually expressed as the coordinates of the bounding box, such as the upper left and lower right corner coordinates or the center point coordinates and width and height. The predicted category represents the model's prediction of the type of traffic sign, which includes but is not limited to speed limit signs, direction signs, warning signs, etc. These prediction information are the model's preliminary judgment based on the features of the input image, providing a basis for subsequent loss calculations.

[0146] Among them, the initial loss function is used to evaluate the difference between the model prediction and the true label, including three components: position information, category information, and confidence.

[0147] Position information: The accuracy of positioning is measured by calculating the intersection over union (IoU) between the predicted box and the true label box or its improved version CIoU. CIoU not only takes into account IoU, but also takes into account the distance between the center point of the predicted box and the true box and the consistency of the aspect ratio, which makes the calculation of positioning loss more comprehensive and helps improve the accuracy of the predicted box.

[0148] Classification information: The classification loss is calculated by comparing the difference between the predicted class and the true class (label class). This is usually done using the cross entropy loss function, which measures the distance between the predicted probability distribution and the true class label. For a binary classification problem, a binary cross entropy loss may be used.

[0149] Confidence: Confidence reflects the degree of match between the predicted box and the label box, which is usually calculated by IoU. The confidence loss aims to ensure that the model not only predicts the correct category, but also accurately locates the traffic sign. The calculation of confidence is based on the evaluation of the overlap between the predicted box and the true box.

[0150] The target loss function is a combination of the three losses (position information, category information, and confidence), and is usually calculated by weighted summation. In the RCS-YOLOv5 model, CIoU_loss is used as the position loss function because it makes the target box regression more stable, converges faster, and has higher accuracy. Category loss functions (such as BCE) and confidence losses (such as GIOU or DIoU) are also taken into account to ensure that the model can achieve high accuracy in identifying the type and position of traffic signs. Different loss terms may have different weight coefficients, which reflect the importance of different types of errors during training. For example, if detection accuracy is the primary goal, the weight of position loss may be higher.

[0151] Specifically, the loss function in YOLOv5 is divided into three parts: the first is cls_loss, which calculates the error between the predicted classification result and the true classification result; the second is the positioning loss box_loss, which calculates the error between the predicted box and the label box; the third is the confidence loss obj_loss. The overall loss function is the result of the weighted sum of the three.

[0152] IoU calculates the ratio of the area of ​​the intersection of the prediction box and the annotation box to the area of ​​the union. IoU close to 1 means higher overlap, and IoU close to 0 means lower overlap. DIoU adds the distance between the center of the prediction box and the annotation box to calculate the distance. The calculation formula is as follows:

[0153]

[0154] Among them, ρ 2 represents the square of the Euclidean distance between the prediction box and the label box, b,bgt Represents the center coordinates between the prediction box and the label box, and d represents the diagonal length of the minimum enclosing rectangle.

[0155] In the embodiment of the present application, CIoU_loss can be used as the loss function, and the aspect ratio loss of the prediction box is added on the basis of the previous one. The calculation formula is as follows:

[0156]

[0157] Among them, α represents the weight coefficient, v represents the aspect ratio match between the predicted box and the label box, and w l 、h l 、w p 、h p Represents the width and height of the prediction box and label box respectively, (x l1 ,y l1 )、(x l2 ,y l2 ) and (x p1 ,y p1 )、(x p2 ,y p2 ) represent the coordinates of the upper left corner and lower right corner of the prediction box and label box respectively,

[0158]

[0159] Among them, m gt and n gt They represent the width and height of the label box, respectively, and m and n represent the width and height of the prediction box, respectively.

[0160] Therefore, the improved objective loss function is as follows:

[0161]

[0162] By determining the target loss function, the model can obtain a feedback signal that combines positioning, classification, and confidence errors, which is used to guide the adjustment of model parameters during the back-propagation process to minimize the value of the loss function. In each training cycle, the model updates its weights based on the calculation results of this loss function, gradually optimizing its performance on the traffic sign detection task until the model converges, that is, the value of the loss function no longer decreases significantly.

[0163] Optionally, the traffic sign recognition model is trained in the following manner: dividing historical traffic sign images into a training set and a test set; using the training set to train the initial traffic sign recognition model, and when the total loss function value of the initial traffic sign recognition model no longer decreases, stopping the training to obtain a first traffic sign recognition model; using the test set to test the first traffic sign recognition model to obtain a test result, and when the test result exceeds a preset threshold, stopping the test to obtain a traffic sign recognition model.

[0164] In an embodiment of the present application, the training process of the traffic sign recognition model is based on a supervised learning process under a deep learning framework, which specifically includes data set division, model training, loss function monitoring and model performance testing.

[0165] The details are as follows:

[0166] First, in order to train and evaluate the model, the historical traffic sign images are divided into a training set and a test set. The training set is used for the actual training of the model, and through repeated iterations, the model learns how to recognize traffic signs from images. The test set is used to evaluate the generalization ability of the model, that is, the performance of the model on unseen data. The purpose of data set division is to ensure that the model can not only perform well on the training data, but also accurately detect traffic signs on unknown data. Usually, the data set is divided into about 80% training set and 20% test set, which can be adjusted according to the needs of the specific task.

[0167] The training process starts with an initial model that may contain randomly initialized weights. Using the training set, the model predicts the location and category of traffic signs through forward propagation, and then adjusts its weights through backpropagation according to the loss function (such as CIoU_loss, the weighted sum of classification loss and confidence loss) to minimize the difference between the predicted results and the true labels. This process continues for multiple cycles (i.e., the number of iterations) until the total loss function value of the model no longer decreases significantly, which usually indicates that the model has converged, that is, it has learned the patterns in the dataset and can no longer significantly improve its performance. At this point, the model is called the first traffic sign recognition model.

[0168] Subsequently, the test set is used to evaluate the performance of the first traffic sign recognition model. During the test, the model will not adjust the weights, but will use the trained weights to predict the location and category of traffic signs. The test results will include performance indicators such as the model's detection accuracy, recall rate, F1 score, etc. on the test set. A test result that exceeds the preset threshold means that the model has reached the pre-set performance standard, which is usually a comprehensive accuracy indicator, such as an mAP (meanAverage Precision) value exceeding a certain threshold. If the test results are satisfactory, that is, they exceed the preset performance threshold, the model is considered to be trained and can be directly used in actual traffic sign detection tasks, becoming the final traffic sign recognition model.

[0169] In the above process, the experimental configuration information of the model training is shown in Table 1:

[0170] Table 1 Experimental environment configuration

[0171]

[0172] In the above process, the parameter settings required for training are shown in Table 2:

[0173] Table 2 Training model parameters

[0174]

[0175]

[0176] Figure 5 is a schematic diagram of a traffic sign model according to an embodiment of the present application. Figure 5 As shown in the figure, the network architecture consists of three main parts: backbone network, neck network and head network. The specific functions are introduced as follows:

[0177] First, the input layer receives the original image data. Secondly, in this backbone network, the Focus module is used to take values ​​from every other pixel in the input original image data to achieve dimensionality reduction and increase the number of channels. The convolution layer is used to perform convolution operations on the output of the Focus module to extract features. BottleNeckCSP1_1 and BottleNeckCSP1_3 represent two bottleneck structures for further extracting and fusing features. CP-Transformer is a parallel Transformer module used to enhance the information extraction ability of the feature extraction network for traffic signs. In the neck network, BottleNeckCSP2_1 represents a bottleneck structure for feature fusion and multi-scale feature extraction; upsampling is used to increase the resolution of the feature map so that it can be fused with feature maps of different scales; the fully connected layer is used for further processing and fusion of features; the convolution layer is applied after upsampling to adjust the number of channels and resolution of the feature map. In the head network, BottleNeckCSP2_1 is used for the final feature fusion and preparation for detection, the SE attention mechanism is used to enhance the weights of key features and suppress unimportant features. The detection head is used as the final detection layer to output the detection results of traffic signs.

[0178] In the embodiment of the present application, a RCS-YOLOv5 traffic sign recognition method based on super-resolution image reconstruction is proposed, which integrates a unique image preprocessing and deep learning detection framework. First, super-resolution reconstruction is performed by generating adversarial networks, which significantly improves the clarity of low-resolution traffic images and solves the problems of missed detection and slow recognition caused by poor image quality. Secondly, the RCS-YOLOv5 model uses CP-Transformer to enhance the globality and accuracy of feature extraction, and optimizes target positioning and recognition in combination with SE attention mechanism and CIoU_loss, especially in the detection of small targets in complex traffic scenes. The traffic sign recognition method effectively overcomes the problems of high data dependence, weak generalization ability and difficulty in small target detection in the prior art, greatly improves the accuracy and efficiency of traffic sign detection, and has significant beneficial effects on ensuring the safe driving of autonomous driving vehicles and the efficient operation of intelligent transportation systems. Through the comprehensive application of the above-mentioned technologies, the present invention not only improves the robustness of the model under various environmental conditions, but also accelerates the training and detection process of the model, and realizes faster and more accurate traffic sign detection capabilities.

[0179] According to an embodiment of the present application, a traffic sign recognition device is provided. It should be noted that the traffic sign recognition device of the embodiment of the present application can be used to execute the traffic sign recognition method provided by the embodiment of the present application. The traffic sign recognition device provided by the embodiment of the present application is introduced below.

[0180] Figure 6 is a structural diagram of a traffic sign recognition device provided according to an embodiment of the present application. Figure 6 As shown, the device comprises:

[0181] An acquisition module 60 is used to acquire a traffic sign image in a traffic driving scene;

[0182] A reconstruction module 62, used for performing super-resolution reconstruction processing on the traffic sign image to obtain a high-resolution image of the traffic sign image;

[0183] The recognition module 64 is used to analyze and process the high-resolution image using a traffic sign recognition model to obtain a recognition result corresponding to the traffic sign image, wherein the traffic sign recognition model includes a backbone network, a neck network and a head network. The backbone network is used to perform multi-level feature extraction on the high-resolution image through a target transformer to obtain a multi-level feature map. The neck network is used to fuse the multi-level feature map through a feature pyramid network and a path aggregation network to obtain a fused feature map. The head network is used to perform position recognition on the fused feature map through a target loss function to obtain a recognition result.

[0184] Through the acquisition module 60, reconstruction module 62 and recognition module 64 in the above-mentioned traffic sign recognition device, the purpose of significantly improving the detection accuracy and recognition speed of traffic signs in low-resolution images based on super-resolution image reconstruction technology and improved RCS-YOLOv5 model is achieved, thereby achieving efficient detection of traffic signs in complex driving environments, especially under conditions of poor image quality, and maintaining a high detection performance. The technical problem of high missed detection rate, poor detection accuracy and slow recognition speed caused by unclear traffic sign images is solved.

[0185] In the traffic sign recognition device provided in an embodiment of the present application, the reconstruction module is also used to perform feature extraction on the traffic sign image through the convolution layer in the image super-resolution reconstruction module to obtain a first feature map, wherein the first feature map is used to represent the shallow feature information of the traffic sign image; perform feature extraction on the first feature map through the residual dense block in the image super-resolution reconstruction module to obtain a second feature map, wherein the residual dense block performs feature extraction and feature fusion on the first feature map through a dense connection layer and a residual learning mechanism, and the second feature map is used to represent the deep feature information of the traffic sign image; perform an upsampling operation on the second feature map to obtain a high-resolution image.

[0186] In the traffic sign recognition device provided in an embodiment of the present application, the reconstruction module is also used to perform an upsampling operation on the second feature map through the convolution layer, activation function layer and sub-pixel convolution layer in the image super-resolution reconstruction module to obtain a first resolution image of the traffic sign image; amplify the second feature map through the bicubic interpolation method to obtain a second resolution image of the traffic sign image, wherein the bicubic interpolation method is used to reduce the jagged effect of the second feature map during the amplification process; and fuse the first resolution image and the second resolution image to obtain a high-resolution image.

[0187] In the traffic sign recognition device provided in the embodiment of the present application, the recognition module is also used to perform feature extraction on the high-resolution image through the first module in the backbone network to obtain a third feature map, wherein the first module is used to perform dimensionality reduction and downsampling operations on the high-resolution image; perform feature extraction on the third feature map through the second module in the backbone network to obtain a fourth feature map, wherein the second module is used to perform convolution processing, normalization processing and nonlinear activation processing on the third feature map; perform feature extraction on the fourth feature map through the third module in the backbone network to obtain a multi-level feature map, wherein the third module includes a target transformer, and the third module is used to perform image segmentation and image fusion on the fourth feature map through the target transformer.

[0188] In the traffic sign recognition device provided in the embodiment of the present application, the recognition module is also used to perform feature fusion on multi-level feature maps through a feature pyramid network to obtain a fifth feature map, wherein the feature pyramid network fuses the multi-level feature maps through a top-down path; perform feature fusion on the multi-level feature maps through a path aggregation network to obtain a sixth feature map, wherein the path aggregation network fuses the multi-level feature maps through a bottom-up path; and fuse the fifth feature map and the sixth feature map to obtain a fused feature map.

[0189] The traffic sign recognition device provided in the embodiment of the present application also includes a determination module 66, which is used to determine the prediction box and prediction category of the historical traffic sign image through a convolution operation, wherein the prediction box is used to indicate the predicted position and size of the traffic sign in the historical traffic sign image, and the prediction category is used to indicate the predicted traffic sign type to which the traffic sign in the historical traffic sign image belongs; the position information, category information and confidence of the historical traffic sign image are determined through an initial loss function, the position information is used to indicate the positioning loss between the prediction box and the label box, the category information is used to indicate the classification loss between the prediction category and the label category, the confidence is used to indicate the degree of matching between the prediction box and the label box, the label box is used to indicate the actual position and size of the traffic sign in the historical traffic sign image, and the label category is used to indicate the actual traffic sign type to which the traffic sign in the historical traffic sign image belongs; the target loss function is determined based on the position information, category information and confidence.

[0190] In the traffic sign recognition device provided in the embodiment of the present application, the determination module is also used to divide the historical traffic sign images into a training set and a test set; the training set is used to train the initial traffic sign recognition model, and when the total loss function value of the initial traffic sign recognition model no longer decreases, the training is stopped to obtain the first traffic sign recognition model; the test set is used to test the first traffic sign recognition model to obtain the test result, and when the test result exceeds a preset threshold, the test is stopped to obtain the traffic sign recognition model.

[0191] An embodiment of the present application also provides an electronic device, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned traffic sign recognition method.

[0192] It should be noted that the above electronic equipment is used to execute Figure 2 The traffic sign recognition method shown in the figure, therefore the relevant explanations and instructions in the above traffic sign recognition method are also applicable to the electronic device and will not be repeated here.

[0193] An embodiment of the present application further provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned traffic sign recognition method by running the computer program.

[0194] It should be noted that the above non-volatile storage medium is used to execute Figure 2 The traffic sign recognition method shown in the figure, therefore the relevant explanations and instructions in the above traffic sign recognition method are also applicable to the non-volatile storage medium, and will not be repeated here.

[0195] An embodiment of the present application also provides a computer program product, including computer instructions, which implement the above-mentioned traffic sign recognition method when executed by a processor.

[0196] It should be noted that the above-mentioned computer program product is used to execute Figure 2 The traffic sign recognition method shown in the figure, therefore the relevant explanations and instructions in the above-mentioned traffic sign recognition method are also applicable to the computer program product, and will not be repeated here.

[0197] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0198] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0199] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0200] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0201] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0202] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.

[0203] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for recognizing a traffic sign, characterized in that: include: Acquire traffic sign images in traffic driving scenarios; Performing super-resolution reconstruction processing on the traffic sign image to obtain a high-resolution image of the traffic sign image; The high-resolution image is analyzed and processed by a traffic sign recognition model to obtain a recognition result corresponding to the traffic sign image, wherein the traffic sign recognition model includes a backbone network, a neck network and a head network, the backbone network is used to perform multi-level feature extraction on the high-resolution image through a target transformer to obtain a multi-level feature map, the neck network is used to fuse the multi-level feature maps through a feature pyramid network and a path aggregation network to obtain a fused feature map, and the head network is used to perform position recognition on the fused feature map through a target loss function to obtain the recognition result.

2. The method according to claim 1, characterized in that Performing super-resolution reconstruction processing on the traffic sign image to obtain a high-resolution image of the traffic sign image includes: Performing feature extraction on the traffic sign image through a convolutional layer in an image super-resolution reconstruction module to obtain a first feature map, wherein the first feature map is used to represent shallow feature information of the traffic sign image; Extracting features from the first feature map through a residual dense block in the image super-resolution reconstruction module to obtain a second feature map, wherein the residual dense block extracts features and fuses features from the first feature map through a dense connection layer and a residual learning mechanism, and the second feature map is used to represent deep feature information of the traffic sign image; An upsampling operation is performed on the second feature map to obtain the high-resolution image.

3. The method according to claim 2, characterized in that Performing an upsampling operation on the second feature map to obtain the high-resolution image includes: Performing an upsampling operation on the second feature map through a convolution layer, an activation function layer, and a sub-pixel convolution layer in the image super-resolution reconstruction module to obtain a first resolution image of the traffic sign image; Amplifying the second feature map by a bicubic interpolation method to obtain a second resolution image of the traffic sign image, wherein the bicubic interpolation method is used to reduce a sawtooth effect of the second feature map during the amplification process; The first resolution image and the second resolution image are fused to obtain the high-resolution image.

4. The method according to claim 1, characterized in that The backbone network is used to perform multi-level feature extraction on the high-resolution image through a target transformer to obtain a multi-level feature map, including: Performing feature extraction on the high-resolution image through the first module in the backbone network to obtain a third feature map, wherein the first module is used to perform dimensionality reduction and downsampling operations on the high-resolution image; Performing feature extraction on the third feature map through the second module in the backbone network to obtain a fourth feature map, wherein the second module is used to perform convolution processing, normalization processing and nonlinear activation processing on the third feature map; The third module in the backbone network is used to perform feature extraction on the fourth feature map to obtain the multi-level feature map, wherein the third module includes the target converter, and the third module is used to perform image segmentation and image fusion on the fourth feature map through the target converter.

5. The method according to claim 1, characterized in that The neck network is used to fuse the multi-level feature maps through a feature pyramid network and a path aggregation network to obtain a fused feature map, including: Performing feature fusion on the multi-level feature maps through the feature pyramid network to obtain a fifth feature map, wherein the feature pyramid network fuses the multi-level feature maps through a top-down path; Performing feature fusion on the multi-level feature maps through the path aggregation network to obtain a sixth feature map, wherein the path aggregation network fuses the multi-level feature maps through a bottom-up path; The fifth feature map and the sixth feature map are fused to obtain the fused feature map.

6. The method according to claim 1, characterized in that The objective loss function is determined in the following way: Determining a prediction box and a prediction category of the historical traffic sign image by a convolution operation, wherein the prediction box is used to indicate the predicted position and size of the traffic sign in the historical traffic sign image, and the prediction category is used to indicate the predicted traffic sign type to which the traffic sign in the historical traffic sign image belongs; Determine the position information, category information and confidence of the historical traffic sign image by an initial loss function, wherein the position information is used to represent the positioning loss between the prediction box and the label box, the category information is used to represent the classification loss between the prediction category and the label category, the confidence is used to represent the matching degree between the prediction box and the label box, the label box is used to represent the real position and size of the traffic sign in the historical traffic sign image, and the label category is used to represent the real traffic sign type to which the traffic sign in the historical traffic sign image belongs; The target loss function is determined based on the position information, the category information and the confidence level.

7. The method according to claim 6, characterized in that The traffic sign recognition model is trained in the following way: Dividing the historical traffic sign images into a training set and a test set; The training set is used to train the initial traffic sign recognition model, and when the total loss function value of the initial traffic sign recognition model no longer decreases, the training is stopped to obtain a first traffic sign recognition model; The first traffic sign recognition model is tested using the test set to obtain a test result. When the test result exceeds a preset threshold, the test is stopped to obtain the traffic sign recognition model.

8. A traffic sign recognition device, characterized in that: include: An acquisition module, used for acquiring traffic sign images in traffic driving scenarios; A reconstruction module, used for performing super-resolution reconstruction processing on the traffic sign image to obtain a high-resolution image of the traffic sign image; A recognition module is used to analyze and process the high-resolution image using a traffic sign recognition model to obtain a recognition result corresponding to the traffic sign image, wherein the traffic sign recognition model includes a backbone network, a neck network and a head network, the backbone network is used to perform multi-level feature extraction on the high-resolution image through a target transformer to obtain a multi-level feature map, the neck network is used to fuse the multi-level feature map through a feature pyramid network and a path aggregation network to obtain a fused feature map, and the head network is used to perform position recognition on the fused feature map through a target loss function to obtain the recognition result.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; The processor is connected to the memory and is used to execute the traffic sign recognition method described in any one of claims 1 to 7.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the traffic sign recognition method according to any one of claims 1 to 7 by running the computer program.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the traffic sign recognition method according to any one of claims 1 to 7 is implemented.

Images