A real-time traffic sign recognition system for automated driving systems

Abstract

A real-time traffic sign recognition system consisting of: a neural network module configured to process input images containing traffic signs; a lightweight, shallow information enhancement module configured to split input feature maps into multiple parts, apply a depth-wise convolution to each part, apply a pointwise convolution to combine fitted values, and concatenate features from each processed part; a customizable channel attention mechanism configured to perform global maximum pooling and global average pooling for input features, combine pooled results using customizable parameters, encode and decode the compressed features through fully connected layers, and generate channel attention coefficients; a cross-stage sub-module configured to implement window-based multi-head self-attention for local feature processing, capture context information around traffic signs, and combine features of convolutional neural networks with transformer features; and an adaptive feature fusion module configured to generate learnable weights for feature fusion, combine features from different network layers, and output detected traffic signs with location information.

Description

FIELD OF THE INVENTIONThe present disclosure relates to a system for real-time traffic sign recognition. In particular, it is a real-time traffic sign recognition system for automatic driving systems, which is based on YOLOv5 and Swin transformers.BACKGROUND OF THE INVENTIONRoad sign recognition systems are essential components of modern intelligent driving technology and play a decisive role for traffic safety and traffic management. These systems capture real-time images of the road using vehicle-mounted cameras. These are then processed to recognize and identify traffic signs. Thus, drivers obtain information about road conditions and regulations early.The challenge in developing effective traffic sign recognition systems is dealing with real complications such as different lighting conditions, bad weather, partial vision impairment, and confusion background elements. Conventional machine learning approaches to solve this problem are based on manual feature extraction, often resulting in incomplete feature sets and limited model realization. While these techniques have shown some success, they have deficiencies in accuracy and real-time performance.Deep learning recognition methods have proven to be promising alternatives and can be broadly classified into two categories. The first category includes region-based methods such as R-CNN and Faster R-CNN that operate in two stages by generating candidate boxes and then processing them to determine object categories and locations. Although these methods achieve high accuracy, they are unsuitable for real-time applications because of their computing effort. The second category includes single-stage recognition methods such as YOLOv3, RetinaNet, and SSD that directly extract features for classification and localization. These methods offer faster recognition speeds, but generally lose some accuracy compared to two-stage approaches.To address these limitations, a novel traffic sign recognition system is developed that combines the advantages of the YOLOv5efficiency architecture with the high performance feature learning functions of Swin transformers. This approach aims at high accuracy and real-time performance in road sign recognition and is therefore particularly suitable for practical applications in intelligent driving systems.SUMMARY OF THE INVENTIONThe present disclosure relates to a system for real-time traffic sign recognition. The present invention relates to a traffic sign recognition system combining the YOLOv5 architecture with the Swin transformer technology. The system comprises four main modules: a lightweight flat information enhancement module for efficient feature processing, an adjustable channel attention mechanism for optimizing information channels, an inter-stage Swin transformer-based submodule for contextual information acquisition, and an adaptive feature fusion module for combining features from different network levels. This integrated approach enables precise real-time recognition of road signs under various conditions.The disclosure relates to providing a system for real-time traffic sign recognition. The system includes: a neural network module for processing input images with traffic signs; a lightweight superficial information enhancement module that decomposes input feature maps into multiple parts, applies depth convolution to each part, combines matched values by point convolution, and associates features from each processed part; an adjustable channel attention mechanism that performs global maximum pooling and global average pooling for input features, combines pooled results with adjustable parameters, encodes and decodes the compressed features over fully connected layers, and generates channel attention coefficients; a cross-step submodule that implements window-based multi-head self attention for local feature processing, captures contextual information around traffic signs, and combines features of the convolutional neural network with transformer features; and an adaptive feature fusion module that generates trainable weights for feature fusion, combines features from different network levels, and outputs detected traffic signs with location information.An object of the present disclosure is to provide a system for real-time traffic sign recognition.Another object of the present disclosure is to achieve high-precision road sign recognition through an optimized neural network architecture while maintaining real-time performance.Another object of the present disclosure is to improve recognition of small traffic signs by effectively acquiring and processing contextual information.Another object of the present disclosure is to reduce computational complexity while maintaining recognition accuracy through efficient feature processing and fusion mechanisms.In order to further clarify the advantages and features of the present disclosure, the invention will be described in more detail with reference to specific embodiments illustrated in the accompanying drawings. These drawings illustrate only typical embodiments of the invention and are therefore not to be considered as limiting the scope thereof. The invention will be described and explained in more detail with reference to the accompanying drawings.BRIEF DESCRIPTION OF THE FIGURESThese and other features, aspects, and advantages of the present disclosure will become more fully understood when the following detailed description is read with reference to the accompanying drawings, in which like characters represent like parts throughout. The following applies here: FIG. 1 shows a block diagram of a system for real-time traffic sign recognition according to an embodiment of the present disclosure. FIG. 2 shows a block diagram of the proposed network architecture of TSDet, according to an embodiment of the present disclosure.Those skilled in the art will also appreciate that the elements in the drawings are shown for simplicity and are not necessarily to scale. For example, the flowcharts illustrate the method using the key steps to improve understanding of aspects of the present disclosure. In addition, regarding the construction of the apparatus, individual or multiple components of the apparatus may be represented by conventional symbols in the drawings. The drawings may only show the specific details relevant to understanding the embodiments of the present disclosure in order not to obscure the drawings with details readily apparent to those skilled in the art after the present description.DETAILED DESCRIPTION:In order to aid in the understanding of the principles of the invention, reference will now be made to the embodiment illustrated in the drawings and will be described in an comprehensible manner. However, the scope of the invention is not limited thereby. Changes and further modifications of the illustrated system, as well as further applications of the principles of the invention, are possible, as would normally occur to a person skilled in the art.It will be understood by those skilled in the art that the foregoing general description and the following detailed description are exemplary and explanatory of the invention and are not intended to be limiting thereof.References throughout this specification to "one aspect," "another aspect," or similar language mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, the phrases "in one embodiment," "in another embodiment," and similar phrases in this specification may or may not refer to the same embodiment.The terms "comprises," "comprising," or variations thereof cover a nonexclusive inclusion. A process or method comprising a list of steps includes not only those steps, but also other steps not expressly listed or inherent to the process or method. Likewise, the phrase "comprises... for" one or more devices, subsystems, elements, structures, or components does not exclude, without further limitations, the existence of further devices, subsystems, elements, structures, or components, or additional devices, subsystems, elements, structures, or components.Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by one of ordinary skill in the art. The systems, methods, and examples provided herein are for illustrative purposes only and are not to be considered limiting.Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.The functional units described in this specification are referred to as devices. A device may be implemented in programmable hardware devices such as processors, digital signal processors, central processing units, field programmable gate arrays, programmable array logic systems, programmable logic devices, cloud processing systems, or the like. The devices may also be implemented in software for execution by various types of processors. An identified device may include executable code and may consist, for example, of one or more physical or logical blocks of computer instructions, which may be organized, for example, as an object, procedure, function, or other construct. However, the executable of an identified device need not be physically stored at the same location, but may consist of different instructions stored at different locations that, logically linked, form the device and serve its purpose.Device or module executable code may consist of one or more instructions and even be distributed over multiple code segments, different applications, and multiple storage devices. Likewise, operational data may be identified and displayed within the device and presented in any form and data structure. The operational data may be acquired as a single data set or distributed across different storage devices and may be at least partially present as electronic signals in a system or network.References throughout this specification to "a selected embodiment," "an embodiment," or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosed subject matter. Therefore, the terms "a selected embodiment," "in one embodiment," or "in one embodiment" in various places throughout this specification do not necessarily refer to the same embodiment.Moreover, the described features, structures, or characteristics may be combined in any manner in one or more embodiments. The following description contains numerous specific details to provide a thorough understanding of the embodiments of the disclosed subject matter. However, those skilled in the art will appreciate that the disclosed subject matter may be practiced without these specific details or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail in order not to obscure aspects of the disclosed subject matter.According to the example embodiments, the disclosed computer programs or modules may be executed in a variety of ways, such as an application in a device's memory or a hosted application on a server that communicates with the device application or browser via various standard protocols such as TCP / IP, HTTP, XML, SOAP, REST, JSON, and other suitable protocols. The disclosed computer programs may be written in example programming languages that execute from the memory of the device or from a hosted server, such as BASIC, COBOL, C, C++, Java, Pascal, or scripting languages such as JavaScript, Python, Ruby, PHP, Perl, or other suitable programming languages.Some of the disclosed embodiments include or otherwise involve data transfer over a network, for example, the transfer of various inputs or files over the network. The network may include, for example, the Internet, wide area networks (WANs), local area networks (LANs), analog or digital wired and wireless telephone networks (e.g., PSTN, Integrated Services Digital Network (ISDN), cellular networks and digital subscriber line (xDS)), radio, television, cable, satellite, and / or other transmission or tunneling mechanisms for data transmission. The network may comprise multiple networks or sub-networks, each including, for example, a wired or wireless data path. The network may comprise a circuit switched voice network, a packet switched data network or other network for transferring electronic communication. For example, the network may comprise networks based on Internet Protocol (IP) or Asynchronous Transfer Mode (ATM) and support voice, for example, via VoIP, voice over ATM, or other comparable protocols for voice data communication. In one implementation, the network includes a cellular network configured to exchange text or SMS messages.Examples of the network include a personal area network (PAN), a storage area network (SAN), a home area network (HAN), a campus area network (CAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a virtual private network (VPN), an enterprise private network (EPN), the Internet, a global area network (GAN), etc.FIG. 1 shows a block diagram of a system ( 100) for real-time traffic sign recognition according to an embodiment of the present disclosure.Referring to FIG. 1, the system (100) includes: a neural network module (102) configured to process input images with traffic signs; a lightweight superficial information enhancement module (104) configured to split input feature maps into multiple parts, apply depth convolution to each part, combine adjusted values by point convolution, and associate features from each processed part; an adjustable channel attention mechanism (106) configured to apply global maximum pooling and global average pooling to input features, combine pooled results with adjustable parameters, encode and decode the compressed features over fully connected layers, and generate channel attention coefficients; an inter-stage submodule (108) configured to implement window-based multi-head self-attention for local feature processing, detect contextual information around traffic signs and combine features of a convolutional neural network with transformer features; and an adaptive feature fusion module (110) configured to generate trainable weights for feature fusion, combine features from different network levels, and output detected traffic signs with location information.In one embodiment, the lightweight superficial information enhancement module (104) also includes: a k×k depth convolution layer for linear matching; and a 1×1 point convolution layer for the feature combination.In one embodiment, the adjustable channel attention mechanism (106) is configured to increase the importance of useful information channels while reducing emphasis on redundant or background channels.In one embodiment, the cross-level submodule (108) includes a multi-head window self-attention layer configured to split feature maps into a plurality of sub-windows and calculate self-attention within local windows, and a multi-layer perceptron configured to process the attention outputs, wherein the multi-head window self-attention layer is configured to process feature maps of size H×B×C, split features into local windows of size K×K, and independently calculate attention within each window.In one embodiment, the adaptive feature fusion module (110) is configured to receive features from different network levels, generate adaptive weights by convolution operations, normalize weights using a sigmoid function, and combine features using the generated weights, wherein the adaptive feature fusion module further comprises: multiple 1×1 convolution layers, channel chaining operations, ReLU activation functions, and a final sigmoid layer for weight normalization.In one embodiment, the neural network module (102) includes: a backbone network based on YOLOv5, a neck network implementing path aggregation, and a detection head for outputting road sign locations and classifications.In one embodiment, the system (100) is configured to recognize traffic signs of different scales, operate in real time at least 80 images per second, and achieve an average accuracy of at least 75% for traffic sign datasets.In one embodiment, the lightweight superficial information enhancement module (104) is configured to: preserve spatial information while reducing computational complexity; enhance recognition of road sign features, including color and shape; and output enhanced feature maps for subsequent processing phases.The present invention relates to a real-time traffic sign recognition system for automated driving systems. The traffic sign recognition system is based on an elaborate combination of neural network components that work together harmonically. In essence, the system is based on a lightweight superficial information enhancement module that processes input images through a special structure of depth and pointwise convolutions. This module efficiently disassembles feature maps and processes them separately before re-combining them. This enables detailed feature extraction with the same computing power. The adjustable channel attention mechanism of the system makes it possible to dynamically emphasize important information channels and to suppress less relevant information channels. This is achieved by a unique combination of maximum and average global pooling operations with adjustable parameters that allow the system to adapt to different detection scenarios. The pooled results are processed through fully connected layers to generate optimal channel attention coefficients. The cross-stage partial module with Swin transformer technology provides a novel approach to processing spatial information. It divides the feature space into local windows and uses self-attention mechanisms within these windows. This allows the system to acquire detailed contextual information around traffic signs. This is especially important for the precise recognition of smaller traffic signs, which might otherwise be missed. The last component, the adaptive feature fusion module, combines intelligent features from different network levels. It generates trainable weights through a series of convolution operations and uses them to optimally blend features. This ensures that both deep semantic information and superficial detailed features effectively contribute to the final recognition results. This comprehensive approach allows the system to have high detection accuracy at real-time speed.FIG. 2 shows a block diagram of the proposed network architecture of TSDet, according to an embodiment of the present disclosure.The proposed system focuses on the balance between speed and accuracy in road sign recognition by integrating a lightweight YOLOv5 framework with the SwinT module. This extension allows the model to grasp relationships between different locations and thus ultimately improve recognition accuracy for small traffic signs. By integrating SuminT into the CSP structure of YOLOv5 (CSP-SuminT), the model enhances its ability to process fine details. Additionally, an adjustable channel attention mechanism (CSP-ACAM) has been introduced within the CSP structure to refine the meaning of useful information channels. To further enhance the ability of the model to process superficial information, a lightweight component for improving superficial information, light focus, has been developed. Finally, an adaptive feature fusion, AFF, approach has been developed to effectively combine deep semantic and superficial detailed features. The architecture of TSDet, comprising light focus, CSP ACAM, CSP-SwinT and AFF, is shown in Fig. 2. Each of these components plays a decisive role in power optimization.A central aspect of the proposed system is the improvement of superficial features essential for the recognition of fine details in traffic signs. To achieve this, light focus has been introduced, an efficient and lightweight component for improving surface information. In contrast to the conventional "focus" structure, light focus uses a depth-separable convolution to reduce the number of parameters and the computational effort. The process includes dividing the feature map and applying a depth-separable convolution (DW) with a k×k kernel for linear matching. This step generates adjustment values which are then processed by a 1×1 point-by-point convolution (PW) to generate new data features. Finally, these features are linked for output. The efficiency of light focus relies on the ability to compact spatial information into channel information, thus improving the model's ability to detect superficial features such as color and shape. Integrating the depth-separate convolution with a large kernel reduces floating point operations and the total parameter count further, and makes the system lightweight and yet effective.Another feature of the proposed system is the Adjustable Channel Attention Mechanism (ACAM) built on the widely distributed SE-Net. SE-Net is known for its simplicity and low computational cost, whereby models can emphasize useful channels and suppress less relevant ones. However, conventional SE-Net approaches are challenging in traffic sign recognition because traffic signs in images are often small and are surrounded by extensive background information. Standard global mean pooling techniques may not fully capture the channel responses required for accurate detection. To address this problem, ACAM introduces an improved compression strategy that ensures that extreme channel responses are also accounted for.ACAM functions by first applying both global maximum pooling (GMP) and global average pooling (GAP) to input features of size H×B×C. The outputs of these pooling operations are combined in proportion to the adjustable parameters α and β to ensure that their sum gives 1. This process generates a compressed feature representation of 1×1×C. The next step is to feed these compressed features into a fully connected network for encoding and decoding. First, the feature dimension is reduced to C / r by the first fully connected (FC) layer, where r is an adjustable parameter. Then, the activation function is applied before the second FC layer restores the original feature dimension. Finally, a sigmoidal activation function converts the processed values to a range between 0 and 1, which is then used to refine the original channel representation. This approach improves system nonlinearity, allows more effective learning of channel relationships, and ultimately improves traffic sign detection performance.The proposed system contains a CSP module based on the Swin transformer. Multi-head self-attention (MHSA) plays a decisive role in understanding the relationships between objects and their environment and enables better capture of context information, in particular in the case of small objects. However, applying global self-attention to high-resolution feature maps results in a considerable computational effort and a high memory requirement. To increase the computational efficiency and at the same time reduce the memory requirement, the Swin transformer is introduced. The CSP module and the CSP module are integrated into the Swin transformer. The red dashed box marks the integration of the Swin transformer into the system.The Swin transformer consists of two main layers. The first layer, Window Multi-Head Self-Attention (W-MHSA), segments the feature map into several smaller sub-windows and calculates the self-attention within these local windows by averaging the feature map. The second layer is a multi-layer perceptron (MLP). For a feature map of size H×W×C and a local window of size K×K, the computational complexity of the global MHSA and the window-based W-MHSA can be calculated using equation (1) and equation (2).The comparison of equation (1) and equation (2) clearly shows that W-MHSA for high-resolution feature maps significantly reduces the computing effort compared to conventional MHSA. To further minimize computational effort and simultaneously obtain the advantages of CNN and Transformer architectures, this invention introduces the CSP SwinT module embedding the Swin Transformer in the CSP framework. The left branch uses a 1×1 convolution to reduce the channel number and thus effectively reduce the computational effort and memory consumption of the transformer. The right branch also uses a 1×1 convolution to maintain the key features extracted by the CNN. Finally, the system merges the outputs by channel splicing, thus providing a rich and differentiated feature representation.The Path Aggregation Network (PANet) was integrated into the neck region of the YOLOv5 network to allow feature fusion at different levels. In the top-down path, different feature levels are generated. These planes are successively ramped up at a rate of 2. At the same time, the feature maps are created by path polymerization corresponding to the respective feature levels. Because the manner in which these different feature levels are merged significantly affects object detection performance, the conventional approach to feature linkage does not effectively detect the correlation between the channels. To address this limitation, an Adaptive Feature Fusion (AFF) system has been integrated into the system.This adaptive approach combines the convolution function of Ci with the upsampling function of F i+1 to F i. The mathematical representation of this operation is given in equation (3):In the above equation, F up denotes the upper scan plane, and the symbol C denotes a channel-level concatenation operation. C i denotes the convolution feature of the i ten layer in the backbone network, while Firepresents the feature map of the i thlayer in the top-down path of PANet. Similarly, F i+1 denotes the feature map of the subsequent layer i+1in the same path. The system assigns trainable weights w1and w2that are tensors having the dimensions H×W×1. Since different feature maps extract different features, F i integrates the superficial feature C i with the deep feature F i+1 through the adaptive system. This process solves information conflicts and at the same time minimizes the influence of deep features on surface features. Changing from fixed to adaptive weights improves the performance of the network for objects of different orders of magnitude.When two feature maps of different levels are fused, they are first adjusted to the same dimensions H×B×C, where H, B, and C represent height, width, and number of channels, respectively. First, a 1×1 convolution is applied to both feature maps, followed by activation by the ReLU function, thereby reducing the number of channels to C / 2. The features are then concatenated along the channel dimension and reset to C dimensions before being re-compressed to C / 2 by another 1×1 convolution. This output is then combined with the result of the first convolution, resulting in a final feature map having the dimensions H×B×C. The number of channels is finally reduced to a single dimension by a further 1×1 convolution and the sigmoid function normalizes the weight values to a range between 0 and 1. As the matrix weight is generated by convolution, it is continuously updated so that the system can determine the optimal weight for the effective fusion of the two feature maps.In one embodiment, the performance of the proposed system is evaluated. To demonstrate the effectiveness of each component of the proposed system, a series of simulations were performed based on specific data sets and evaluation criteria. The experiments were performed on a hardware setup consisting of a central processor and a graphics processor. The software environment included an operating system, a deep learning framework, and the use of a programming language. During the actual training, the optimizer SGD was used. The batch size was set to 4, with an initial learning rate of 0.001. The Momentum value was adjusted to 0.937 while the weight loss was maintained at 0.0005. The total training iterations was set to 300 and all other parameters followed the default settings of the official implementation. Moreover, both training and test images were processed with a resolution of 1024×1024. The simulation is performed with the traffic sign recognition datasets TT100K and DFG and their mAP, AP50 and FPS were (75.3, 94.8, 82) and (79.7, 85.9, 118), respectively.The results of the above-mentioned simulation showed that the proposed TSDet system effectively improves the recognition accuracy of small traffic signs and at the same time meets the requirements for real-time recognition. The experimental results show that the proposed method for road sign recognition has a high accuracy and real-time performance and meets the requirements for road sign recognition in the real scene.The drawings and the foregoing description show examples of embodiments. Those skilled in the art will appreciate that one or more of the described elements may well be combined into a single functional element. Alternatively, certain elements may be divided into multiple functional elements. Elements of one embodiment may be added to another embodiment. For example, the order of the processes described herein may be changed and is not limited to the manner described herein. Moreover, the actions of a flow chart need not be performed in the order shown; nor do all actions necessarily need to be performed. Also, actions that are not dependent on other actions may be performed in parallel with the other actions. The scope of the embodiments is by no means limited by these specific examples. Numerous variations, whether or not explicitly stated in the specification, such as differences in structure, dimensions, and material use, are possible. The scope of the embodiments is at least as broad as recited in the following claims.Advantages, other advantages and solutions to problems have been described above with reference to specific embodiments. However, the advantages, merits, solutions to problems and any components that may result in an advantage, benefit or solution being introduced or enhanced are not to be understood as critical, required or essential features or components of individual or all claims.REFERENCES100 A system for real-time traffic sign recognition. 102 Neural network module 104 Lightweight module for improving surface information 106 Adjustable channel attention mechanism 108 Cross-step submodule 110 Adaptive feature fusion module

Claims

A system for real-time traffic sign recognition, comprising: a neural network module configured to process input images with traffic signs; a lightweight, superficial information enhancement module configured to split input feature maps into multiple parts, apply depth convolution to each part, apply point-by-point convolution to combine adjusted values, and concatenates from each processed part; an adjustable channel attention mechanism configured to perform global maximum pooling and global average pooling for input functions, combine pooled results using adjustable parameters that encode and decode compressed functions through fully connected layers and generate channel attention coefficients; a cross-step submodule configured to implement window-based multi-head self attention for local feature processing, detect contextual information around traffic signs, and combine convolutional neural network features with transformer features; and an adaptive feature fusion module configured to generate trainable weights for feature fusion, combine features from different network levels, and output detected traffic signs with location information.The system of claim 1, wherein the lightweight superficial information enhancement module further comprises: a k×k depth convolution layer for linear matching; and a 1×1 point convolution layer for the feature combination.The system of claim 1, wherein the adjustable channel attention mechanism is configured to increase the importance of useful information channels while reducing emphasis of redundant or background channels.The system of claim 1, wherein the cross-level submodule comprises a multi-head self-attention layer for windows configured to split feature maps into multiple sub-windows, calculate self-attention within local windows, and process the attention outputs.The system of claim 1, wherein the adaptive feature fusion module is configured to receive features from different network levels, generate adaptive weights by convolution operations, normalize weights using a sigmoid function, and combine features using the generated weights.The system of claim 1, wherein the neural network module comprises: a backbone network based on YOLOv5; a neck network implementing path aggregation; and a detection head for outputting road sign locations and classifications.The system of claim 4, wherein the multi-head self-attention layer of the window is configured to: process feature maps of size H × B × C, split features into local windows of size K × K, and independently calculate attention within each window.The system of claim 5, wherein the adaptive feature fusion module further comprises: multiple 1×1 convolutional layers; channel chaining operations; ReLU activation functions; and a final sigmoid layer for weight normalization.The system of claim 1, wherein the system is configured to recognize traffic signs of different scales, operate in real time at least 80 images per second, and achieve an average accuracy of at least 75% for traffic sign datasets.The system of claim 1, wherein the lightweight superficial information enhancement module is configured to: preserve spatial information while reducing computational complexity; enhance recognition of traffic sign features including color and shape; and output enhanced feature maps for subsequent processing phases.

Images