CNN (Convolutional Neural Network) and Transform fused traffic sign detection method in complex environment

Abstract

The invention discloses a CNN and Transform fused traffic sign detection method in a complex environment, and the method comprises the steps: obtaining and marking traffic sign image data, and making a traffic sign detection data set; constructing a traffic sign detection model which is fused with a CNN (Convolutional Neural Network) and a Transform; training the constructed detection model by using the made traffic sign detection data set; and detecting an input to-be-detected image by using the trained detection model and outputting a detection result. Wherein the detection model fuses a CNN and a Transform, the CNN module is particularly effective for capturing high-frequency features such as edges and corners, and the Transform module is more suitable for capturing low-frequency features such as a global mode; besides, a locality module is further added to the model to enhance the local sensing ability, and meanwhile, a new module capable of simultaneously incorporating local and global context information is also added, so that the traffic sign detection efficiency and precision of the model in a complex environment are improved.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and intelligent transportation, and specifically to a traffic sign detection method in a complex environment by integrating CNN and Transformer. Background Art

[0002] In the context of rapid technological development and rapid socio-economic growth, intelligent transportation technology is gradually emerging and becoming a new focus of attention in the transportation industry. Faced with the growing demand for traffic management, traditional road safety inspection methods are difficult to adapt to current needs due to their low efficiency and high cost. Especially in rural roads where accidents frequently occur, highways where accidents are frequent, and intersections with complex traffic conditions, it is particularly critical and urgent to conduct a thorough inspection of road safety hazards. Traffic sign detection plays a vital role in the field of intelligent transportation systems. However, in actual road environments, accurate recognition of traffic signs faces many challenges. For example, different lighting conditions, complex background environments, aging and fading of traffic signs, and other factors may interfere with the recognition of traffic signs. These factors work together to make the task of traffic sign detection complex and challenging.

[0003] At present, traffic sign detection algorithms are mainly divided into two categories: traditional traffic sign detection algorithms and traffic sign detection algorithms based on deep learning. Traditional traffic sign detection algorithms mainly rely on manual feature extraction and classical computer vision techniques. These algorithms usually include color segmentation, shape detection and methods based on multi-feature fusion. Their advantages are relatively low computing resource consumption and have certain effects on traffic sign detection in simple scenes. However, the disadvantage of these methods is that they are sensitive to environmental changes, such as lighting conditions, occlusion and scale changes of signs, which will significantly affect the detection performance. In addition, traditional methods are relatively limited in feature extraction and are difficult to cope with traffic sign recognition tasks under complex backgrounds and changing conditions. Traffic sign detection algorithms based on deep learning have stronger feature expression and generalization capabilities by automatically learning features from a large amount of data. Deep learning methods, especially convolutional neural networks (CNNs), can extract high-level features from images and have better robustness to external factors such as lighting changes and occlusions. These methods are generally better than traditional methods in terms of traffic sign detection accuracy, especially in complex scenes. However, the disadvantage of deep learning methods is that they require a large amount of labeled data for training, and the model has many parameters, which leads to high consumption of computing resources and slow reasoning speed. In addition, the complexity of deep learning models also leads to longer training time and higher storage requirements. Summary of the invention

[0004] In order to solve the problem of poor effect of traffic sign detection in complex environments, the present invention provides a traffic sign detection method in complex environments that integrates CNN and Transformer to improve the detection accuracy of traffic signs.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0006] A traffic sign detection method in a complex environment integrating CNN and Transformer includes the following steps:

[0007] S1: Obtain traffic sign images and annotate them to create a traffic sign detection dataset, including training set, validation set and test set;

[0008] S2: Build a traffic sign detection model that integrates CNN and Transformer;

[0009] S3: Use the prepared traffic sign detection dataset to train the constructed detection model and save the optimal model;

[0010] S4: Input the traffic sign image into the trained traffic sign detection model for detection, and obtain the detection frame and target category of the traffic sign image;

[0011] Furthermore, the step S1 includes:

[0012] S101. Collect traffic sign image data using a drone, where the image is from a patrol perspective.

[0013] S102. Use labelimg tool to label the image as the basic data set, and use data enhancement methods to expand the data set, including affine transformation and adding noise, etc. Affine transformation mainly includes rotation, scaling, translation and tilt operations to change the geometric structure of the image, and adding noise mainly includes Gaussian noise and salt and pepper noise. This simulates the common shooting or transmission interference, insufficient lighting, aging of signs and other complex environments in the image, helping the model to better adapt to the noisy environment, improve the adaptability to real data, and let the model learn a wider feature distribution to improve the generalization of the model. At the same time, the TT00K data set is integrated. The resolution of the TT100K data set is 2048×2048 pixels, including 221 categories, of which 45 categories include at least 100 instances. The 45 categories with more than 100 instances are integrated into the traffic sign detection data set to form the final data set. Finally, the data set is divided into training set, validation set and test set according to the ratio of 8:1:1.

[0014] Furthermore, the step S2 includes:

[0015] S201. Build a traffic sign detection model that integrates CNN and Transformer, mainly including the introduction of Next Convolution Block (NCB) module, Next Transformer Block (NTB) module and Next HybridStrategy (NHS).

[0016] S202, the Next Convolution Block (NCB) module introduced NCB follows the general architecture of MetaFormer, which has been proven to be crucial to the Transformer block. It is constructed through MHCA and MLP layers, where MHCA, as an efficient attention mechanism, uses deployment-friendly convolution operations to achieve fast inference speed. The design of MHCA is inspired by MHSA, which jointly pays attention to information at different locations in a multi-head manner to effectively learn local features. The introduced NCB module can be expressed as follows:

[0017]

[0018] where z l-1 represents the input from the l-1 block, and z l are the outputs of MHCA and the lth NCB respectively.

[0019] S203, Multi-head Convolutional Attention (MHCA) is defined as follows:

[0020] MHCA(z)=Concat(CA1(z1),CA2(z2),…,CA h (z h ))W p

[0021] MHCA captures information from h parallel representation subspaces. h ] indicates that the input feature z is split into multiple forms in the channel dimension. MHCA is equipped with a projection layer (W p ) to promote information interaction between multiple heads. CA is a single-head convolutional attention, which can be defined as:

[0022] CA(z)=O(W·T {m,n} )

[0023] in

[0024] T {m,n} ∈z

[0025] In this formula, T m and T nrepresents the neighboring tags in the input feature z. O represents the inner product operation with W as the learnable parameter and the input patch. By iteratively optimizing the parameter W, O can effectively learn the connectivity between different tags in the local receptive field. CA is able to learn the affinity between different tags in the local receptive field by iteratively optimizing the trainable parameter W.

[0026] S204, the introduced Next Transformer Block (NTB) module uses CGA (Cascaded Group Attention), a new attention module inspired by group convolution in efficient CNN, which can be described as:

[0027]

[0028] Among them, the jth head calculates X ij Self-attention on X ij is the input feature X i The jth segmentation of X i =[X i1 ,X i2 ,…,X ih ], where 1≤j≤h, h is the total number of heads, is a projection layer that maps the input features into different subspaces, W i P It is a linear layer that projects the output features of the connection back to the same dimension as the input. Although using local features is more efficient and computationally efficient than using the complete features of each head, it is still necessary to promote the Q, K, and V layers to learn feature projections with richer information. The attention map of each head is calculated in a cascade manner, and the output of each head is added to the subsequent heads to gradually improve the feature representation:

[0029]

[0030] Among them, X′ ij is the jth input split point X ij With the (j-1)th head output When calculating self-attention, it replaces X ijAs the new input feature of the jth head. CGA enables self-attention to jointly capture local and global relationships, and further enhances the feature representation, so that the features of traffic signs can still be well extracted under complex environments such as rainy and snowy weather, complex lighting conditions, etc. Afterwards, the output features of CGA and MHCA are connected to mix high- and low-frequency information. Finally, an MLP layer is borrowed to extract more basic and unique features at the end. The local feed-forward (LFF) module is exclusively used in the first NTB block, while MLP is still used in the remaining NTB blocks. The LFF module is an inverted residual module that uses convolution technology, so it enhances the local visual effect of the network compared to the simple MLP. In short, the implementation of the first NTB can be stated as follows:

[0031]

[0032] The implementation of the rest of NTB can be expressed as follows:

[0033]

[0034]

[0035] S205, the introduced Next Hybrid Strategy (NHS) uses (N+1)*L hybrid paradigm to stack NCB and NTB, and the final pattern is (NCB×N+NTB×1)×L, where N NCBs and one NTB are stacked in sequence at each stage. Specifically, the Transformer block (NTB) is located at the end of each stage, enabling the model to learn global representations at a shallow level. Generally, N is taken as 4. Therefore, the final hybrid stacking pattern can be defined as:

[0036]

[0037] where i∈(1,2,3,4) represents the stage index, ψ represents the current NCB, Γ represents the identity layer when i=1, otherwise it is NTB, and finally, ∮ represents the operation of sequentially stacking stages.

[0038] Beneficial effects of the present invention:

[0039] A traffic sign detection method in complex environments that integrates CNN and Transformer is proposed, which improves the accuracy and efficiency of traffic sign detection in complex environments and conditions such as rainy and snowy weather, poor lighting, and sign aging. It can better assist drivers and unmanned driving, improve driving safety, and is also of great help in road inspections and other tasks.

[0040] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0042] Figure 1 It is a flow chart of the present invention;

[0043] Figure 2 This is a structural diagram of the detection model that integrates CNN and Transformer in the present invention;

[0044] Figure 3 This is the module structure diagram of the Next Convolution Block (NCB) of the present invention;

[0045] Figure 4 This is a structural diagram of the Multi-Head Convolutional Attention (MHCA) module of the present invention;

[0046] Figure 5 This is a block diagram of the Next Transformer Block (NTB) module of the present invention;

[0047] Figure 6 This is a structural diagram of the Cascaded Group Attention (CGA) module of the present invention;

[0048] Figure 7 This is a structural diagram of the Local Feed-Forward (LFF) module of the present invention; DETAILED DESCRIPTION

[0049] In order to make the technical solutions, purposes and advantages of the present invention clearer, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. According to the described embodiments of the present invention, all other embodiments that can be obtained by ordinary technicians in the field without creative work belong to the scope of protection of the present invention.

[0050] The present invention relates to a traffic sign detection method in a complex environment that integrates CNN and Transformer. The flow chart is as follows: Figure 1 As shown, including:

[0051] Step S1: Obtain traffic sign images and annotate them to create a traffic sign detection dataset, including a training set, a validation set, and a test set;

[0052] (1) In a specific embodiment of the present invention, traffic sign image data is first acquired. Specifically, the use of drones for data collection is to overcome the inconvenience and inefficiency of traditional manual data collection. The resolution of the collected image data is 1920*1080. At the same time, the image data is expanded by affine transformation, mainly including rotation, scaling, translation and tilt operations to change the geometric structure of the image and adding noise, mainly including Gaussian noise and salt and pepper noise. In this way, the common shooting or transmission interference, insufficient lighting, sign aging and other complex environments in the image are simulated to help the model better adapt to the noisy environment, improve the adaptability to real data, allow the model to learn a wider range of feature distributions, and improve the generalization of the model.

[0053] (2) When using labelimg annotation, the categories mainly include road signs, guide signs and their subdivision signs, etc. At the same time, the TT00K dataset is integrated. The resolution of the TT100K dataset images is 2048×2048 pixels, including 221 categories, of which 45 categories include at least 100 instances. The 45 categories with more than 100 instances are integrated into the traffic sign detection dataset to form the final dataset. Finally, the dataset is divided into training set, validation set and test set in a ratio of 8:1:1.

[0054] Step S2: construct a traffic sign detection model integrating CNN and Transformer;

[0055] (1) The construction of the traffic sign detection model integrating CNN and Transformer mainly includes the introduction of Next Convolution Block (NCB) module, Next Transformer Block (NTB) module and Next HybridStrategy (NHS). The specific structure is as follows: Figure 2 shown.

[0056] (2) The Next Convolution Block (NCB) module structure introduced is as follows Figure 3 As shown in Figure 2, NCB follows the general architecture of MetaFormer. NCB is constructed using MHCA and MLP layers in the MetaFormer paradigm. The introduced NCB module can be described as follows:

[0057]

[0058] where z l-1 represents the input from the l-1 block, and z l are the outputs of MHCA and the lth NCB respectively.

[0059] (3) The structure of Multi-Head Convolutional Attention (MHCA) is as follows Figure 4 As shown, the definition can be summarized as follows:

[0060] MHCA(z)=Concat(CA1(z1),CA2(z2),…,CA h (z h ))W p

[0061] MHCA captures information from h parallel representation subspaces. h ] indicates that the input feature z is split into multiple forms in the channel dimension. MHCA is equipped with a projection layer (W p ) to promote information interaction between multiple heads. CA is a single-head convolutional attention, which can be defined as:

[0062] CA(z)=O(W·T {m,n} )

[0063] in:

[0064] T {m,n} ∈z

[0065] In this formula, T m and T n Represents adjacent tags in the input feature z. O represents the inner product operation with W as the learnable parameter and the input patch. By iteratively optimizing the parameter W, O can effectively learn the connectivity between different tags in the local receptive field. CA is able to learn the affinity between different tags in the local receptive field by iteratively optimizing the trainable parameter W. Specifically, the implementation of MHCA is completed through group convolution (multi-head convolution) and point-by-point convolution. The head dimension is uniformly set to 32 in all MHCAs to achieve fast inference speed for various data types on TensorRT. In addition, efficient BatchNorm (BN) and ReLU activation functions are used in NCB instead of LaverNom (LN) and GELU in traditional Transformer blocks, which further speeds up the inference speed and enables better detection efficiency in complex environments.

[0066] (4) The introduced Next Transformer Block (NTB) module structure is as follows Figure 5As shown in Figure 2, NTB first uses the CGA (Cascaded Group Attention) module, which is a new attention module inspired by the group convolution in efficient CNN. Its structure is as follows: Figure 6 As shown, it can be described as:

[0067]

[0068] Among them, the jth head calculates X ij Self-attention on X ij is the input feature X i The jth segmentation of X i =[X i1 ,X i2 ,…,X ih ], where 1≤j≤h, h is the total number of heads, is a projection layer that maps the input features into different subspaces, W i P It is a linear layer that projects the output features of the connection back to the same dimension as the input. Although using local features is more efficient and computationally efficient than using the complete features of each head, it is still necessary to promote the Q, K, and V layers to learn feature projections with richer information. The attention map of each head is calculated in a cascade manner, and the output of each head is added to the subsequent heads to gradually improve the feature representation:

[0069]

[0070] Among them, X′′ ij is the jth input split point X ij With the (j-1)th head output When calculating self-attention, it replaces X ij As the new input feature of the jth head. CGA enables self-attention to jointly capture local and global relationships, and further enhances the feature representation, so that the features of traffic signs can still be well extracted under complex environments such as rainy and snowy weather and complex lighting conditions. Afterwards, the output features of CGA and MHCA are connected to mix high- and low-frequency information. Finally, an MLP layer is used to extract more basic and unique features at the end. The local feed-forward (LFF) module is exclusively used in the first NTB block, while MLP is still used in the remaining NTB blocks.

[0071] (5) The Local Feed-Forward (LFF) module is an inverted residual module that uses convolution technology. Therefore, compared with the simple MLP, it enhances the local visual effect of the network, can better extract local features, and achieve better results. Its structure is as follows Figure 7 As shown in the figure. It mainly includes two 1*1 convolutions and one 3*3 deep convolution, as well as "Seq2Img" and "Img2Seq" to increase the conversion between sequence and image feature maps. The calculation process of reintroducing deep convolution into the Transformer feedforward network can be expressed as

[0072]

[0073] in, is the kernel of the deep convolution. The input (i.e., sequence of tokens) is first reshaped into a feature map rearranged on a 2D grid. The feature map is then subjected to two 1*1 convolutions and one depthwise convolution. Afterwards, the feature map is reshaped into a string of tokens that are used as self-attention in the network's transformation layer.

[0074] (6) MLP is a feed-forward artificial neural network consisting of multiple layers (usually two or more fully connected layers). It mainly includes a BN layer, two 1*1 convolutions, and an activation function.

[0075] (7) In short, the implementation of the first NTB can be expressed as follows:

[0076]

[0077] The implementation of the rest of NTB can be expressed as follows:

[0078]

[0079]

[0080] (8) The introduced Next Hybrid Strategy (NHS) uses the (N+1)*L hybrid paradigm to stack NCB and NTB. The final pattern is (NCB×N+NTB×1)×L. Each stage stacks N NCBs and one NTB in sequence. Specifically, the Transformer block (NTB) is located at the end of each stage, enabling the model to learn global representations at a shallow level. Generally, N is taken as 4. Therefore, the final hybrid stacking pattern can be defined as:

[0081] ∮ i {[Γ(ψ(Z)×N i )]×L i}

[0082] where i∈(1,2,3,4) represents the stage index, ψ represents the current NCB, Γ represents the identity layer when i=1, otherwise it is NTB, and finally, ∮ represents the operation of sequentially stacking stages.

[0083] Step S3: Use the prepared traffic sign detection data set to train the constructed detection model and save the optimal model;

[0084] Step S4: input the traffic sign image into the trained traffic sign detection model for detection, and obtain the detection frame and target category of the traffic sign image;

[0085] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0086] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A traffic sign detection method in complex environments integrating CNN and Transformer, characterized in that: The method comprises: S1: Obtain traffic sign images and annotate them to create a traffic sign detection dataset, including training set, validation set and test set; S2: Build a traffic sign detection model integrating CNN and Transformer; S3: Use the prepared traffic sign detection dataset to train the constructed detection model and save the optimal model; S4: Input the traffic sign image into the trained traffic sign detection model for detection to obtain the detection frame and target category of the traffic sign image.

2. According to claim 1, a method for detecting traffic signs in complex environments by integrating CNN and Transformer, characterized in that: In step S1, traffic sign image data is collected and the image is annotated using the labelimg tool as a basic data set. At the same time, the data set is expanded using data enhancement methods, including affine transformation and adding noise, and the TT00K data set is integrated. The category data with more than 100 instances are integrated into the traffic sign detection data set to form the final data set. Finally, the data set is divided into a training set, a validation set, and a test set in a ratio of 8:1:

1.

3. The method for detecting traffic signs in complex environments by integrating CNN and Transformer according to claim 1, characterized in that: In the step S2, the traffic sign detection model integrating CNN and Transformer is constructed, which mainly includes introducing Next Convolution Block (NCB) module, Next Transformer Block (NTB) module and Next Hybrid Strategy (NHS).

4. The method for detecting traffic signs in complex environments by integrating CNN and Transformer according to claim 3, characterized in that: The introduced Next Convolution Block (NCB) module NCB follows the general architecture of MetaFormer. NCB is constructed with MHCA and MLP layers in the MetaFormer paradigm. The introduced NCB module can be described as follows: where z l-1 represents the input from the l-1 block, and z l are the outputs of MHCA and the lth NCB respectively.

5. The method for detecting traffic signs in complex environments by integrating CNN and Transformer according to claim 4, characterized in that: The Multi-Head Convolutional Attention (MHCA) definition can be summarized as follows: MHCA(z)=Concat(CA1(z1),CA2(z2),…,CA h ( z h ))W p MHCA captures information from h parallel representation subspaces, z = [z1, z2, …, z h ] means that the input feature z is split into multiple forms in the channel dimension, and a projection layer (W p ) to promote information interaction between multiple heads, CA is a single-head convolutional attention, which can be defined as: CA(z)=O(W·T {m,n} ) in T {m,n} ∈z In this formula, T m and T n Represents the adjacent labels in the input feature z, O represents the inner product operation with W as the learnable parameter and the input patch. By iteratively optimizing the parameter W, O can effectively learn the connectivity between different labels in the local receptive field. CA can learn the affinity between different labels in the local receptive field by iteratively optimizing the trainable parameter W.

6. The method for detecting traffic signs in complex environments by integrating CNN and Transformer according to claim 3, characterized in that: The introduced Next Transformer Block (NTB) module uses CGA (Cascaded Group Attention), a new attention module inspired by group convolution in efficient CNNs, which can be described as: Among them, the jth head calculates X ij Self-attention on X ij is the input feature X i The jth segmentation of X i =[X i1 ,X i2 ,…,X ih ], where 1≤j≤h, h is the total number of heads, is a projection layer that maps the input features into different subspaces, W i P It is a linear layer that projects the concatenated output features back to the same dimension as the input, computes the attention map of each head in a cascaded manner, and adds the output of each head to the subsequent heads to gradually improve the feature representation: Among them, X′ ij is the jth input split point X ij With the (j-1)th head output , it replaces X when calculating self-attention ij As the new input feature of the jth head, CGA enables self-attention to jointly capture local and global relationships and further enhance feature representation. After that, the output features of CGA and MHCA are concatenated to mix high- and low-frequency information. Finally, an MLP layer is used to extract more basic and unique features at the end. The local feed-forward LocalFeed-Forward (LFF) module is used exclusively in the first NTB block, while MLP is still used in the remaining NTB blocks. The LFF module is an inverted residual module that uses convolution technology, so compared with the simple MLP, it enhances the local visual effect of the network and can better extract image features. In short, the implementation of the first NTB can be described as follows: The implementation of the rest of NTB can be expressed as follows:

7. The method for detecting traffic signs in complex environments by integrating CNN and Transformer according to claim 3, characterized in that: The introduced Next Hybrid Strategy (NHS) uses the (N+1)*L hybrid paradigm to stack NCB and NTB. The final pattern is (NCB×N+NTB×1)×L. Each stage stacks N NCBs and one NTB in sequence. Specifically, the Transformer block (NTB) is located at the end of each stage, enabling the model to learn global representations at a shallow level. Generally, N is taken as 4, so the final hybrid stacking pattern can be defined as: where i∈(1,2,3,4) represents the stage index, ψ represents the current NCB, Γ represents the identity layer when i=1 and NTB otherwise, and finally, ∮ represents the operation of sequentially stacking stages.

Images