A traffic sign detection method fusing a transformer mechanism

Abstract

The application discloses a traffic sign detection method fusing a Transformer mechanism, and comprises the following steps: step 1, according to the Transformer mechanism, an encoding module Trans combining global features of an image is designed; step 2, a feature fusion network is built by replacing an ordinary convolution module with a GhostConv, a light and convenient linear operation of the GhostConv is used to realize a light and convenient extraction process of redundant features, and calculation power and memory occupation are released; and step 3, a simplified decoupled detection head Slim Decoupled Head is designed, classification and regression tasks are separated and analyzed, and the output capacity of the network model is strengthened. The application can complete detection of small traffic signs in a complex road scene with high accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a traffic sign detection method fusing a Transformer mechanism. BACKGROUND

[0002] In the road traffic scene, the visual perception technology of intelligent vehicles can help the automatic driving system to efficiently and accurately perceive the information of the road environment. Traffic sign recognition (TSR) is an important component module in the application of automatic driving and intelligent transportation, and is an inevitable demand for safe driving. Existing TSR methods can be roughly divided into two categories: traditional target detection methods and deep learning-based target detection methods. Traditional traffic sign detection and recognition algorithms are mostly based on color features, shape priors matching or edge detection methods. For example, traffic sign detection methods based on color space such as HSV and HSI are used to realize color feature matching; methods using histogram of oriented gradients (HOG) and scale invariant feature transform (SIFT) to detect shape features of traffic signs for matching. Although these algorithms can complete the detection of specific tasks, the hand-designed features result in poor robustness and high time complexity, and do not meet the needs of small traffic sign detection tasks in complex backgrounds. Deep learning-based target detection algorithms have strong feature extraction ability and model generalization ability, and are widely used in traffic sign detection tasks.

[0003] Current deep learning-based target detection algorithms can be divided into two-stage target detection algorithms and one-stage target detection algorithms. Typical two-stage detection algorithm models include R-CNN, SPPNet, Fast RCNN, etc.; typical one-stage detection algorithms include SSD, RetinaNet, YOLO, etc. These classic algorithms have achieved certain results in traffic sign detection tasks, but considering the size, imaging angle, and complex background of traffic signs in real road scenes, detecting small traffic signs in street scenes still has great challenges. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a traffic sign detection method fusing a Transformer mechanism, which can complete the detection of small traffic signs with high accuracy in complex road scenes.

[0005] To solve the above technical problems, the present application provides a traffic sign detection method fusing a Transformer mechanism, comprising the following steps:

[0006] Step 1: According to the Transformer mechanism, design an encoding module Trans combining image global features;

[0007] Step 2, use GhostConv instead of ordinary convolution module to build feature fusion network, through the lightweight and convenient linear operation of GhostConv, the extraction process of redundant features is lightened, and the calculation power and memory occupation are released;

[0008] Step 3, design a simplified decoupled detection head Slim DecoupledHead, separate the classification and regression tasks, and strengthen the output ability of the network model.

[0009] Preferably, in step 1, according to the Transformer mechanism, an encoding module Trans combined with image global features is designed, specifically: the constructed Trans includes two sub-layers, the first layer is a multi-head attention layer, and the second layer is a fully connected layer MLP, which are connected by residual connection at the input and output ends; wherein the multi-head attention layer Multi-headAttention is the core layer of Trans, which is integrated by multiple different self-attention, each branch of multi-head attention has different Q, K and V, wherein Q represents the specific attribute of the data being queried, K represents the keyword index of the data being queried, and V represents the specific data content obtained by querying, Q, K and V are obtained by linear transformation of image matrix, and the calculation formula of the attention weight obtained after the self-attention mechanism is as shown in formula (1):

[0010]

[0011] d k is the length of Q and K, used as a scaling factor to alleviate the gradient vanishing problem that may be brought by the introduction of the softmax function, after the inner product of Q and K is obtained to obtain the similarity representation, the weight is obtained by softmax, and finally acts on V to obtain the final output.

[0012] Preferably, in step 2, a GhostConv module is built in the feature fusion stage, which lightens the feature extraction process through a lightweight and convenient operation, and reduces the memory occupation of redundant features, specifically: a depth separable convolution is used as a linear operation to build a GhostConv module, assuming that the size of the input feature map is D H ×D w ×C, the size of the convolution kernel is D F ×D F ×C, the padding is 1, the number is N, and the total calculation amount of the ordinary convolution is:

[0013] D H ×D W ×D F ×D F ×C×N (2)

[0014] The total computation of depthwise separable convolution is divided into two parts. One part is the computation of depthwise convolution, as shown in formula (3):

[0015] D H ×D W ×D F ×D F ×C (3)

[0016] Another is the calculation of pointwise convolution, as shown in formula (4):

[0017] C×N×D H ×D W (4)

[0018] The total amount is calculated as shown in (5):

[0019] D H ×D W ×D F ×D F ×C+C×N×D H ×D W (5)

[0020] The ratio of depthwise separable convolution to ordinary convolution is shown in equation (6):

[0021]

[0022] The computational parameters of depthwise separable convolutions are much smaller than those of normal convolutions. Furthermore, in depthwise separable convolution operations, a 5×5 kernel is used to expand the receptive field and include richer contextual information; 1×1 convolutions are used for processing... The channel's feature information retains most of the original feature information of the input features, and also... The channel extracts features through depthwise separable convolution, which reduces the extraction of redundant features. Finally, the feature layers extracted from the two branches are merged and output through a cascade operation to obtain the final output feature layer.

[0023] Preferably, in step 3, the design of the Slim Decoupled Head parsing output result is as follows: the simplified decoupled detection head separates the classification task from the regression task, has three output detection heads, respectively: the classification output class_output, the regression output regression_output and the target output object_output; first, a 1x1 convolutional layer is used to reduce the dimension of the channel, then two parallel subnets, wherein class_output belongs to the classification subnet, which predicts the category of the object in the target frame, regression_output and object_output belong to the regression subnet, wherein regression_output detects the coordinate information (x, y, w, h) of the target frame, and object_output judges whether the target frame framed is foreground or background; a full connection detection head is used in the class_output detection branch, and a convolution detection head is used in the other two detection branches; finally, the three output detection heads are fused together through the concat operation to obtain the final output result.

[0024] The beneficial effects of the present application are: the present application proposes a traffic sign detection method TRD-YOLO based on YOLOv5, which has rich context awareness and small target detection ability, and integrates Trans Module based on the Transformer mechanism, which can extract global features, solve the problem of complex image background and easy confusion of foreground and background, and the core Self-Attention mechanism can help the network focus on the salient region of the image and more accurately extract key information; the GhostConv module is used for light and convenient linear operation to calculate the redundant part of feature extraction, which simplifies the feature information and releases a large amount of computing resources and memory resources, greatly improving the computing efficiency of the network model; the design of Slim Decoupled Head more specifically uses the feature information extracted by the network model, separates the classification task from the regression task, and enhances the ability of the network model to parse the output; through various experiments of the present application, the TRD-YOLO model can complete the detection of small traffic signs in a complex road scene with high accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 It is the overall structure schematic diagram of the TRD-YOLO of the present application.

[0026] Figure 2 It is the structure schematic diagram of the image global feature encoding module Trans Module of the present application.

[0027] Figure 3 A GhostConv schematic for the present application.

[0028] Figure 4 A Slim Decoupled Head structure diagram for the present application.

[0029] Figure 5 An application data set for the present application. DETAILED DESCRIPTION

[0030] A traffic sign detection method fusing a Transformer mechanism, comprising the following steps:

[0031] Step 1, according to the Transformer mechanism, design an encoding module Trans Module combined with image global features;

[0032] Step 2, use GhostConv to replace the ordinary convolution module to build a feature fusion network, through the lightweight linear operation of GhostConv to lightweight the redundant feature extraction process, release the computing power and memory occupation;

[0033] Step 3, design a Slim Decoupled Head, separate the classification and regression tasks, and strengthen the output ability of the network model.

[0034] The present application selects a large-scale traffic sign data set (TT100K) jointly produced by Tsinghua University and Tencent. The TT100K data set contains a total of 221 types of traffic signs, and the resolution of each data sample is 2048x2048. Compared with the GTSDB German traffic sign data set and the CCTSDB (China traffic sign detection data set), the data set contains various real road scenes under different light and weather conditions, such as Figure 5 As shown, the traffic signs are generally small, far away, and have a complex background, which has a certain detection difficulty. The original data set has a total of 6105 training data and 3107 test data, according to experimental analysis, there are a large number of traffic signs with extremely low proportion in the original data set, which cannot be effectively learned. In view of this problem, the present application extracts the annotation information in the data set from the official json file, and selects 45 types of traffic signs with a proportion greater than 100 to balance the sample difference problem caused by different types of traffic signs in the data set. The data set is shown in Figure 5 As shown, the class names are shown in Table 1, wherein the signs starting with "w" are warning signs, the signs starting with "p" are prohibition signs, and the signs starting with "i" are indication signs. The signs with a symbol at the end represent the numerical size, such as pl50, pl60 and pl100.

[0035] Table 1: Class name table

[0036]

[0037]

[0038] Figure 1 is the overall structure of the TRD-YOLO model, which is improved in the backbone network part, the neck part and the detection head part on the basis of the YOLOv5s network. We first design the image global feature coding module Trans based on the Transformer mechanism, introduce the global feature extraction capability, and increase the comprehensiveness and robustness of the model in extracting features; then, in the feature fusion stage, the GhostConv module of the application is built by referring to the idea of GhostNet, which reduces the invalid consumption of computing resources and memory resources by lightening the calculation of redundant parts in the feature extraction process, and speeds up the training of the network; finally, in order to further improve the output ability of the algorithm, the Slim Decoupled Head is designed to analyze the output results, and the classification and regression ability of the model is improved.

[0039] Figure 2 The image global feature coding module Trans is shown, each Trans contains two sub-layers, the first layer is a multi-head attention layer, and the second layer is a fully connected layer (MLP), which are connected by residual connection at the input and output ends to avoid the degradation problem of deep network. Among them, the multi-head attention layer (Multi-head Attention) is the core layer of the Trans Module, which is equivalent to integrating multiple different self-attention (Self-Attention), as shown in formula (7):

[0040] MultiHeadAttn(Q,K,V)=Concat(head1,…,head n )W o

[0041]

[0042] Therefore, the multi-head attention can balance according to the weight of different self-attention, so that the network model can learn more diversified feature information.

[0043] Based on YOLOv5, we add Trans in the deep backbone network. Because in the shallow network layer, the feature map resolution is too large, the calculation cost and memory cost caused by adding Trans are very huge, and the semantic information in the shallow feature is not rich, Trans cannot play a good effect of enhancing the association of semantic information, but may lose some context information, while in the deep network, it contains rich semantic information, which can better play the advantage of Trans in perceiving global features.

[0044] Figure 3 The schematic diagram of the GhostConv module is shown, and the GhostConv module is built by using deep separable convolution as linear operation instead of ordinary convolution by referring to the idea of GhostNet.

[0045] First, the essential feature map is obtained by ordinary convolution operation, then the deep separable convolution is performed on the feature map of 1 / 2 channels to obtain the Ghost feature map (redundant part), and finally the essential feature map and the Ghost feature map are concatenated to obtain the same number of feature maps as the output of the ordinary convolution operation.

[0046] Figure 4 is the Slim Decoupled Head module designed by us, and we use fewer convolution layers to build the decoupled head, which not only ensures that the parameter quantity will not be too large, but also ensures the detection accuracy. The Slim Decoupled Head has three output detection heads in total, which are class_output, regression_output and object_output. First, we reduce the channel dimension through a 1×1 convolution layer, and then two parallel subnets. The class_output belongs to the classification subnet, which mainly predicts the category of the object in the target frame; the regression_output and the object_output belong to the regression subnet, in which the regression_output mainly detects the coordinate information (x, y, w, h) of the target frame, and the object_output mainly judges whether the target frame in the frame is foreground or background. We use a fully connected detection head in the class_output detection branch, and a convolution detection head in the other two detection branches. Finally, the three output detection heads are fused together through the concat operation to obtain the final output result.

[0047] We inherit the hyperparameters in YOLOv5 to train the dataset, the input image size is 640×640, the batch size is set to 8, the num_workers is set to 6, the iteration is 600 epochs, the SGD optimizer with momentum of 0.8 is used for training, and the initial learning rate is 1×10 -2 .

[0048] Table 2 is the experimental results of the ablation experiment of the present application. Analysis shows that the present application has rich context awareness and small target detection capability, integrates Trans based on the Transformer mechanism, can extract global features, solves the problem of complex image background and easy confusion of foreground and background, and the core Self-Attention mechanism can help the network focus on the salient region of the image and more accurately extract key information. The GhostConv module uses lightweight linear operations to calculate the redundant part of the feature extraction, simplifies the feature information, releases a large amount of computing resources and memory resources, and greatly improves the computing efficiency of the network model. The design of Slim DecoupledHead more targetedly uses the feature information extracted by the network model, separates the classification task and the regression task, and enhances the ability of the network model to analyze the output. After various experiments of the present application, the TRD-YOLO model can detect small traffic signs in complex road scenes with high accuracy, and all kinds of evaluation indexes are improved, the precision is finally improved by 3%, the recall is finally improved by 2.1%, and the average precision (mAP) is finally improved by 3.5%, which improves the ability of the YOLOv5 model to detect small targets in complex backgrounds.

[0049] Table 2 is the experimental results of the ablation experiment of the present application. Analysis shows that the present application has rich context awareness and small target detection capability, integrates Trans based on the Transformer mechanism, can extract global features, solves the problem of complex image background and easy confusion of foreground and background, and the core Self-Attention mechanism can help the network focus on the salient region of the image and more accurately extract key information. The GhostConv module uses lightweight linear operations to calculate the redundant part of the feature extraction, simplifies the feature information, releases a large amount of computing resources and memory resources, and greatly improves the computing efficiency of the network model. The design of Slim DecoupledHead more targetedly uses the feature information extracted by the network model, separates the classification task and the regression task, and enhances the ability of the network model to analyze the output. After various experiments of the present application, the TRD-YOLO model can detect small traffic signs in complex road scenes with high accuracy, and all kinds of evaluation indexes are improved, the precision is finally improved by 3%, the recall is finally improved by 2.1%, and the average precision (mAP) is finally improved by 3.5%, which improves the ability of the YOLOv5 model to detect small targets in complex backgrounds, as shown in Table 2.

[0050]

[0051]

Claims

1. A traffic sign detection method integrating the Transformer mechanism, characterized in that, Includes the following steps: Step 1: Based on YOLOv5, Trans is added to the deep backbone network. According to the Transformer mechanism, an encoding module Trans combining global image features is designed. The constructed Trans contains two sub-layers. The first layer is a multi-head attention layer and the second layer is a fully connected layer MLP. The input and output are connected by residual connections. Among them, the multi-head attention layer is the core layer of Trans. It is composed of multiple different self-attention layers. Each branch of the multi-head attention layer has different Q, K, and V. Q represents the specific attributes of the queried data, K represents the keyword index of the data being queried, and V represents the specific data content retrieved. Q, K, and V are all obtained by linear transformation of the image matrix. The formula for calculating the weighted attention weights obtained by the self-attention mechanism is shown in (1). The lengths of Q and K are used as scaling factors to mitigate the gradient vanishing problem that may be caused by introducing the softmax function. After taking the inner product of Q and K to obtain the similarity representation, the weights are obtained through softmax and finally applied to V to obtain the final output. Step 2: Use GhostConv to build a feature fusion network instead of ordinary convolutional modules. GhostConv's lightweight and convenient linear operation reduces the extraction process of redundant features, freeing up computing power and memory usage. Step 3: Design a simplified decoupled detection head to separate the classification and regression tasks and enhance the output capability of the network model.

2. The traffic sign detection method integrating the Transformer mechanism as described in claim 1, characterized in that, In step 2, the GhostConv module is built during the feature fusion stage. This lightweight and convenient operation reduces the memory footprint of redundant features during feature extraction. Specifically, depthwise separable convolution is used as a linear operation to build the GhostConv module. Assuming the size of the input feature map is... The size of the convolution kernel is With padding of 1 and a quantity of N, the total computational cost of a regular convolution is: The total computation of depthwise separable convolution is divided into two parts. One part is the computation of depthwise convolution, as shown in formula (3): Another is the calculation of pointwise convolution, as shown in formula (4): The total amount is calculated as shown in (5): The ratio of depthwise separable convolution to ordinary convolution is shown in equation (6): The computational parameters of depthwise separable convolutions are much smaller than those of normal convolutions. Furthermore, in depthwise separable convolution operations, a 5×5 kernel is used to expand the receptive field and include richer contextual information; 1×1 convolutions are used for processing... The channel's feature information retains most of the original feature information of the input features, and also... The channel extracts features through depthwise separable convolution, which reduces the extraction of redundant features. Finally, the feature layers extracted from the two branches are merged and output through a cascade operation to obtain the final output feature layer.

3. The traffic sign detection method integrating the Transformer mechanism as described in claim 1, characterized in that, In step 3, the design of the Slim Decoupled Head for parsing the output is as follows: The simplified decoupled detection head separates the classification and regression tasks, and has three output detection heads: classification output (class_output), regression output (regression_output), and object output (object_output). First, a 1×1 convolutional layer is used to reduce the channel dimension, followed by two parallel sub-networks. The class_output sub-network predicts the category of the object within the bounding box, while the regression_output and object_output sub-networks detect the coordinate information (x, y, w, h) of the bounding box, and the object_output determines whether the bounding box is foreground or background. A fully connected detection head is used in the class_output detection branch, while convolutional detection heads are used in the other two detection branches. Finally, the three output detection heads are fused together through a concat operation to obtain the final output result.

Images