A traffic sign detection method based on transformer

Abstract

The application relates to a traffic sign detection method based on a Transformer, belonging to the field of image processing, S1: preparing a pre-training data set, which is divided into a training set and a test set; S2: constructing a traffic sign detection model based on the Transformer, including an information enhancement module: performing information enhancement on an input image, outputting feature maps A1, A2, A3 and A4; a Muti-Scale Transformer module: performing feature extraction and multi-scale feature fusion on the feature maps A1, A2, A3 and A4, outputting a feature map B; a target detection module: fusing the feature map B, generating a feature map F, and detecting the category and position of the traffic sign in the feature map F; S3: training and testing the model, and using the converged model to perform target detection on a to-be-detected image, and the application improves the performance of small target detection in traffic signs.

Description

Technical Field

[0001] This invention belongs to the field of image processing and relates to a traffic sign detection method based on Transformer. Background Technology

[0002] Object detection is one of the most fundamental and important tasks in computer vision. It typically utilizes deep neural networks to extract and model features from visual data and predict the location and category of target objects. Current deep learning methods attempt to solve object detection tasks as either classification or regression problems, or a combination of both.

[0003] The Transformer network was first proposed and applied to various scenarios and tasks in Natural Language Processing (NLP), achieving great success. In recent years, the Transformer has also been widely used in various visual tasks such as image classification, semantic segmentation, and object detection, achieving excellent results and, to some extent, surpassing Convolutional Neural Networks (CNNs). The self-attention mechanism is a core component of the Transformer network, which automatically establishes relationships between features by measuring the responses between features and reorganizing features based on the response values. Existing detection methods based on traditional Feature Processing Network (FPN) techniques typically achieve cross-scale feature fusion directly by interpolating and summing features, but their ability to model objects of different scales remains limited. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a traffic sign detection method based on Transformer, which can improve the performance of small target detection.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A traffic sign detection method based on Transformer includes the following steps:

[0007] S1: Prepare the pre-training dataset and divide it into training and test sets;

[0008] S2: Construct a traffic sign detection model based on Transformer, including an information enhancement module, a Muti-Scale Transformer module, and an object detection module; the information enhancement module is used to enhance the information of the input image and output feature maps A1, A2, A3, A4; the Muti-Scale Transformer module is used to perform feature extraction and multi-scale feature fusion on the feature maps A1, A2, A3, A4 and output a feature map B; the object detection module is used to fuse the feature map B, generate a feature map F, and detect the category and position of traffic signs in the feature map F;

[0009] S3: Use the training set and the test set to train and test the traffic sign detection model based on Transformer, and use the converged traffic sign detection model based on Transformer to perform object detection on the image to be detected.

[0010] Furthermore, in the step S1, obtain the publicly available traffic sign image dataset TT-100K; then select N images from the TT-100K dataset, where 100 < N < 1000, and after normalization processing, obtain images with a unified size as the training sample set S.

[0011] Furthermore, the information enhancement module includes three branches, specifically including:

[0012] Branch 1: Convert the original features into one-dimensional features through global average pooling operation to obtain the global information of a single channel, and then use a 1×1 convolution to adjust the number of channels and fuse the global information;

[0013] Branch 2: Use a 3×3 convolution to fuse the local information of the feature map;

[0014] Branch 3: Use a feature pyramid network to fuse three different scale features;

[0015] Finally, add and fuse the features output by the three branches to obtain enhanced features.

[0016] Furthermore, in Branch 3, for the feature pyramid network, use 5×5, 3×3, and 1×1 convolutions to fuse the three features.

[0017] Furthermore, the Muti-Scale Transformer module is used to perform feature extraction and multi-scale feature fusion on the feature maps A1, A2, A3, A4 and output a feature map B, and its implementation steps are as follows:

[0018] First, for the four input feature maps F1, F2, F3, and F4, positional embedding and level positional embedding are used. Positional embedding is the same as that used in the Transformer, used to learn the positional information of reference points in a single feature map. Level positional embedding is used in feature maps of different scales to find the relative position of a single reference point in different feature maps, achieving alignment of reference points at different scales. The level positional embedding is then normalized to restrict its position in the feature map, as shown in the following formula:

[0019] lvl_pos_embed=pos_embed+level_embed

[0020] Where pos_embed represents single-layer position embedding, level_embed represents inter-layer relative embedding, and lvl_pos_embed represents hierarchical position embedding;

[0021] Then, for the feature f at the nth level n The data is mapped to three spaces—Query, Key, and Value—through a linear layer, resulting in weight matrices Q, K, and V for each space. Self-attention is then calculated on these three weight matrices to obtain attention vectors for each scale. Finally, the attention vectors calculated at different scales are summed and averaged to obtain a cross-scale attention vector, as shown in the following formula:

[0022]

[0023] Among them, W m denoted by , where 'a' represents the similarity calculated using Q and K, 'L' represents the number of layers in the multi-scale feature hierarchy, 'I' represents the sequence length, 'i' and 'j' represent the index values ​​of different Q and K values, respectively, and 'MSAttn' represents the attention vector of the reference point x.

[0024] Furthermore, the target detection module inputs the feature map output by the Multi-Scale Transformer module into the RPN network for processing. The RPN network generates candidate boxes through anchor boxes and boundary regression, and obtains preliminary candidate regions. Then, the candidate regions are normalized through the ROIPooling layer, and a fully connected layer and a softmax classifier are used to determine whether the candidate feature map is a traffic signal sign and to give the corresponding probability, thereby realizing traffic sign detection.

[0025] The beneficial effects of this invention are as follows:

[0026] This invention fully utilizes branch networks to establish information flow between global and local information to learn the correlation between sub-regions and highlight regions containing the target. This invention also fully leverages the connection between multi-scale feature maps and the Transformer, addressing the Transformer's weakness in detecting small targets by fusing feature maps of different scales and employing an efficient self-attention mechanism to improve small target detection capabilities.

[0027] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0028] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0029] Figure 1 This is a simplified diagram of the structure of a Transformer-based traffic sign detection network model according to the present invention.

[0030] Figure 2 This is a structural diagram of the information enhancement module described in this invention;

[0031] Figure 3 This is a structural diagram of the Multi-Scale Transformer module described in this invention;

[0032] Figure 4 This is a structural diagram of the target detection module described in this invention. Detailed Implementation

[0033] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0034] Among them, the attached drawings are only for illustrative purposes, showing only schematic diagrams rather than physical diagrams, and should not be construed as a limitation to the present invention; in order to better illustrate the embodiments of the present invention, some components in the attached drawings will be omitted, enlarged or reduced, which does not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the attached drawings may be omitted.

[0035] In the attached drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "rear", etc. indicating the orientation or positional relationship, they are based on the orientation or positional relationship shown in the attached drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the positional relationship in the attached drawings are only for illustrative purposes and should not be construed as a limitation to the present invention. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to specific circumstances.

[0036] Please refer to Figures 1-4 , this embodiment provides a traffic sign detection method based on Transformer. The specific process of this method is as Figure 1 shown, and this method includes:

[0037] Step S1: Obtain a data set and then divide it into a training set and a test set;

[0038] Specifically, in this step S1, the obtained traffic sign image data set is the publicly available data set TT-100K;

[0039] Then select N images from the TT-100K data set, where 100 < N < 1000, perform normalization processing to unify the image size, that is, the height × width is h × w, and use it as the training sample set S;

[0040] Step S2: Construct a traffic sign detection model based on Transformer. This network model includes: an information enhancement module, a Muti-Scale Transformer module, and an object detection module; the training process of this network model includes the following steps:

[0041] S2-1: The information enhancement module performs information enhancement on the input image and outputs feature maps A1, A2, A3, A4;

[0042] S2-2: The Muti-Scale Transformer module performs feature extraction and multi-scale feature fusion on the feature maps A1, A2, A3, A4 and outputs the feature map B;

[0043] S2-3: The target detection module fuses feature map B to generate feature map F, and detects the category and location of traffic signs in feature map F.

[0044] Specifically, in this embodiment, step S2 includes:

[0045] The above-mentioned Transformer object detection model has the following network structure: Figure 1 It includes: an information enhancement module, a Multi-Scale Transformer module, and a target detection module.

[0046] The network structure of the aforementioned information enhancement module is as follows: Figure 2 The algorithm comprises three branches. Branch 1 first performs global average pooling to transform the original features into one-dimensional features, thereby obtaining global information for each channel. A 1×1 convolution is then used to adjust the number of channels and fuse the global information. Branch 2 uses a 3×3 convolution to fuse the local information of the feature map. Branch 3 employs a feature pyramid network to fuse features at three different scales, using 5×5, 3×3, and 1×1 convolutions respectively. The features output from the three branches are then summed and fused to obtain the enhanced feature.

[0047] The network structure of the Multi-Scale Transformer module described above is as follows: Figure 3 This can be achieved through the following steps:

[0048] For the four input feature maps F1, F2, F3, and F4, positional embedding and level positional embedding are used. Positional embedding is the same as that used in the Transformer, used to learn the positional information of reference points in a single feature map. Level positional embedding is used in feature maps of different scales to find the relative position of a single reference point in different feature maps, achieving alignment of reference points at different scales. Simultaneously, the level positional embedding is normalized to constrain its position in the feature maps. The formula is as follows:

[0049] lvl_pos_embed=pos_embed+level_embed

[0050] The feature f at the nth level nThe algorithm maps the data to three spaces: Query, Key, and Value, through a linear layer, resulting in weight matrices Q, K, and V. Self-attention is then calculated on these three weight matrices to obtain attention vectors for each scale. Finally, the attention vectors calculated at different scales are summed and averaged to obtain the cross-scale attention vector. The formula is as follows:

[0051]

[0052] Among them, W m denoted by , where 'a' represents the similarity calculated using Q and K, 'L' represents the number of layers in the multi-scale feature hierarchy, 'I' represents the sequence length, 'i' and 'j' represent the index values ​​of different Q and K values, respectively, and 'MSAttn' represents the attention vector of the reference point x.

[0053] Specifically, in this embodiment, the network structure of the target detection module described above is as follows: Figure 4 The feature map output from the Multi-Scale Transformer module is input into the RPN network for processing. The RPN network generates candidate boxes through anchor boxes and boundary regression, obtaining preliminary candidate regions. Then, the candidate regions are normalized through the ROIPooling layer, and a fully connected layer and a softmax classifier are used to determine whether the candidate feature map is a traffic signal sign and to give the corresponding probability, thus achieving accurate traffic sign detection.

[0054] Step S3: Train the traffic sign detection model constructed in step S2 using the training set obtained in step S1 to obtain a converged network model. Input the image to be detected described in step S1 into the network model to obtain the target detection result.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A traffic sign detection method based on Transformer, characterized in that: It includes the following steps: S1: Prepare a pre-training dataset and divide it into a training set and a test set; S2: Construct a traffic sign detection model based on Transformer, including an information enhancement module, a Muti-Scale Transformer module, and an object detection module; the information enhancement module is used to perform information enhancement on the input image and output feature maps A1, A2, A3, A4; the Muti-Scale Transformer module is used to perform feature extraction and multi-scale feature fusion on the feature maps A1, A2, A3, A4 and output a feature map B; the object detection module is used to fuse the feature map B, generate a feature map F, and detect the category and position of traffic signs in the feature map F; The Muti-Scale Transformer module is used to perform feature extraction and multi-scale feature fusion on the feature maps A1, A2, A3, A4 and output a feature map B. Its implementation steps are as follows: First, for the input 4 feature maps F1, F2, F3, F4, perform processing using positional encoding and hierarchical positional encoding. The positional encoding is the same as the positional encoding used in Transformer and is used to learn the position information of the reference point in a single feature map; the hierarchical positional encoding is used in feature maps of different scales to find the relative position of a single reference point in different feature maps and achieve the alignment of reference points at different scales; normalize the hierarchical positional encoding to limit the position of the hierarchical positional encoding in the feature map, and the formula is as follows: where pos_embed represents single-layer position embedding, level_embed represents inter-layer relative embedding, and lvl_pos_embed represents hierarchical position embedding; Then, for the features of the nth level The data is mapped to three spaces—Query, Key, and Value—through a linear layer, resulting in weight matrices Q, K, and V for each space. Self-attention is then calculated on these three weight matrices to obtain attention vectors for each scale. Finally, the attention vectors calculated at different scales are summed and averaged to obtain a cross-scale attention vector, as shown in the following formula: in, This represents the weight of each attention head. This represents the similarity calculated using Q and K. The number of layers represents the multi-scale features, I represents the sequence length, i and j represent the index values ​​of different Q and K respectively, and MSAttn represents the attention vector of the reference point x. S3: Use the training set and the test set to train and test the traffic sign detection model based on Transformer, and use the converged traffic sign detection model based on Transformer to perform object detection on the image to be detected.

2. The traffic sign detection method based on Transformer according to claim 1, characterized in that: In step S1, obtain the publicly available traffic sign image dataset TT-100K; then select N images from the TT-100K dataset, where 100 < N < 1000, and after normalization processing, obtain images with a unified size as the training sample set S.

3. The traffic sign detection method based on Transformer according to claim 1, characterized in that: The information enhancement module contains three branches, specifically including: Branch 1: Convert the original features into one-dimensional features through global average pooling operation to obtain global information of a single channel, and then use a 1×1 convolution to adjust the number of channels and fuse the global information; Branch 2: Use a 3×3 convolution to fuse the local information of the feature map; Branch 3: Use a feature pyramid network to fuse three different-scale features; Finally, add and fuse the features output by the three branches to obtain enhanced features.

4. The traffic sign detection method based on Transformer according to claim 3, characterized in that: In Branch 3, for the feature pyramid network, use 5×5, 3×3, and 1×1 convolutions to fuse the three features.

5. The traffic sign detection method based on Transformer according to claim 1, characterized in that: The target detection module inputs the feature map output by the Multi-Scale Transformer module into the Region Proposal Network (RPN) for processing. The RPN generates candidate boxes through anchor boxes and boundary regression, and obtains preliminary candidate regions. Then, the candidate regions are normalized through the ROIPooling layer, and a fully connected layer and a softmax classifier are used to determine whether the candidate feature map is a traffic signal sign and give the corresponding probability, thereby realizing traffic sign detection.

Images