Traffic sign detection method based on multi-scale feature fusion

Abstract

The invention discloses a traffic sign detection method based on multi-scale feature fusion, and the method comprises the steps: S1, constructing a data set which comprises a plurality of traffic sign pictures and corresponding labels; s2, constructing a detection model, and importing pictures of the data set in the S1 into a chromatic aberration feature extraction module to obtain chromatic aberration features; s3, the chromatic aberration features in the S2 are imported into backbone feature extraction, and extraction features of different scales are obtained; s4, the extracted features in the S3 are imported into a multi-view collaborative fusion module, and multi-view assisted fusion features are obtained; s5, performing residual connection on the multi-view collaborative fusion features and the chromatic aberration features, inputting the features into a detection head, and obtaining prediction output; and S6, optimizing model parameters. According to the method, residual connection is carried out on the color difference features and the multi-view collaborative fusion features, so that image completion is carried out in the recognition process in combination with color information and visual information of the traffic sign pictures, and a more accurate detection result can be obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and specifically is a traffic sign detection method based on multi-scale feature fusion. Background Art

[0002] The development of autonomous driving is of great significance to reducing the occurrence of traffic accidents, and traffic sign detection and recognition is an important part of autonomous driving. The significance of traffic sign detection and recognition is mainly reflected in the following aspects: improving traffic safety, intelligent traffic management, and the development of automatic timing and counting. However, how to efficiently, accurately and automatically detect and recognize numerous traffic signs on complex traffic roads is a difficult problem.

[0003] Traffic sign detection and recognition is an important application scenario for target detection and recognition. In the development of traffic sign detection and recognition technology, traffic sign detection and recognition algorithms based on deep learning have become the main research direction. As the detection and recognition accuracy continues to improve, the number of parameters in the network model also increases, which leads to a decrease in detection speed, aggravating the difficulty of model deployment, and the problem of insufficient detection and recognition accuracy due to the small size of traffic signs and complex traffic road environment backgrounds. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes a traffic sign detection method based on multi-scale feature fusion, comprising the following steps: Step S1: construct a data set, which includes several traffic sign images and corresponding labels; Step S2: construct a detection model based on the YOLOv8 network, which includes a color difference feature extraction module, a backbone feature extraction module, a multi-view collaborative fusion module and a detection head, and import the image of the data set in step S1 into the color difference feature extraction module to obtain the color difference feature; Step S3: importing the color difference feature in step S2 into the backbone feature extraction module to obtain extraction features of different scales; Step S4: Import the extracted features of different scales in step S3 into the multi-view collaborative fusion module to obtain multi-view assisted fusion features; in this process, the multi-view assistance module performs data enhancement on the extracted features to obtain multi-view features of different view angles and multi-distance features at different distances, and uses the multi-view features as supplementary semantics to supplement the multi-distance features to obtain multi-view collaborative features of different scales, and performs channel fusion on the multi-view collaborative features of different scales to obtain multi-view collaborative fusion features; Step S5: performing dimension conversion on the multi-view collaborative fusion features in step S4 and the color difference features in step S2, adjusting the multi-view collaborative fusion features and the color difference features to a unified dimension, performing residual connection on the adjusted multi-view collaborative fusion features and the color difference features, and inputting them into the detection head to obtain a prediction output; Step S6: Construct a loss function and minimize the loss function to optimize the model parameters.

[0005] Furthermore, step S2 is specifically as follows: Step S21: The color difference feature extraction module assigns different weight coefficients to the red color difference map and the blue color difference map of the input image based on the three-component color difference method, and calculates the red, blue, and yellow color difference maps of the image, which are expressed as: ; ; ; in, Indicates that the input image is at position The intensity value of the red component, Indicates that the input image is at position The intensity value of the blue component of Indicates that the input image is at position The intensity value of the green component of represents the red color difference map, represents the blue color difference diagram, Represents the yellow color difference diagram; Step S22: Based on the image features of the input image Adaptively adjust the weights of the red color difference map, blue color difference map, and yellow color difference map based on image features To calculate the weight; picture features This includes the lighting, contrast, and color distribution of the input image; Step S23: weighted concatenation of the red color difference map, the blue color difference map, and the yellow color difference map is performed using adaptive weights to obtain color difference features.

[0006] Furthermore, step S4 is specifically as follows: Step S41: the multi-view assisting module performs data enhancement on the extracted features to obtain supplementary features of different viewing angles and supplementary features of views at different distances; wherein the data enhancement includes scaling, enlarging, diagonal flipping and vertical flipping operations; Step S42: down-sampling the perspective supplementary features of different perspectives, aligning them with the original feature map, that is, the corresponding extracted features, and then convolving the perspective supplementary features and the extracted features to compress the channel dimension to obtain the perspective compression features, and then inputting the perspective compression features into the multivariate modular multiplication unit of three parallel tensors to calculate the perspective attention weights, and splicing the perspective supplementary features and the extracted features based on the attention weights to obtain multi-perspective features; Step S42: down-sample the distance supplementary features of different perspectives, align them with the original feature map, that is, the corresponding extracted features, and then convolve the distance supplementary features and the extracted features to compress the channel dimension to obtain the distance compression features, and then input the distance compression features into three multivariate modular multiplication units of parallel tensors different from step S41 to calculate the distance attention weights, and splice the distance supplementary features and the extracted features based on the attention weights to obtain multiple distance features; Step S43: using the multi-view feature as a supplementary semantic to supplement the multi-view feature so as to obtain multi-view collaborative features of different scales; specifically: The multi-view features are processed by channel attention and spatial attention in turn to obtain supplementary features, and the supplementary features are concatenated with the multi-distance features in the channel dimension, and then further fused by convolution to obtain multi-view collaborative features of different scales; Step S44: performing channel fusion on multi-view collaborative features of different scales to obtain multi-view collaborative fusion features.

[0007] Furthermore, the process of obtaining multi-view features in step S42 is expressed as: ; ; ; ; ; in, represents the perspective supplement feature, represents the linear activation function, Represents a splicing operation, represents the convolution operation, represents the extracted features, represents the perspective supplement feature under the first perspective, represents the perspective supplement feature under the second perspective, represents the visual compression feature, represents the Sigmoid activation function, , and represents the weight matrix of the first multivariate modular multiplication unit, represents the visual attention weight of the first multivariate modular multiplication unit, , and represents the weight matrix of the second multivariate modular multiplication unit, represents the perspective attention weight of the second multivariate modular multiplication unit, , and represents the weight matrix of the third multivariate modular multiplication unit, represents the perspective attention weight of the third multivariate modular multiplication unit, Represents multi-view features, Represents element-wise product.

[0008] Further, step S44 is specifically as follows: The multi-view collaborative features of different scales are divided into n blocks along the channel dimension, and the number of channels in each block is the same; each block of multi-view collaborative features is further divided into two parts along the channel dimension, one part is used for channel-based interaction with the multi-view collaborative features of adjacent blocks, and the other part is used for output; the outputs of all blocks are re-joined to obtain multi-view collaborative fusion features.

[0009] Furthermore, the backbone feature extraction module in step S2 includes a first convolutional layer, a second convolutional layer, a first C2f module, a third convolutional layer, a second C2f module, a fourth convolutional layer, a third C2f module, a fifth convolutional layer, a fourth C2f module and an SPPF module connected in series in sequence; wherein the second C2f module, the third C2f module and the SPPF module respectively output extracted features of different scales.

[0010] Furthermore, the second C2f module, the third C2f module and the fourth C2f module use deformable convolution to dynamically sample the input.

[0011] 1) The present invention can more comprehensively retain color information and enhance the model's ability to express traffic sign color features by extracting color difference features from traffic sign images and using the color difference features as residual input for detection, thereby improving recognition accuracy. Furthermore, the color of traffic sign images may change under different lighting conditions. By using color difference features as residual input, the model can better adapt to lighting changes, reduce the impact of lighting on recognition results, and help the model better focus on the color features of the traffic sign itself, thereby enhancing the model's anti-interference ability. It can also simplify the model structure to a certain extent and reduce the number of model parameters. This makes it easier for the model to converge during training, and the training time is shorter, thereby improving the efficiency of model training.

[0012] 2) The multi-view collaborative fusion module of the present invention performs implicit interaction and semantic association mining on features at different viewing angles and distances through two-stage attention, which makes up for the difference between the two and makes the distinction between traffic sign images and background environments clearer.

[0013] 3) The present invention performs residual connection on the color difference features and the multi-view collaborative fusion features, thereby combining the color information and visual information of the traffic sign image to complete the image during the recognition process, thereby obtaining more accurate detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 The figure is a flowchart of the steps of a traffic sign detection method based on multi-scale feature fusion. DETAILED DESCRIPTION

[0015] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, in the description of the present invention, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate the orientation or position relationship based on the orientation or position relationship shown in the accompanying drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "No. 1", "No. 2" and "No. 3" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. The present invention is further explained below in conjunction with specific embodiments.

[0016] Reference Figure 1 , a traffic sign detection method based on multi-scale feature fusion, comprising the following steps: Step S1: construct a data set, which includes several traffic sign images and corresponding labels; Step S2: construct a detection model based on the YOLOv8 network, which includes a color difference feature extraction module, a backbone feature extraction module, a multi-view collaborative fusion module and a detection head, and import the image of the data set in step S1 into the color difference feature extraction module to obtain the color difference feature; Step S3: importing the color difference feature in step S2 into the backbone feature extraction module to obtain extraction features of different scales; Step S4: Import the extracted features of different scales in step S3 into the multi-view collaborative fusion module to obtain multi-view assisted fusion features; in this process, the multi-view assistance module performs data enhancement on the extracted features to obtain multi-view features of different view angles and multi-distance features at different distances, and uses the multi-view features as supplementary semantics to supplement the multi-distance features to obtain multi-view collaborative features of different scales, and performs channel fusion on the multi-view collaborative features of different scales to obtain multi-view collaborative fusion features; Step S5: performing dimension conversion on the multi-view collaborative fusion features in step S4 and the color difference features in step S2, adjusting the multi-view collaborative fusion features and the color difference features to a unified dimension, performing residual connection on the adjusted multi-view collaborative fusion features and the color difference features, and inputting them into the detection head to obtain a prediction output; Step S6: Construct a loss function and minimize the loss function to optimize the model parameters.

[0017] Furthermore, step S2 is specifically as follows: Step S21: The color difference feature extraction module assigns different weight coefficients to the red color difference map and the blue color difference map of the input image based on the three-component color difference method, and calculates the red, blue, and yellow color difference maps of the image, which are expressed as: ; ; ; in, Indicates that the input image is at position The intensity value of the red component, Indicates that the input image is at position The intensity value of the blue component of Indicates that the input image is at position The intensity value of the green component of represents the red color difference map, represents the blue color difference diagram, Represents the yellow color difference diagram; Step S22: Based on the image features of the input image Adaptively adjust the weights of the red color difference map, blue color difference map, and yellow color difference map based on image features To calculate the weight; picture features This includes the lighting, contrast, and color distribution of the input image; Step S23: weighted concatenation of the red color difference map, the blue color difference map, and the yellow color difference map is performed using adaptive weights to obtain color difference features.

[0018] Furthermore, step S4 is specifically as follows: Step S41: the multi-view assisting module performs data enhancement on the extracted features to obtain supplementary features of different viewing angles and supplementary features of views at different distances; wherein the data enhancement includes scaling, enlarging, diagonal flipping and vertical flipping operations; Step S42: down-sampling the perspective supplementary features of different perspectives, aligning them with the original feature map, that is, the corresponding extracted features, and then convolving the perspective supplementary features and the extracted features to compress the channel dimension to obtain the perspective compression features, and then inputting the perspective compression features into the multivariate modular multiplication unit of three parallel tensors to calculate the perspective attention weights, and splicing the perspective supplementary features and the extracted features based on the attention weights to obtain multi-perspective features; Step S42: down-sample the distance supplementary features of different perspectives, align them with the original feature map, that is, the corresponding extracted features, and then convolve the distance supplementary features and the extracted features to compress the channel dimension to obtain the distance compression features, and then input the distance compression features into three multivariate modular multiplication units of parallel tensors different from step S41 to calculate the distance attention weights, and splice the distance supplementary features and the extracted features based on the attention weights to obtain multiple distance features; Step S43: using the multi-view feature as a supplementary semantic to supplement the multi-view feature so as to obtain multi-view collaborative features of different scales; specifically: The multi-view features are processed by channel attention and spatial attention in turn to obtain supplementary features, and the supplementary features are concatenated with the multi-distance features in the channel dimension, and then further fused by convolution to obtain multi-view collaborative features of different scales; Step S44: performing channel fusion on multi-view collaborative features of different scales to obtain multi-view collaborative fusion features.

[0019] Furthermore, the process of obtaining multi-view features in step S42 is expressed as: ; ; ; ; ; in, represents the perspective supplement feature, represents the linear activation function, Represents a splicing operation, represents the convolution operation, represents the extracted features, represents the perspective supplement feature under the first perspective, represents the perspective supplement feature under the second perspective, represents the visual compression feature, represents the Sigmoid activation function, , and represents the weight matrix of the first multivariate modular multiplication unit, represents the visual attention weight of the first multivariate modular multiplication unit, , and represents the weight matrix of the second multivariate modular multiplication unit, represents the perspective attention weight of the second multivariate modular multiplication unit, , and represents the weight matrix of the third multivariate modular multiplication unit, represents the perspective attention weight of the third multivariate modular multiplication unit, Represents multi-view features, Represents element-wise product.

[0020] Further, step S44 is specifically as follows: The multi-view collaborative features of different scales are divided into n blocks along the channel dimension, and the number of channels in each block is the same; each block of multi-view collaborative features is further divided into two parts along the channel dimension, one part is used for channel-based interaction with the multi-view collaborative features of adjacent blocks, and the other part is used for output; the outputs of all blocks are re-joined to obtain multi-view collaborative fusion features.

[0021] Furthermore, the backbone feature extraction module in step S2 includes a first convolutional layer, a second convolutional layer, a first C2f module, a third convolutional layer, a second C2f module, a fifth convolutional layer, a third C2f module, a fourth convolutional layer, a fourth C2f module and an SPPF module connected in series in sequence; wherein the second C2f module, the third C2f module and the SPPF module respectively output extracted features of different scales.

[0022] Furthermore, the second C2f module, the third C2f module and the fourth C2f module use deformable convolution to dynamically sample the input.

[0023] The specific implementation modes as described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation mode of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A traffic sign detection method based on multi-scale feature fusion, characterized in that: The following steps are involved: Step S1: construct a data set, which includes several traffic sign images and corresponding labels; Step S2: construct a detection model based on the YOLOv8 network, which includes a color difference feature extraction module, a backbone feature extraction module, a multi-view collaborative fusion module and a detection head, and import the image of the data set in step S1 into the color difference feature extraction module to obtain the color difference feature; Step S3: importing the color difference feature in step S2 into the backbone feature extraction module to obtain extraction features of different scales; Step S4: importing the extracted features of different scales in step S3 into the multi-view collaborative fusion module to obtain multi-view assisted fusion features; In this process, the multi-view assistance module performs data enhancement on the extracted features to obtain multi-view features at different viewpoints and multi-distance features at different distances, and uses the multi-view features as supplementary semantics to supplement the multi-distance features to obtain multi-view collaborative features at different scales. The multi-view collaborative features at different scales are channel-fused to obtain multi-view collaborative fusion features. Step S5: performing dimension conversion on the multi-view collaborative fusion features in step S4 and the color difference features in step S2, adjusting the multi-view collaborative fusion features and the color difference features to a unified dimension, performing residual connection on the adjusted multi-view collaborative fusion features and the color difference features, and inputting them into the detection head to obtain a prediction output; Step S6: Construct a loss function and minimize the loss function to optimize the model parameters.

2. The traffic sign detection method based on multi-scale feature fusion according to claim 1, characterized in that: Step S2 is specifically as follows: Step S21: The color difference feature extraction module assigns different weight coefficients to the red color difference map and the blue color difference map of the input image based on the three-component color difference method, and calculates the red, blue, and yellow color difference maps of the image, which are expressed as: ； ； ； in, Indicates that the input image is at position The intensity value of the red component, Indicates that the input image is at position The intensity value of the blue component of Indicates that the input image is at position The intensity value of the green component of represents the red color difference map, represents the blue color difference diagram, represents the yellow color difference diagram; Step S22: Based on the image features of the input image Adaptively adjust the weights of the red color difference map, blue color difference map, and yellow color difference map based on image features To calculate the weight; picture features This includes the lighting, contrast, and color distribution of the input image; Step S23: weighted concatenation of the red color difference map, the blue color difference map, and the yellow color difference map is performed using adaptive weights to obtain color difference features.

3. The traffic sign detection method based on multi-scale feature fusion according to claim 1, characterized in that: Step S4 is specifically as follows: Step S41: the multi-view assisting module performs data enhancement on the extracted features to obtain supplementary features of different viewing angles and supplementary features of views at different distances; wherein the data enhancement includes scaling, enlarging, diagonal flipping and vertical flipping operations; Step S42: down-sampling the perspective supplementary features of different perspectives, aligning them with the original feature map, that is, the corresponding extracted features, and then convolving the perspective supplementary features and the extracted features to compress the channel dimension to obtain the perspective compression features, and then inputting the perspective compression features into the multivariate modular multiplication unit of three parallel tensors to calculate the perspective attention weights, and splicing the perspective supplementary features and the extracted features based on the attention weights to obtain multi-perspective features; Step S42: down-sample the distance supplementary features of different perspectives, align them with the original feature map, that is, the corresponding extracted features, and then convolve the distance supplementary features and the extracted features to compress the channel dimension to obtain the distance compression features, and then input the distance compression features into three multivariate modular multiplication units of parallel tensors different from step S41 to calculate the distance attention weights, and splice the distance supplementary features and the extracted features based on the attention weights to obtain multiple distance features; Step S43: using the multi-view feature as a supplementary semantic to supplement the multi-view feature so as to obtain multi-view collaborative features of different scales; specifically: The multi-view features are processed by channel attention and spatial attention in turn to obtain supplementary features, and the supplementary features are concatenated with the multi-distance features in the channel dimension, and then further fused by convolution to obtain multi-view collaborative features of different scales; Step S44: performing channel fusion on multi-view collaborative features of different scales to obtain multi-view collaborative fusion features.

4. The traffic sign detection method based on multi-scale feature fusion according to claim 3 is characterized in that: The process of obtaining multi-view features in step S42 is expressed as: ； ； ； ； ； in, represents the perspective supplement feature, represents the linear activation function, Represents a splicing operation, represents the convolution operation, represents the extracted features, represents the perspective supplement feature under the first perspective, represents the perspective supplement feature under the second perspective, represents the visual compression feature, represents the Sigmoid activation function, , and represents the weight matrix of the first multivariate modular multiplication unit, represents the visual attention weight of the first multivariate modular multiplication unit, , and represents the weight matrix of the second multivariate modular multiplication unit, represents the perspective attention weight of the second multivariate modular multiplication unit, , and represents the weight matrix of the third multivariate modular multiplication unit, represents the perspective attention weight of the third multivariate modular multiplication unit, Represents multi-view features, Represents element-wise product.

5. The traffic sign detection method based on multi-scale feature fusion according to claim 3 is characterized in that: Step S44 is specifically as follows: The multi-view collaborative features of different scales are divided into n blocks along the channel dimension, and the number of channels in each block is the same; each block of multi-view collaborative features is further divided into two parts along the channel dimension, one part is used for channel-based interaction with the multi-view collaborative features of adjacent blocks, and the other part is used for output; the outputs of all blocks are re-joined to obtain multi-view collaborative fusion features.

6. The traffic sign detection method based on multi-scale feature fusion according to claim 1, characterized in that: The backbone feature extraction module in step S2 includes a first convolutional layer, a second convolutional layer, a first C2f module, a third convolutional layer, a second C2f module, a fourth convolutional layer, a third C2f module, a fifth convolutional layer, a fourth C2f module and an SPPF module connected in series in sequence; wherein the second C2f module, the third C2f module and the SPPF module respectively output extracted features of different scales.

7. The traffic sign detection method based on multi-scale feature fusion according to claim 6 is characterized in that: The second C2f module, the third C2f module, and the fourth C2f module use deformable convolution to dynamically sample the input.

Images