Traffic sign image segmentation method based on improved space-time graph convolution

Abstract

The application discloses a traffic sign image segmentation method based on improved space-time graph convolution, relates to the technical field of traffic sign image segmentation and comprises the following steps: pre-processing traffic sign image data sets to realize image enhancement; performing eight-degree convolution-based RGB saliency detection network on the enhanced image to obtain a preliminary feature map; feeding the preliminary feature map into a space-time graph convolution module to obtain output features; splicing the output features with initial features and realizing feature matching by using a SETR algorithm; inputting the matched feature map into an eight-degree convolution residual network to perform detailed processing and edge optimization, and outputting a segmented image; and the application replaces ordinary convolution in the RGB image saliency detection module and the eight-degree convolution residual network with eight-degree convolution, greatly reduces the operation burden brought by a complex environment, realizes lightweight network architecture design, improves the prediction speed, greatly improves the operation efficiency, and finally improves the edge optimization effect of the traffic sign image.

Description

Technical Field

[0001] This invention relates to the field of traffic sign image segmentation technology, and in particular to a traffic sign image segmentation method based on improved spatiotemporal graph convolution. Background Technology

[0002] Traffic sign recognition is a crucial component of real-time road navigation in autonomous driving systems, and its accuracy and speed directly impact the safety of these systems. However, there is still room for improvement in traffic sign recognition under complex environments, such as accuracy in extreme weather and speed of recognition while vehicles are in motion. These issues urgently need to be addressed.

[0003] Deng Xiangyu et al. proposed a shape recognition algorithm for traffic sign classification combining a BP neural network. The algorithm utilizes color information for traffic sign region segmentation, but it suffers from a certain false recognition rate for circles and regular octagons, and is susceptible to adverse weather conditions and traffic congestion, leading to low recognition rates. Xu Jingcheng et al. proposed an improved traffic sign recognition method based on the Alex model, introducing batch normalization and adding a global average pooling layer to reduce network depth. However, the algorithm still needs improvement in handling various complex traffic signs in real-world traffic environments. For complex environments, He Ruibo et al. proposed an improved deep learning-based road traffic sign recognition algorithm, combining the SENet and ResNet models, extracting their respective advantages, and using a smaller number of network layers to achieve a higher recognition level. However, the algorithm requires significant manual intervention, involving numerous neural network parameter adjustments and inputs. Dewi et al. used the SPP concept to improve the backbone networks of Yolo V3, ResNet 50, DenseNet, and Tiny Yolo V3 for traffic sign feature extraction, but the algorithm has high computational complexity and a cumbersome implementation process. Cao et al. proposed an improved LeNet-5 convolutional neural network model for intelligent vehicle traffic sign detection and recognition to address the problem of traditional traffic sign recognition being susceptible to environmental factors. However, deep learning-based traffic sign recognition methods suffer from high computational cost and poor real-time performance. Yazdan et al. proposed a shape classification algorithm based on an SVM classifier to improve segmentation accuracy, filtering out erroneous pixels in the classification results through symbolic geometry. However, this algorithm is time-consuming and cannot meet the real-time requirements of vehicles in motion. Di Lan et al. proposed a road traffic sign recognition algorithm based on probabilistic clustering and convolutional neural networks, primarily addressing the high time consumption caused by noise and complex backgrounds in images. However, this algorithm has certain errors in traffic sign recognition. Jie Wei et al. proposed a real-time traffic sign classification and recognition method based on multi-feature fusion, mainly targeting the poor real-time performance caused by differences in sample categories during the recognition process. Mannan et al. proposed a fully data-driven segmentation technique to solve the problem of complete separation between traffic sign pixels and background objects, but this method comes at the cost of increased computational cost. Handoko, while reducing the algorithm's operating cost by implementing color and shape segmentation of traffic signs, still exhibits certain errors under extreme weather conditions. James et al. proposed using capsule neural networks to replace commonly used CNNs and RNNs, but the accuracy was only improved to a certain extent on the Indian traffic dataset, which is highly regional and lacks universality. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes a traffic sign image segmentation method based on improved spatiotemporal graph convolution.

[0005] A traffic sign image segmentation method based on improved spatiotemporal graph convolution includes the following steps: Preprocessing of traffic sign image datasets to achieve image enhancement; The enhanced image is processed by an RGB saliency detection network based on octave convolution to obtain preliminary feature maps; The preliminary feature map is fed into a spatiotemporal component graph convolutional network. After constructing a spatiotemporal component graph from the preliminary feature map, feature one is obtained through the spatiotemporal graph convolutional module. After constructing a spatial component graph from the preliminary feature map, feature two is obtained through the spatial graph convolutional module. The SETR algorithm is used to achieve feature matching between feature one and feature two. The matched feature map is then input into an octave convolutional residual network for detail processing and edge optimization, and the segmented image is output.

[0006] Specifically, the RGB saliency detection network based on octave convolution includes an octave convolution module, a feature enhancement module, a pyramid pooling module, and a channel attention module. The feature map extracted by the octave convolution is fed into the pyramid pooling module. In the pyramid pooling module, the feature map is subjected to average pooling operations at different scales to obtain a multi-semantic module. Finally, it is processed... The convolutional layer further extracts features, and the output features are denoted as... ,Will The output after passing through the channel attention module is denoted as .

[0007] Specifically, the process of constructing the spatiotemporal component graph includes: The model constructs a spatiotemporal component map based on the initial feature map input. Secondly, based on graphs The relationship determines the weights of the proximity matrix A.

[0008]

[0009] in, It is the identity matrix. It is the sum of the weights of the neighboring matrices, which combines the neighboring matrices and the characteristic matrix. If the input is given to a graph convolutional network, then the output of the graph convolutional network is updated to... ,

[0010] in, ,in This represents the total number of layers in the graph convolutional network. For layer index; It requires training the weight matrix for a specific layer. ReLU is a non-linear activation function. Then, a two-layer graph convolutional network is used to output the matrix. ,in ), utilizing max pooling to aggregate spatiotemporal component features,

[0011] The construction of the spatial component diagram includes: Reconstruct an undirected spatial component graph , With Figure The similarities lie in the number of images. The number of images is 1. The number of images is K; a two-layer graph convolutional network is used to obtain spatial component features. .

[0012] Specifically, the SETR algorithm in the SETR module includes: Two-dimensional feature image Transforming into a one-dimensional sequence, the Transformer accepts embeddings. The one-dimensional sequence is taken as input, where L seq For sequence length, To hide the channel size; used to implement the input image Convert to Z for image serialization; Each serialized image patch is further mapped to a potential linear map f. 3D embedding space: p e Where p is an image patch, e is a sequence of image patches, and e has A sequence, namely a sequence of images embedded in one-dimensional blocks; The spatial information of the image patch is encoded, and a specific embedding p is learned for each position i and added to it. In this process, the final sequence input E is formed. + , + ,…, + , where L seq The sequence length is used; spatial information is preserved using this method, allowing for the acquisition of the input sequence. ,

[0013]

[0014]

[0015] in, For the learnable parameters of the three linear projection layers, d k for dimensionality; The Transformer encoder consists of multiple layers of self-attention mechanisms and multiple layers of perceptron blocks. The self-attention mechanism is described as follows:

[0016] Where, d k Typically set to / m, ; MSA is an extension with m independent SA operations, which are concatenated and projected onto the output.

[0017] in, The MSA output is transformed through a residual skip MLP block, and the output...

[0018] Finally, the SETR algorithm performs pixel-level image segmentation using a decoder.

[0019] Specifically, the three decoders are Naive Upsampling, Progressive Upsampling (PUP), and Feature Multi-Level Aggregation.

[0020] Specifically, the preprocessing of the traffic sign image dataset employs the IPT preprocessing module. The IPT preprocessing module is a pre-trained model for end-to-end image processing, consisting of multiple head-tail structures for different tasks and a single shared entity. The framework comprises a multi-head structure, an encoder, a decoder, and a multi-tail structure, and includes the following steps: The multi-head structure processes images into feature maps. f H =H i (x) f H ∈R C×H×W Among them, H i ( i = {1, ..., N} t}) represents the first i Task header, N t This indicates the number of tasks, i.e., the size of the input dataset; Multi-head structure generates feature maps with C channels and the same width and height. f H ∈R C×H×W The feature map is cut and stretched by performing segmentation and stretching operations. The feature map is divided into N blocks of size P×P, and each block is flattened into a dimension P.2 ×C vector, obtain the cut features.

[0021] in, These are the feature vectors that have been cut and flattened; The flattened feature vectors are fed into a Transformer for processing to obtain the final result.

[0022] in, These are the same-dimensional output features obtained after processing by the Transformer. Features The image is fed into a multi-tailed structure for dimensionality transformation and then decoded into an enhanced image.

[0023] Specifically, the octave convolution module includes the following steps: The feature maps are classified and then convolutional.

[0024]

[0025] in, Represents high-frequency components, Representing the low-frequency component, f( () indicates a convolution operation; the process of converting low-frequency components to high-frequency output involves first processing the low-frequency components... Perform convolution and upsampling operations to restore its resolution to match that of the high-frequency components.

[0026] The process of converting high-frequency components to low-frequency output is the process of... Perform downsampling and convolution.

[0027] Among them, pool( ) indicates a downsampling operation, upsample( ) indicates an upsampling operation; Convolutional layer The number of input channels equals the number of low-frequency component channels, and the number of output channels equals the number of high-frequency component channels. (Convolutional layer) The number of input / output channels and Conversely, to maintain a uniform number of channels and the superposition of characteristics between high and low frequencies,

[0028]

[0029] in, To output low-frequency characteristics, To output high-frequency characteristics, m represents , There are four types of interaction paths between high and low frequencies.

[0030] Specifically, the octave convolutional residual network outputs the features of the spatiotemporal component graph convolutional network, denoted as... Send in Convolutional layers then enter the main branch and the attention mechanism branch, respectively. Features entering the attention mechanism branch pass through the channel attention mechanism module, which constructs the importance of feature channels, enhances or suppresses the generation weights of different channels, identifies the interdependencies between channel features, and finds useful features.

[0031] Where σ represents the sigmoid activation function, and MLP represents a multilayer perceptron. Features representing the entry channel attention mechanism module; The feature F processed by the channel attention mechanism module is multiplied by the feature after octave convolution to obtain the output feature, denoted as F'; F' enters the spatial attention mechanism module, which will perform refined processing on the region of interest.

[0032] in, This represents the Sigmoid function. This indicates that the filter size is 7. Convolution operation of 7; Multiply by the corresponding element of F' to obtain the final output feature of the CBAM branch. ; Main branches retain original features The output obtained from the CBAM branch The summation yields the final output of the residual attention module. ,

[0033] Where i represents the spatial location and c represents the index of the feature channel.

[0034] This invention proposes a traffic sign image segmentation method based on an improved spatiotemporal graph convolution module, aiming to enhance the segmentation speed and accuracy of traffic sign images under various extreme weather conditions. Experiments on three publicly available traffic datasets in a specific implementation demonstrate the beneficial effects of this invention as follows: 1) In terms of improving precision and computational efficiency, the RGB image saliency detection module that replaces ordinary convolution with octave convolution and the octave convolution residual network greatly reduce the computational burden caused by complex environments, realize a lightweight network architecture design, and improve prediction speed.

[0035] 2) In terms of feature extraction of spatiotemporal component graph convolutional networks, the application of spatiotemporal component graph convolution in traffic was initially proposed for traffic flow prediction. The research cross-domain application is applied to traffic sign recognition, and the ordinary fully convolutional module in the original algorithm is replaced with the SETR algorithm, which greatly improves the computational efficiency.

[0036] 3) In the improved residual module, the traditional ordinary convolution is replaced with octave convolution to improve the edge optimization effect of traffic sign images. Attached Figure Description

[0037] Figure 1 This is a diagram of the algorithm framework of the present invention; Figure 2 This is a diagram of the octave convolution module of the present invention; Figure 3 This is a flowchart illustrating the SETR algorithm steps of the present invention. Figure 4 This is a comparison chart of the image processing effects of related algorithms in specific embodiments of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0039] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0040] The various steps of the present invention will now be described in detail based on what is known to those skilled in the art.

[0041] Traffic sign image segmentation is often difficult due to factors such as inclement weather and lighting conditions. Environmental factors have a significant negative impact on traffic sign acquisition, but traffic signs are images with significant structural and color features, so RGB images can be directly used as the research object. In extreme cases, accurately detecting salient targets increases the computational burden and reduces computational efficiency. This invention uses octave convolution to replace the ordinary convolution module in the original algorithm, greatly reducing the computational burden caused by complex environments and achieving a lightweight network architecture design.

[0042] Specifically, this invention provides a traffic sign image segmentation method based on improved spatiotemporal graph convolution, such as... Figure 1 As shown, it includes the following steps: S1. Preprocess the traffic sign image dataset to achieve image enhancement. The preprocessing method uses the IPT preprocessing module. The IPT preprocessing module is a pre-trained model for end-to-end image processing, consisting of multiple head-tail structures for different tasks and a single shared entity. The framework is a multi-head structure, encoder, decoder, and multi-tail structure, including the following steps: S101. The multi-head structure processes images into feature maps. f H =H i (x) f H ∈R C×H×W Among them, H i ( i = {1, ..., N} t}) represents the first i Task header, N t This indicates the number of tasks, i.e., the size of the input dataset; S102. Multi-head structure generation of feature maps with C channels and the same width and height. f H ∈R C×H×W The feature map is cut and stretched by performing segmentation and stretching operations. The feature map is divided into N blocks of size P×P, and each block is flattened into a dimension P. 2 The ×C vector yields the cut features.

[0043] in, These are the feature vectors that have been cut and flattened; S103. The flattened feature vector is fed into the Transformer for processing to obtain the desired result.

[0044] in, These are the same-dimensional output features obtained after processing by the Transformer. S104. Features The image is fed into a multi-tailed structure for dimensionality transformation and then decoded into an enhanced image.

[0045] S2. Apply an RGB saliency detection network based on octave convolution to the enhanced image to obtain preliminary feature maps; such as Figure 2 As shown, the RGB saliency detection network based on octave convolution includes an octave convolution module, a FEM (Flexible Image Processing), a pyramid pooling module, and a channel attention module. The feature map extracted by the octave convolution is fed into the pyramid pooling module, where the feature map is averaged at different scales (i.e., 1 / 2 scale). , , , Convolutional layer operations yield multiple semantic modules, which are then processed by... The convolutional layer further extracts features, and the output features are denoted as... ,Will The output after passing through the channel attention module is denoted as . It will be used as input to the CAM (channel attention module) module, pass through an adaptive layer, and consist of two... Convolutional operations are used to expand the receptive field of features. This is then combined with the decoder. The output features are concatenated to obtain preliminary fused features. A channel feature vector is generated using global average pooling. After processing by a fully connected layer, the interdependencies between channels are obtained. The sigmoid function is then used to weigh the importance W of each channel, aligning it with... Multiply to obtain the weighted features Finally, a w1×1 convolutional layer is used to... The number of feature channels is restored to the input feature size, which is then used as the output of the CAM module, denoted as... .

[0046] The octave convolution module includes the following steps: The feature maps are classified and then convolutional.

[0047]

[0048] in, Represents high-frequency components, Representing the low-frequency component, f( () indicates a convolution operation; the process of converting low-frequency components to high-frequency output involves first processing the low-frequency components... Perform convolution and upsampling operations to restore its resolution to match that of the high-frequency components.

[0049] The process of converting high-frequency components to low-frequency output is the process of... Perform downsampling operation (to Convolution operations (or average pooling) are performed for the convolution stride, and convolution is performed.

[0050] Among them, pool( ) indicates a downsampling operation, upsample( ) indicates an upsampling operation; Convolutional layer The number of input channels equals the number of low-frequency component channels, and the number of output channels equals the number of high-frequency component channels. (Convolutional layer) The number of input / output channels and Conversely, to maintain a uniform number of channels and the superposition of characteristics between high and low frequencies,

[0051]

[0052] in, To output low-frequency characteristics, To output high-frequency characteristics, m represents , There are four types of interaction paths between high and low frequencies.

[0053] S3. The preliminary feature map is fed into a spatiotemporal component graph convolutional network. After constructing a spatiotemporal component graph from the preliminary feature map, feature one is obtained through the spatiotemporal graph convolutional module. After constructing a spatial component graph from the preliminary feature map, feature two is obtained through the spatial graph convolutional module. The SETR algorithm is used to achieve feature matching between feature one and feature two. The application of spatiotemporal component graph convolution in traffic was initially proposed for traffic flow prediction. It is based on a structured graph representation model of components or nodes. This convolution will connect each component or node... Dense grids are considered as image features to simplify and improve computational efficiency.

[0054] Specifically, the process of constructing the spatiotemporal component graph includes: Feature map Each 1×1× Dense meshes are considered as image feature components, where t represents the index of the input image, K represents a positive number, and represents the spatiotemporal target model. An undirected spatiotemporal component graph is constructed based on the input feature 1, using the component nodes and K (i.e., tK, ..., t-1) as time series. , Where V and E are the set of nodes and the set of edges in an undirected graph, respectively, and the node set V = { |k = t -1,…,t – K ; n =1,…,N} contains all nodes in K, F( The feature vector is denoted as . The edge set E contains two types of edges: the first type is spatial edges. ={ |1 ≤i, j≤N, i≠j} represents the relationship between nodes within the features of each image. Since the features in the image change over time, the study uses a fully connected graph to describe the spatial relationships; the second type is temporal edges. This represents the relationship between nodes in similar feature images. Connecting components or nodes with the same position in similar image features can be regarded as a tracking trajectory of a specific component changing over time.

[0055] Based on graph The relationship determines the weights of the proximity matrix A.

[0056]

[0057] in, It is the identity matrix. It is the sum of the weights of the neighboring matrices, which combines the neighboring matrices and the characteristic matrix. If the input is given to a graph convolutional network, then the output of the graph convolutional network is updated to... ,

[0058] in, ; It requires training the weight matrix for a specific layer. ReLU is a non-linear activation function. Then, a two-layer graph convolutional network is used to output the matrix. ,in ), utilizing max pooling to aggregate spatiotemporal component features,

[0059] The construction of the spatial component diagram includes: Reconstruct an undirected spatial component graph , With Figure The similarities lie in the number of images. The number of images is 1. The number of images is K. A two-layer graph convolutional network is used to obtain spatial part features. .

[0060] Specifically, such as Figure 3 As shown, the SETR algorithm in the SETR module includes: Two-dimensional feature image Transforming into a one-dimensional sequence, the Transformer accepts embeddings. The one-dimensional sequence is taken as input, where L seq For sequence length, To hide the channel size; used to implement the input image Convert to Z for image serialization; Each serialized image patch is further mapped to a potential linear map f. 3D embedding space: p e Where p is an image patch, e is a sequence of image patches, and there are a total of A sequence of channels, i.e., a sequence of images with one-dimensional block embedding; The spatial information of the image patch is encoded, and a specific embedding p is learned for each position i and added to it. In this process, the final sequence input E is formed. + , + ,…, + , where L seq The sequence length is used; spatial information is preserved using this method, allowing for the acquisition of the input sequence. ,

[0061]

[0062]

[0063] Where l1 is the layer index of the encoder. For the learnable parameters of the three linear projection layers, d k for dimensionality; The Transformer encoder consists of multiple layers of self-attention mechanisms and multiple layers of perceptron blocks. The self-attention mechanism is described as follows:

[0064] Where, d k Typically set to / m, ; MSA is an extension with m independent SA operations, which are concatenated and projected onto the output.

[0065] in, The MSA output is transformed through a residual skip MLP block, and the output...

[0066] Finally, the SETR algorithm designs three decoders to perform pixel-level image segmentation: Naive Upsampling, Progressive Upsampling (PUP), and Feature Multi-Level Aggregation.

[0067] Traffic sign images in complex scenes can typically be decomposed into low-frequency and high-frequency signals. High-frequency signals represent rich details with dramatic changes in the image, where adjacent regions have significant differences in grayscale. Low-frequency signals represent gently changing edge structures, with slow changes in grayscale.

[0068] In traffic sign segmentation, to improve segmentation accuracy and reduce computational load, this study proposes octave convolution to replace the fully convolutional module and the ordinary convolutional module in the residual module of the spatiotemporal component graph convolution algorithm. Octave convolution can effectively improve image segmentation accuracy, solve the spatial redundancy problem in convolution calculation, and achieve a lightweight network architecture design.

[0069] S4. Input the matched feature maps into the octave convolutional residual network for detail processing and edge optimization, and output the segmented image. The octave convolutional residual network includes a channel attention mechanism module, a first octave convolution module, a spatial attention mechanism module, and a second octave convolution module. The residual structure adds the features input to the channel attention mechanism module with the features obtained from the second octave convolution module. Octave convolution replaces the full convolution in the original residual module for optimization, and the channel attention mechanism merges features to reduce computation and improve traffic sign recognition speed. Specifically, the steps are as follows: S401. The octave convolutional residual network outputs the features of the spatiotemporal component graph convolutional network, denoted as... Send in Convolutional layers then enter the main branch and the attention mechanism branch, respectively. S402. Features entering the attention mechanism branch pass through the channel attention mechanism module, which constructs the importance of feature channels, enhances or suppresses the generation weights of different channels, identifies the interdependencies between channel features, and finds useful features.

[0070] Where σ represents the sigmoid activation function, and MLP represents a multilayer perceptron. Features representing the entry channel attention mechanism module; S403. Multiply the feature F processed by the channel attention mechanism module with the feature after octave convolution to obtain the output feature, denoted as F'; F' enters the spatial attention mechanism module, which will perform refined processing on the region of interest.

[0071] in, This represents the Sigmoid function. This indicates that the filter size is 7. Convolution operation of 7; S404. Multiply with the corresponding element of F' to obtain the final output feature of the CBAM branch. ; S405. Main branches retain their original characteristics. The output obtained from the CBAM branch The summation yields the final output of the residual attention module. ,

[0072] Where i represents the spatial location and c represents the index of the feature channel.

[0073] The datasets used in this specific implementation are from the Chinese Traffic Sign Detection Dataset, the Chinese Traffic Sign Database, and the Traffic Sign Dataset, as shown in Table 1.

[0074] Table 1 Traffic sign dataset

[0075] The experiment used a server with a main frequency of 2.5GHz, an NVIDIA RTX 3090 12GB GPU, 32GB of memory, and PyCharm as the software development environment.

[0076] This implementation employed both quantitative and qualitative comparisons. For the quantitative experimental evaluation, the mean absolute error (MAE) was selected; a smaller MAE value indicates a better model.

[0077] The root mean square error (RMSE) represents the expected value of the squared error; the smaller the value, the smaller the error.

[0078] The smaller the mean absolute percentage error (MAPE), the better the accuracy of the prediction model.

[0079] Where m and n represent the length and width of the image, respectively. y represents the image segmentation algorithm, and y represents the image segmentation done manually.

[0080] The maximum F-measure is a comprehensive evaluation index; the larger the value, the more effective the experimental method.

[0081] Where y2 is defined as 0.3, and recall is the percentage of relevant images detected by the algorithm and the total number of relevant images; the higher the recall, the more relevant images are segmented.

[0082] Precision

[20] Precision is the percentage of the number of relevant images segmented out of the total number of images; the higher the precision, the more accurately relevant images are segmented.

[0083] Where sum(S) and sum(A) are the salient image and the manually segmented image, respectively, and sum(S,A) is the sum of the values ​​of the corresponding pixels in the two images.

[0084] The method described in this specific implementation is compared with other algorithms in terms of evaluation metrics.

[0085] Table 2 Comparison of relevant algorithms on evaluation metrics

[0086] Table 2 shows that the experimental results are based on 400 images from the CCTSDB dataset (400 images randomly selected from the "China Traffic Sign Detection Dataset"). The experimental results show that the MAE, RMSE, and MAPE values ​​of the algorithm in this specific implementation are lower than those of other algorithms; the F-measure values ​​are improved by 6.9%, 13.8%, 9.8%, and 7.7% respectively compared to other algorithms; and the Precision values ​​are improved by 16.5%, 10.1%, 6.1%, and 5.1% respectively compared to other algorithms. Since smaller MAE, RMSE, and MAPE values ​​indicate less error and higher accuracy, a larger F-measure value indicates a more effective experimental method and clearer results, and a larger Precision value indicates more accurate target recognition in sign images, the experimental results show that the algorithm in this paper is significantly better than other algorithms. A comparison of the image processing performance of the algorithm in this specific implementation with other algorithms is shown below. Figure 4 As shown.

[0087] As shown in Figure 4, in a qualitative comparison with other algorithms, the study selected typical cases under complex environments in the dataset for testing. Experimental results show that, when selecting traffic sign images under conditions of strong light, heavy fog, heavy rain, night, and snow, the algorithm of this invention significantly outperforms other algorithms in image processing, especially under low-contrast conditions such as heavy rain and strong light, where the traffic sign images processed by the algorithm of this specific implementation are clearer.

[0088] Table 3 Comparison of processing time for single images in datasets of different sizes and types

[0089] Table 3 shows that four different datasets—CCTSDB·400, CCTSDB·600, DFG·700, and CTSD·800—were selected to test the processing time of a single image for traffic sign recognition using different algorithms. Experimental results show that, compared to the detection and recognition algorithm based on the improved LeNet-5 model, the algorithm in this specific implementation reduces the single traffic sign image recognition time by 40.6%, 38.5%, 36.5%, and 33.2% on the four different datasets, respectively; compared to sign classification algorithms, it reduces the time by 23.1%, 18.8%, 24.5%, and 21.0%, respectively; compared to the AlexNet algorithm, it reduces the time by 40.3%, 39.2%, 36.6%, and 32.7%, respectively; and compared to the improved deep learning algorithm, it reduces the time by 43.9%, 39.9%, 35.4%, and 32.8%, respectively. Therefore, the algorithm in this specific implementation significantly outperforms other algorithms in terms of single traffic sign image recognition processing time on different datasets.

[0090] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0091] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A traffic sign image segmentation method based on improved spatiotemporal graph convolution, characterized in that, Includes the following steps: Preprocessing of traffic sign image datasets to achieve image enhancement; The enhanced image is processed by an RGB saliency detection network based on octave convolution to obtain preliminary feature maps; The preliminary feature map is fed into a spatiotemporal component graph convolutional network. After constructing a spatiotemporal component graph from the preliminary feature map, feature one is obtained through the spatiotemporal graph convolutional module. After constructing a spatial component graph from the preliminary feature map, feature two is obtained through the spatial graph convolutional module. The SETR algorithm is used to achieve feature matching between feature one and feature two. The matched feature maps are input into an octave convolutional residual network for detail processing and edge optimization, and the segmented image is output. The octave convolutional residual network outputs the features of the spatiotemporal component graph convolutional network, denoted as... Send in Convolutional layers then enter the main branch and the attention mechanism branch, respectively. Features entering the attention mechanism branch pass through the channel attention mechanism module, which constructs the importance of feature channels, enhances or suppresses the generation weights of different channels, identifies the interdependencies between channel features, and finds useful features. ； Where σ represents the sigmoid activation function, and MLP represents a multilayer perceptron. Features representing the entry channel attention mechanism module; The feature F processed by the channel attention mechanism module is multiplied by the octave convolution feature to obtain the output feature, denoted as F'; F' enters the spatial attention mechanism module, which will perform fine-grained processing on the region of interest. ； in, This represents the Sigmoid function. This indicates that the filter size is 7. Convolution operation of 7; Multiply by the corresponding element of F' to obtain the final output feature of the CBAM branch. ; Main branches retain original features The output obtained from the CBAM branch The summation yields the final output of the residual attention module. , ； Where i represents the spatial location and c represents the index of the feature channel.

2. The traffic sign image segmentation method based on improved spatiotemporal graph convolution according to claim 1, characterized in that, The RGB saliency detection network based on octave convolution includes an octave convolution module, a feature enhancement module, a pyramid pooling module, and a channel attention module. The feature maps extracted by the octave convolution are fed into the pyramid pooling module. In the pyramid pooling module, the feature maps undergo average pooling operations at different scales to obtain a multi-semantic module. Finally, the signal is processed... The convolutional layer further extracts features, and the output features are denoted as... ,Will The output after passing through the channel attention module is denoted as .

3. The traffic sign image segmentation method based on improved spatiotemporal graph convolution according to claim 1, characterized in that, The specific process of constructing the spatiotemporal component diagram includes: The model constructs a spatiotemporal component map based on the initial feature map input. Secondly, based on graphs The relationship determines the weights of the proximity matrix A. ； ； in, It is the identity matrix. It is the sum of the weights of the neighboring matrices, which combines the neighboring matrices and the characteristic matrix. If the input is a graph convolutional network input, then the graph convolutional network output is updated to... , ； in, ; It requires training the weight matrix for a specific layer. ReLU is a non-linear activation function. Then, a two-layer graph convolutional network is used to output the matrix. ,in ), utilizing max pooling to aggregate spatiotemporal component features, ； The construction of the spatial component diagram includes: Reconstruct an undirected spatial component graph , With Figure The similarities lie in the number of images. The number of images is 1. The number of images is K; a two-layer graph convolutional network is used to obtain spatial component features. .

4. The traffic sign image segmentation method based on improved spatiotemporal graph convolution according to claim 2, characterized in that, The SETR algorithm includes: Two-dimensional feature image Transforming into a one-dimensional sequence, the Transformer accepts embeddings. The one-dimensional sequence is taken as input, where L seq For sequence length, To hide the channel size; used to implement the input image Convert to Z for image serialization; Each serialized image patch is further mapped to a potential linear map f. 3D embedding space: p e Where p is an image patch, e is a sequence of image patches, and e has A sequence, namely a sequence of images embedded in one-dimensional blocks; The spatial information of the image patch is encoded, and a specific embedding p is learned for each position i and added to it. In this process, the final sequence input E is formed. + , + ,…, + , where L seq The sequence length is used; spatial information is preserved using this method, allowing for the acquisition of the input sequence. , ； ； ； in, For the learnable parameters of the three linear projection layers, d k for dimensionality; The Transformer encoder consists of multiple layers of self-attention mechanisms and multiple layers of perceptron blocks. The self-attention mechanism is described as follows: ； Where, d k Typically set to / m, ; MSA is an extension with m independent SA operations, which are concatenated and projected onto the output. ； in, The MSA output is transformed through a residual skip MLP block, and the output... ； Finally, the SETR algorithm performs pixel-level image segmentation using a decoder.

5. The traffic sign image segmentation method based on improved spatiotemporal graph convolution according to claim 4, characterized in that, The decoder employs one of three decoders: Naive Upsampling, Progressive Upsampling (PUP), or Feature Multi-Level Aggregation.

6. The traffic sign image segmentation method based on improved spatiotemporal graph convolution according to claim 1, characterized in that, The preprocessing of the traffic sign image dataset employs the IPT preprocessing module. The IPT preprocessing module is a pre-trained model for end-to-end image processing, composed of multiple head-tail structures for different processing tasks and a single shared entity. Its framework includes a multi-head structure, encoder, decoder, and multi-tail structure, and includes the following steps: The multi-head structure processes images into feature maps. f H =H i (x); f H ∈R C×H×W ； Among them, H i ( i = {1, ..., N} t }) represents the first i Task header, N t This indicates the number of tasks, i.e., the size of the input dataset; Multi-head structure generates feature maps with C channels and the same width and height. f H ∈R C×H×W The feature map is cut and stretched by performing segmentation and stretching operations. The feature map is divided into N blocks of size P×P, and each block is flattened into a dimension P. 2 The ×C vector yields the cut features. ； in, These are the feature vectors that have been cut and flattened; The flattened feature vectors are fed into a Transformer for processing to obtain the final result. ； in, These are the same-dimensional output features obtained after processing by the Transformer. Features The image is fed into a multi-tailed structure for dimensionality transformation and then decoded into an enhanced image.

7. The traffic sign image segmentation method based on improved spatiotemporal graph convolution according to claim 1, characterized in that, The steps for the octave convolution are as follows: The feature maps are classified and then convolutional. ； ； in, Represents high-frequency components. Representing the low-frequency component, f( () indicates a convolution operation; the process of converting low-frequency components to high-frequency output involves first processing the low-frequency components... Perform convolution and upsampling operations to restore its resolution to match that of the high-frequency components. ； The process of converting high-frequency components to low-frequency output is the process of... Perform downsampling and convolution. ； Among them, pool( ) indicates a downsampling operation, upsample( ) indicates an upsampling operation; Convolutional layer The number of input channels equals the number of low-frequency component channels, and the number of output channels equals the number of high-frequency component channels. (Convolutional layer) The number of input / output channels and Conversely, to maintain a uniform number of channels and the superposition of characteristics between high and low frequencies, ； ； Where m represents , There are four types of interaction paths between high and low frequencies.

Images