A Traffic Text Detection Method Based on Feature Shaping and Character-Gap Feature Representation

Abstract

This invention discloses a traffic text detection method based on feature shuffling and character-gap feature representation, aiming to solve the technical problems of limited performance in small text detection and easy text fragmentation in existing methods. This method uses the FSNet backbone network to extract multi-scale hierarchical features, and achieves information enhancement interaction between high- and low-resolution features through a cross-scale shuffling layer. It utilizes the instance collaborative learning submodule in the context-aware fusion module to construct multiple receptive field paths, uniformly refine and represent the character and gap features of the text, and then the weighted feature fusion submodule adaptively fuses the multi-scale features. After feature fusion, a boundary proposal module combines dilated convolution and prior information to generate candidate text boundaries. Finally, the boundary Transformer module iteratively optimizes the boundary control points to output accurate text boundaries. This invention effectively improves the detection accuracy and boundary integrity of multi-scale text in complex traffic scenarios.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, specifically relating to a traffic text detection method based on feature shuffling and character-gap feature representation. Background Technology

[0002] Currently, with the acceleration of urbanization and the continuous growth of car ownership, the complexity of transportation systems is constantly increasing. Intelligent Transportation Systems (ITS), as a key means to solve traffic congestion and improve traffic safety and management efficiency, have received widespread attention and rapid development in recent years. Traffic text detection and recognition, as one of the core technologies of ITS, plays a crucial role in promoting the intelligentization of transportation. Among them, text-based traffic sign detection technology, due to its unique functional value, has become an important research direction in the fields of autonomous driving and intelligent transportation.

[0003] Traffic sign text contains rich semantic information, including key data such as location coordinates, distance parameters, directional indications, and real-time traffic conditions. This information provides crucial decision-making support for drivers and autonomous driving systems. Especially in autonomous driving applications, accurate traffic text detection and recognition not only assist vehicles in precise positioning and route planning but also provide reliable data support for intelligent traffic monitoring and traffic management decisions. This has significant theoretical and practical implications for improving road traffic safety and traffic efficiency.

[0004] Despite the existence of many effective methods, there is still room for in-depth research in the field of traffic text detection, and the rapidly developing autonomous driving industry has placed higher demands on the timeliness of traffic text detection technology.

[0005] First, traffic text exhibits large-scale variations, and due to factors such as shooting angle, it occupies a small portion of the image and contains a large amount of small textual information. Feature maps generated by commonly used convolutional neural networks (such as VGG and ResNet) are relatively coarse and struggle to capture the detailed information of small text. These features are highly effective for detecting large text but perform poorly in small text detection tasks, thus limiting the performance of small text detection.

[0006] Secondly, different traffic signs have different information, and the text is sparse and messy, which easily leads to text detection breakage problems, causing text information to be fragmented, which greatly limits the performance and applicability of the detection technology.

[0007] Furthermore, existing feature fusion techniques have not fully utilized inter-layer correlations. Feature pyramid networks fuse multi-scale features through a top-down path; however, the unidirectional information flow of this top-down path limits its performance. The Feature Fusion Module (FFM) in PAN improves text segmentation performance by constructing multi-scale features and forming output feature maps, but directly stacking multi-scale features introduces redundancy, affecting the feature representation effect.

[0008] Therefore, existing technologies still need further improvement to address the inherent challenges of traffic text detection, such as large scale variations, diverse text arrangement, and rich details, as well as the shortcomings of existing methods in effectively fusing features and accurately representing text gap features. Summary of the Invention

[0009] To overcome the shortcomings of existing technologies, this invention provides a traffic text detection method based on feature shuffling and character-gap feature representation. This method utilizes the FSNet backbone network to extract traffic text image features, performs cross-scale shuffling through a shuffling layer to enhance noise resistance and detail preservation; expands the receptive field and refines character and gap features through the instance collaborative learning submodule of the context-aware fusion module, and then adaptively adjusts the weights to fuse multi-scale features through the weighted feature fusion submodule; after obtaining the feature map, a boundary proposal module combines prior text information and multi-diffraction convolution to generate a coarse boundary; finally, the boundary is iteratively optimized through the encoder-decoder structure of the boundary Transformer module to achieve accurate text boundary detection.

[0010] The technical solution of this invention includes a feature shuffling module, a context-aware fusion module, a boundary proposal module, and a boundary Transformer module. First, features are extracted from traffic text images using the FSNet backbone network, completing initial convolutions, hierarchical downsampling, and other operations to output features at different scales. A shuffling layer is used to achieve cross-scale shuffling of low-resolution and high-resolution features, enhancing the diversity of feature representation. Second, the extracted features are processed by the context-aware fusion module (CAFM). The instance collaborative learning submodule (ICL) expands the receptive field through multi-path construction, refining the character and gap features of the text region to generate a unified feature representation. The weighted feature fusion submodule (WFFM) dynamically adjusts the weights to complete the fusion of multi-scale features. Finally, the boundary proposal module, based on the fused features, combines multi-diffraction dilated convolutions to generate classification maps, distance fields, and other key information to determine the initial coarse boundary. The boundary Transformer module then iteratively optimizes the control point offsets to correct the coarse boundary, ultimately achieving accurate text detection.

[0011] In the feature extraction stage, the method includes a backbone feature initialization module, a downsampling block, a convolution block, and a shuffling layer. The modules work together to complete the entire processing flow from the original image input to the multi-scale feature output.

[0012] The original image is first input into the backbone feature initialization module to complete the initial feature mapping; this module processes the image sequentially through a series of convolutional layers, batch normalization layers and ReLU activation functions, and outputs an initial feature map with a preset number of channels, laying the foundation for subsequent multi-scale feature extraction.

[0013] The initial feature map enters a hierarchical scaling and depth extraction stage: the feature map is downsampled sequentially through four downsampling blocks (the downsampling blocks are composed of...). (Constructed by concatenating convolutional layers, batch normalization layers, and ReLU activation functions) to generate Feature maps at four different scale levels.

[0014] In the multi-scale feature extraction process, efficient interaction of features at different scales is achieved through a shuffling layer (corresponding to the "shuffling operation" in the diagram). The specific process is as follows: First, the features participating in the fusion are... Each scale feature map is divided into channels. The feature maps are then divided into two parts; subsequently, based on the hierarchical difference between the source and target scales, upsampling or downsampling operations are used to adapt the size of each split feature map; finally, the adapted feature maps are concatenated along the channel dimension. After generation, two-scale and three-scale feature fusion is completed.

[0015] After collaborative processing by the above modules, the final output is... Four deep feature maps at different scales are used, and this set of feature maps covers semantic and detailed information at different levels.

[0016] The specific workflow of the feature fusion stage is as follows: First, the ICL module enhances the feature maps output by the FSNet module. The Instance Collaborative Learning (ICL) module performs this enhancement through sequential stacking. , , At each convolutional scale, asymmetric horizontal convolutions, vertical convolutions, and standard convolutions are combined in parallel to expand the receptive field of text features. The module uses large convolutional kernels in shallow layers and small convolutional kernels in deeper layers, allowing the network to gradually focus on the central regions of text characters and gaps, learning complete text representations. A multi-receptive field path is constructed through a combination of serial and parallel structures, deploying ICL modules at each scale of features to fully utilize the long-range dependencies between characters and gaps, implicitly learning unified feature representations and alleviating the text detection fragmentation problem.

[0017] The feature enhancement formula for the ICL module is:

[0018]

[0019] in, The feature map obtained by the FSNet module. For the enhanced feature map, For instance collaborative learning blocks, feature transformation and enhancement operations are performed; the specific process will be given later.

[0020] Subsequently, the WFFM module performs cross-scale weighted fusion on the multi-scale features enhanced by ICL.

[0021] First, the feature maps at each scale are upsampled and aligned:

[0022] With the highest resolution features Space dimensions Based on this, bilinear interpolation upsampling is performed on the remaining scale features:

[0023] ,

[0024] in, , , This is a scale-aligned feature map with spatial dimensions and Maintain consistency This is a bilinear interpolation upsampling function.

[0025] Next, the learnable weight parameters are normalized:

[0026] in, Indicates the first Learnable weights of layer features, This represents the weights after Softmax normalization. , corresponding to 4 scale features;

[0027] Then, multi-scale feature weighted fusion is performed to obtain the initial fused features:

[0028]

[0029] Channel compression is performed on the initial fusion features to unify the number of channels to the target dimension. Finally, deconvolution upsampling is used to improve the feature resolution, resulting in the final output features of the WFFM module.

[0030]

[0031] in, This is a channel compression module used to reduce the dimension of feature channels and reduce computational load. This is a transposed convolutional layer with a kernel size of 4, a stride of 2, and padding of 1, outputting a high-resolution single-level feature map. .

[0032] The boundary proposal module of this invention consists of multiple layers of dilated convolutions, specifically including two convolutions with different dilation rates. Convolutional layer and one Convolutional layers, which use single-level feature maps Using this as input, the module sequentially completes the entire process of feature map generation, candidate boundary selection, and confidence optimization: First, through feature transformation using multiple dilated convolutions, the module simultaneously generates a classification map, orientation field map, and distance field. Figure 3 Class feature maps—where the classification map represents the text / non-text classification confidence for each pixel; the orientation field map consists of two-dimensional unit vectors, for text instances... pixels within The vector is Pointing to its nearest text boundary pixel The unit vector (the pixel vector of the non-text region is) The distance field map is a normalized distance field, and the text pixels are... To the nearest boundary pixel The distance across the scale of its text instance After normalization, the pixel distance in non-text areas is 0.

[0033] Subsequently, based on the generated range field map, a threshold is set. The distance field is binarized to filter out pixel regions that meet the boundary features, generating initial candidate boundary proposals. To address false positives in the candidate proposals, a location confidence score is calculated for each proposal based on the classification map, and a confidence threshold is set. Candidate proposals with lower scores are removed, resulting in accurate coarse boundary proposals.

[0034] The boundary Transformer module is primarily used to convert coarse boundary proposals into precise text boundaries, enabling arbitrary-shaped text detection. It iteratively learns and predicts vertex offsets based on the coarse boundary proposals. The specific process is as follows: First, for the coarse boundary proposals represented as closed polygons, uniform sampling is performed. Control points (denoted as) ,gather These control points contain both sequence context and topological context (such as shape and spatial distribution); subsequently, a feature vector is constructed for each control point. Input features 32-dimensional shared features With 4-dimensional prior features The stitching of information (including classification diagrams, distance fields, direction fields, etc.), i.e. ,in, and The feature matrix of the boundary proposal is finally obtained by extracting it from the corresponding feature map through bilinear interpolation.

[0035] After obtaining the feature matrix, the boundary Transformer module performs feature learning and boundary refinement through an encoder-decoder structure: the encoder contains three Transformer blocks with residual connections, which transform the feature matrix into a boundary matrix of size 1.5. Boundary proposal feature map Encoded as Embedded feature map Its operational logic is as follows (⊕ represents addition); Each Transformer block adopts a standard architecture with multi-head self-attention and multilayer perceptron network (MLP), and this module removes the position encoding in the traditional Transformer.

[0036] After the encoder outputs the feature map, the decoder learns and predicts the offset between the control point and the target point based on the encoded features: first through a 128-dimensional single layer Convolutional processing encodes feature maps Then the processing result is compared with the feature map after the three-layer decoder. The data is concatenated along the feature dimension, and the final output by the decoder is a sequence of vertex offsets (expressed as follows). The decoder consists of a three-layer perceptron network and ReLU activated. Convolutional components.

[0037] Finally, the boundary transformation is performed iteratively using the boundary Transformer module. The maximum vertex prediction offset is limited to 16 pixels in each iteration to ensure the convergence of model training. Ultimately, the coarse boundary proposal is refined into a precise text boundary. Attached Figure Description

[0038] Figure 1 This is a flowchart of a traffic text detection method based on feature shuffling and character-gap feature representation.

[0039] Figure 2 This is a diagram illustrating the overall framework of a traffic text detection method based on feature shuffling and character-gap feature representation.

[0040] Figure 3 This is a structural diagram of the Context-Aware Fusion Module (CAFM).

[0041] Figure 4 This is a structural diagram of the boundary Transformer module. Detailed Implementation

[0042] To make the technical solution, objectives, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0043] like Figure 2 As shown, the present invention provides a traffic text detection method based on feature shuffling and character-gap feature representation, which includes a feature shuffling module, a context-aware fusion module, a boundary proposal module, and a boundary Transformer module.

[0044] In the feature shuffling module, the input image ( Feature extraction and shuffling operations are performed to generate feature maps at four scales. ,satisfy This process can be expressed by the formula:

[0045] The Stem layer, which extracts the backbone features, performs initial convolution and activation on the input image, compresses its size, and extracts basic features. The calculation formula for the steps module is as follows:

[0046]

[0047]

[0048] in, Indicates the kernel size as Fill as Convolution operation, Presentation layer normalization operation, This represents the activation function. Input image. Output feature size. .

[0049] FSNet implements downsampling through the steps module and feature depth extraction through the block module. Let... This represents the number of stacked convolutional blocks at each level. The block operation formula is as follows:

[0050] Residual branch (channel matching): ,in, This represents the number of input channels in the input feature map. This indicates the number of output channels in the output feature map.

[0051] Main branch (convolution + activation): ,in, Indicates the kernel size as Fill as Convolution operation, Presentation layer normalization operation, This represents the activation function.

[0052] Residual fusion output: , This represents the activation function.

[0053] Hierarchical feature generation:

[0054]

[0055]

[0056]

[0057]

[0058]

[0059]

[0060]

[0061]

[0062] in, For operations in the steps module, For operations of the block module, This is a feature-based shuffling module.

[0063] Output four scale feature maps:

[0064] like Figure 3 As shown, the CAFM module includes the ICL submodule and the WFFM submodule, and the specific process is as follows:

[0065] ICL submodule (Instance Collaborative Learning Module): Input is a single-scale feature map from FSNet. The output is an enhanced feature map with a unified character-gap representation. .

[0066] The calculation formula is:

[0067] in, For the first Input feature maps at various scales, This is the feature map enhanced by the ICL module. For instance collaborative learning blocks, perform feature transformation and enhancement operations.

[0068] The The specific calculation process is as follows:

[0069] Channel compression: , Indicates the kernel size as The convolution operation is used to compress 256-dimensional channels into 64-dimensional channels;

[0070] Enhanced Level 3 Multi-Receptive Field Feature:

[0071]

[0072] in, Indicates the kernel size as Convolution operation, Represents standard convolution. This represents asymmetric convolution in the vertical direction. This represents asymmetric convolution in the horizontal direction. They represent , , Enhancement features at three scales;

[0073] Channel restoration and normalized activation: , Indicates the kernel size as Convolution operation, Indicates the batch normalization layer. This represents the activation function. This represents the learned feature relation mapping;

[0074] Residual connection: .

[0075] Subsequently, the WFFM module performs cross-scale weighted fusion on the multi-scale features enhanced by ICL.

[0076] First, the feature maps at each scale are upsampled and aligned:

[0077] With the highest resolution features Space dimensions Based on this, bilinear interpolation upsampling is performed on the remaining scale features:

[0078] ,

[0079] in, , , This is a scale-aligned feature map with spatial dimensions equal to... Maintain consistency This is a bilinear interpolation upsampling function.

[0080] Next, the learnable weight parameters are normalized:

[0081] in, Indicates the first Learnable weights of layer features, This represents the weights after Softmax normalization. , corresponding to 4 scale features;

[0082] Then, multi-scale feature weighted fusion is performed to obtain the initial fused features:

[0083]

[0084] Channel compression is performed on the initial fusion features to unify the number of channels to the target dimension. Finally, deconvolution upsampling is used to improve the feature resolution, resulting in the final output features of the WFFM module.

[0085]

[0086] in, This is a channel compression module used to reduce the dimension of feature channels and reduce computational load. This module employs a method... The structure consisting of convolutional layers, batch normalization layers, and ReLU activation functions is used to uniformly compress the number of channels of the primary fusion features to the target dimension. This is a transposed convolutional layer with a kernel size of 4, a stride of 2, and padding of 1, outputting a high-resolution single-level feature map. .

[0087] The boundary proposal module employs multi-diffusion convolution to output a single-level feature map from the CAFM module. As input, a classification map, orientation field map, and distance field map are generated, and a rough boundary proposal is obtained by filtering.

[0088] First, by using dilated convolutions with different dilation rates, a classification map, orientation field map, and range field map are generated:

[0089]

[0090]

[0091] ,in, This represents the convolution operation. Presentation layer normalization operation, This represents the activation function.

[0092] from The split yields three types of feature maps:

[0093] Classification diagram:

[0094] Directional field:

[0095] Distance field:

[0096] Set distance threshold for distance field map Filter boundary pixels:

[0097]

[0098] Depend on Initial candidate boundary proposals are generated for regions with a median value of 1. .

[0099] Calculate the confidence score of candidate proposals using the classification graph, and filter out low-scoring proposals: The score is retained to satisfy The proposal yields a rough boundary proposal: if ,in This is the confidence threshold.

[0100] like Figure 4 As shown, the Boundary Transformer module takes a coarse boundary proposal as input, iteratively learns the control point offsets, and refines them into precise text boundaries.

[0101] For each coarse boundary proposal ( ), extract the features of each control point:

[0102]

[0103]

[0104] ,in, Indicates bilinear interpolation. This indicates a channel splicing operation.

[0105] The boundary proposal feature matrix is ​​obtained as follows: .

[0106] Transformer encoder feature encoding:

[0107] The encoder contains 3 layers of Transformer blocks and performs residual feature encoding:

[0108]

[0109] The computational logic for a single Transformer block is as follows:

[0110]

[0111]

[0112] Final encoder output: ,in, This indicates the bulls' self-attention. Representation layer normalization, This represents a multilayer perceptron.

[0113] The decoder predicts control point offsets based on encoded features:

[0114] ,in, Indicates the kernel size as Fill as The convolution operation.

[0115] This means that the two feature maps are spliced ​​along the channel dimension.

[0116] , This indicates a decoding operation. This is a 3-layer perceptron network that outputs the pixel offset of the control points; limitations. .

[0117] Boundary refinement and iterative update: Control points are updated using offsets to obtain precise boundaries. The process of "feature construction → encoding → decoding → offset update" is repeated to complete the boundary iterative refinement and finally output the accurate text boundary. .

Claims

1. A traffic text detection method based on feature shuffling and character-gap feature representation, characterized in that, Includes the following steps: Feature extraction steps: Input the traffic text image into the FSNet backbone network to perform hierarchical feature extraction and output multiple hierarchical feature maps of different scales; through the cross-scale shuffling layer, perform cross-scale feature shuffling operation on the hierarchical feature maps of different scales to realize information interaction between low-resolution and high-resolution features and obtain shuffled feature maps of different scales. Feature fusion step: The shuffled feature map is processed by a context-aware fusion module, which includes an instance collaborative learning submodule and a weighted feature fusion submodule. The instance cooperative learning submodule is configured to construct a three-level multi-receptive field convolution structure on each scale of feature map, the convolution kernel size of each level of convolution structure is set in the order from large to small, and the asymmetric horizontal convolution, vertical convolution and standard convolution are combined in parallel inside each level of convolution structure to construct a multi-receptive field feature extraction path, the features output by the three-level convolution structure are fused, and the fused features are up-sampled to the original size through deconvolution to obtain a single-level feature map. After channel restoration through convolution, the enhanced feature map is obtained by residual connection addition with the input feature map; and the weighted feature fusion submodule is configured to up-sample the enhanced feature map of each scale of the instance cooperative learning submodule for scale alignment, assign a learnable weight parameter to each scale of feature, obtain a fusion weight after Softmax normalization, and perform weighted fusion on the features of each scale after alignment based on the fusion weight, then perform channel compression and up-sample through deconvolution to improve the resolution, and generate a single-level feature map. Coarse boundary generation steps: Based on the single-level feature map, multi-diffraction dilated convolution is used in conjunction with text prior information to predict and generate classification map, orientation field and distance field; The distance field is thresholded to generate candidate text boundaries, and the confidence of each candidate text boundary is calculated using the classification map. Candidate boundaries with confidence scores below the threshold are filtered out to obtain coarse text boundaries. Precise boundary generation steps: Using the boundary Transformer module, through an encoder-decoder structure, iterative offset prediction and position correction are performed on the control point sequence on the coarse text boundary, gradually refining the coarse text boundary, and finally outputting accurate text boundary detection results.

2. The traffic text detection method of claim 1, wherein, The cross-scale shuffling operation in the feature extraction step specifically includes: The feature maps of each scale to be shuffled are divided into several groups along the channel dimension. Based on the difference between the source scale and the target scale, upsampling or downsampling operations are performed on each group of feature maps to achieve scale uniformity. The feature maps of each group after scale uniformity are spliced ​​along the channel dimension to obtain the shuffled feature map.

3. The traffic text detection method of claim 1, wherein, The specific processing of the instance collaborative learning submodule in the feature fusion step includes: The three-level multi-receptive field convolution structure with the convolution kernel size arranged from large to small is adopted on each scale feature map, multi-receptive field paths are constructed through asymmetric horizontal convolution, vertical convolution and standard convolution in parallel in each level, and character features and gap features of the text region are aggregated and refined gradually; after the enhanced features output by the three-level convolution structure are fused, the enhanced features are added to the input feature map through residual connection to obtain an enhanced feature map. The convolution is completed to restore the channel, and then the residual connection is added to the input feature map to obtain an enhanced feature map.

4. The traffic text detection method of claim 1, wherein, The weighted feature fusion submodule in the feature fusion step specifically includes the following processing: For each scale feature map of the enhanced instance cooperative learning sub-module, first, the spatial size of the highest resolution feature map is taken as a benchmark, and the remaining scale feature maps are adjusted to the same spatial resolution through upsampling alignment operation, so as to realize the scale unification of all features to be fused; then a learnable initial weight is given to each scale feature map, and the weight is normalized through a Softmax function to obtain the fusion weight corresponding to each scale feature; then the weighted sum is used to fuse the scale-aligned feature maps based on the normalized fusion weight to obtain a primary fusion feature; finally, a channel compression operation composed of convolution is performed on the primary fusion feature to unify the channel number to a target dimension, and then the resolution is improved through deconvolution upsampling to generate a single-level feature map. A channel compression operation composed of convolution is performed on the primary fusion feature to unify the channel number to a target dimension, and then the resolution is improved through deconvolution upsampling to generate a single-level feature map.

5. The traffic text detection method of claim 1, wherein, In the coarse boundary generation step, the direction field is a two-dimensional unit vector field used to indicate the direction of each pixel in the text region pointing to its nearest text boundary pixel; the distance field is a normalized distance field, representing the distance of each pixel in the text region to its nearest text boundary pixel, and is obtained by scale normalization processing of the text instance where the pixel is located.

6. The traffic text detection method of claim 1, wherein, The precise boundary generation step specifically includes: Uniform sampling is performed on the rough text boundaries to obtain There are several control points; for each control point, the feature vector at the corresponding position is extracted from the shared feature map and the prior feature map using bilinear interpolation and then concatenated to form a control point feature matrix; the encoder of the boundary Transformer module contains multiple stacked Transformer blocks, and the control point feature matrix is ​​context-encoded using a self-attention mechanism; the decoder of the boundary Transformer module predicts the coordinate offset of each control point based on the encoder output, and iteratively corrects the control point position. In each iteration, a maximum limit value is set for the coordinate offset, ultimately refining the coarse text boundary.

Images