A natural scene text detection method based on semantic feature step-by-step reorganization

Abstract

The application relates to a natural scene text detection method based on semantic feature step-by-step reorganization. The specific operation of the method is as follows: a visual concept matching module constructs a visual concept cluster through a CLIP encoder and a clustering algorithm, and divides mixed concepts and pure concepts according to a text co-occurrence rate; a reorganization weight controller determines a reorganization weight in combination with concept purity differences and a text saliency estimation value; a semantic feature reorganization module reorganizes feature maps at two levels of local and global in combination with adaptive reorganization weights; and a double-network collaborative optimization module respectively predicts coordinate offsets and system errors by independently training a boundary point displacement prediction network and a system error prediction network, and iteratively optimizes text box coordinates. The application improves the generalization and robustness of the model through the semantic feature reorganization module, corrects the initial text detection box coordinates in combination with the double-network collaborative optimization module, and significantly improves the comprehensive performance of natural scene text detection.

Description

Technical Field

[0001] This invention relates to the field of natural scene text detection methods, and discloses a natural scene text detection method based on the hierarchical reorganization of semantic features. Background Technology

[0002] Natural scene text detection is a key area of ​​interdisciplinary research in artificial intelligence and computer vision. Its goal is to accurately identify text regions in images and obtain the coordinates of the corresponding text detection boxes. This technology plays a crucial role in image translation, traffic management, and autonomous driving.

[0003] Compared to document text images, text detection tasks in natural scene images face significant challenges, mainly including: the text shape has strong randomness, making it difficult for the model to generate detection boxes that accurately match the shape of the real text; background and text features are easily confused, increasing the risk of misidentification; and changes in lighting or object occlusion in complex scenes destroy the visual integrity of the text.

[0004] Traditional text detection methods for natural scenes mainly rely on image processing techniques and mathematical models, such as thresholding, connected component analysis, region growing, and region proposal. These methods may be effective when processing images with relatively simple structures, but they have significant limitations when dealing with complex and varied natural scene images.

[0005] With the advancement of computer vision technology, deep learning-based text detection methods for natural scenes have become an important research direction.

[0006] Current mainstream methods for improving text detection performance in natural scenes are highly dependent on the encoder structure.

[0007] Whether it's a method based on convolutional neural networks to capture local feature information of an image, or a method based on visual Transformer (ViT) to model global context information, an inherent limitation is often overlooked: the learned feature representation is essentially a passive fit of all statistical associations in the training data, which inevitably contains spurious associations between text and specific textures or overall styles that do not have a causal relationship.

[0008] In existing technologies, most deep learning-based natural scene text detection methods focus on improving the internal architecture of the encoder. However, their training process always relies on spurious relevance and is easily affected by background prior knowledge, leading to semantic confusion and limiting the model's generalization ability and robustness in complex natural scenes.

[0009] Some existing methods attempt to introduce additional loss functions or structures to improve the model's discriminative ability. However, such methods typically fail to fundamentally identify and remove visual confounding factors or actively reconstruct clean feature maps, leaving room for improvement in establishing true causal relationships.

[0010] It is worth noting that suppressing confounding factors at the semantic level, thereby reducing the undue influence of background information on text detection in terms of causality, helps to cut off false associations and build a pure discrimination environment. However, the application of semantic feature reconstruction in image processing, especially in fields with high accuracy requirements such as text detection in natural scenes, has not yet been fully explored.

[0011] Therefore, given the inherent challenges of text detection in natural scenes (such as false statistical associations and system noise interference), and the fact that the local and global information fused by existing deep learning methods is mixed with a large number of visual confounding factors introduced by data bias, there is an urgent need to develop an efficient, accurate, and effective method for text detection in natural scenes that can cut off false associations. Summary of the Invention

[0012] To overcome the shortcomings of existing technologies, this invention provides a natural scene text detection method based on hierarchical semantic feature recombination. The core of this method lies in: designing a semantic feature recombination module to clean and recombine feature maps at both the regional and global levels according to recombination weights at a high-level semantic level, resulting in multi-layered feature maps free from visual prior bias; and designing a dual-network collaborative optimization module to predict coordinate offsets through a boundary point displacement prediction network, and modeling and predicting system noise during the process for small text, low-resolution, and occluded images through a system error prediction network, generating high-precision text boxes through a few iterative steps.

[0013] The technical solution of this invention is achieved through the following steps: a natural scene text detection system based on semantic feature hierarchical reorganization (e.g., Figure 1 As shown, the module includes a visual concept matching module, a reorganization weight controller, a semantic feature reorganization module, and a dual-network collaborative optimization module. In the visual concept matching module, high-level semantic feature vectors are extracted using CLIP, visual concept clusters are generated through clustering, and mixed concepts are distinguished from pure concepts based on text co-occurrence rate. In the reorganization weight controller module, reorganization weights are adaptively set to enhance the purification of mixed interference factors and maximize the protection of text regions. In the semantic feature reorganization module, a dual reorganization mechanism of region and global is constructed to proactively cut off false statistical associations at the high-level semantic level. In the dual-network collaborative optimization module, coordinate offset and system noise are modeled separately to further improve the accuracy of text boxes and solve the problem of text box geometric positioning accuracy.

[0014] To construct hybrid-pure concept matching pairs, a visual concept matching module is proposed before the semantic feature reorganization module. First, local and global feature vectors are extracted from a large-scale image dataset using the CLIP encoder, and overall feature matrices are constructed accordingly. Then, the K-Means++ algorithm is used to cluster the feature vectors, resulting in 512 visual concept cluster centers. A lightweight text detection model is used for automated diagnosis of each cluster: by randomly sampling images within the cluster and detecting text regions, the co-occurrence rate between visual concepts and text is calculated. Concepts with a co-occurrence rate below 5% are classified as pure concepts, and those above 30% are classified as hybrid concepts. Finally, for each hybrid visual concept, it is matched with the visually similar, semantically relevant, and completely pure concept within the same cluster based on similarity. This process is then used to purify the hybrid visual concepts.

[0015] like Figure 2 In the framework diagram shown, the semantic feature reorganization module contains two serial units: the region reorganization unit actively reconstructs biased local background regions (such as brick wall texture, advertising signs, etc.) through pure local concepts (such as cement wall surface, marble texture, etc.) to prevent the model from misjudging due to prior information of local background; the global reorganization unit intervenes in the overall style and lighting conditions of the image through pure global concepts, forcing the model to stably recognize text under different shooting conditions and improving the generalization of the model. The semantic feature reorganization module is one of the key innovations of this invention.

[0016] Furthermore, to enhance the model's ability to purify visually cluttered concepts and mitigate their impact on the text, this invention designs a reorganized weight controller. First, ResNet is used as the backbone network, and FPN is employed to extract multi-scale features. The resulting feature maps are then divided into... The system identifies several regions and then adaptively determines the reorganization weights for each region.

[0017]

[0018]

[0019] in, This represents the Sigmoid activation function. , All represent trainable positive weight parameters. Indicates the first The difference in conceptual purity of each region's feature blocks Represents the concept purity function. Indicates the first The visual concept after purification of the feature blocks in each region Indicates the first The original visual concept of each region's feature blocks. Indicates the first Preliminary estimates of the text saliency of feature blocks in each region This represents the trainable bias parameters. If... A value close to 1 indicates that the region is a strong reorganization region. A value close to 0 indicates that the region is a weak reorganization region.

[0020] The region recombination unit uses a clean visual concept library as a benchmark and the region adaptive recombination weight generated by the recombination weight controller as the adjustment basis. Based on the difference between the matched local clean visual features and local mixed visual features at the high-level semantic level, the input feature map is compensated and optimized, and finally a multi-layer region recombination feature map is output.

[0021] The global reconstruction unit is used to clean and reconstruct the global semantics of the entire image. First, the multi-layer reconstructed feature map output by the region reconstruction unit is concatenated and fused with the high-level feature map as the input of the global reconstruction unit. Hybrid-clean concept matching pairs are constructed at the global image level and the global reconstruction weights are determined. Then, the same adjustment vector is superimposed at all positions of the feature map to obtain the multi-layer feature map after global reconstruction.

[0022] In the dual-network collaborative optimization module, coordinate offset and system error are predicted by independently training a boundary point displacement prediction network and a system error prediction network, respectively. The coordinate offset refers to the two-dimensional offset of the text box boundary points in the image coordinate system, while the system error refers to the inherent, consistent prediction error of the upstream model due to model bias, imbalanced training data, etc. For example, for small text or partially occluded text instances, the text detection boxes output by the model often shift to the left or right. This module combines the coordinate offset and removes the system error to obtain the final high-precision text box output.

[0023] This invention designs a dual-network collaborative optimization module, which aims to use a boundary point displacement prediction network and a system error prediction network to collaboratively optimize the initial text candidate boxes, so as to guide the system to more accurately locate the text region.

[0024] The workflow of the dual-network collaborative optimization module is as follows: Figure 3 As shown, the specific details are as follows: (1) At the feature map scale, the intermediate text box state samples between the initial text box state and the target text box state are obtained by point-by-point coordinate linear interpolation; (2) A boundary point displacement prediction network is constructed to predict the coordinate offset; (3) In order to improve the accuracy of text boxes for small text, occluded text, and low-resolution text, a system error prediction network is constructed to predict the system error, and the initial text candidate box is corrected by the predicted coordinate offset and the system error together.

[0025] In summary, this invention first constructs a pure visual concept library and matches pure factors to confounding factors through a visual concept matching module; secondly, it designs a recombination weight controller to enhance the purification of visual confounding factors and protect text regions; crucially, through a semantic feature recombination module, it actively reconstructs visual confounding regions and standardizes image style and illumination intensity to obtain multi-layer feature maps that remove background prior knowledge interference; finally, a dual-network collaborative optimization module predicts coordinate offsets and corrects system errors to obtain high-precision text boxes. This mechanism significantly improves the coordinate accuracy of text boxes in complex natural scenes by recombining feature maps to cut off false associations and using dual-network collaborative optimization to remove biases. Attached Figure Description

[0026] Figure 1 This is a flowchart of a natural scene text detection method based on the hierarchical reorganization of semantic features.

[0027] Figure 2 This is a framework diagram for semantic feature reorganization.

[0028] Figure 3 This is a framework diagram for dual-network collaborative optimization.

[0029] Figure 4 This is a visualization of the effect of the method of the present invention. Detailed Implementation

[0030] To make the technical solution, objectives, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The following embodiments are merely examples provided to facilitate understanding of the present invention and are not intended to limit the scope of protection of the present invention. Other embodiments that can be obtained by those skilled in the art based on the disclosure of the present invention without creative effort should be considered to fall within the scope of protection of the present invention.

[0031] like Figure 1 As shown, this invention provides a natural scene text detection method based on hierarchical recombination of semantic features. The method mainly includes a visual concept matching module, a recombination weight controller, a semantic feature recombination module, and a dual-network collaborative optimization module.

[0032] In the encoder section, this invention selects ResNet-50 as the backbone network and utilizes FPN to extract multi-scale features and generate multi-level feature maps. , , , .right , Upsample to 1 / 4 and 1 / 16 of the input image respectively, which can be expressed by the following formula:

[0033]

[0034]

[0035] in Indicates an upsampling operation. Indicates the height of the original input image. This represents the width of the original input image. This represents the third layer feature map. This represents the fifth layer feature map. , Both represent the number of channels in the feature map.

[0036] like Figure 2 As shown, this invention proposes a semantic feature reorganization module to sever false visual associations, firstly by splicing... , As input to the regional reorganization unit:

[0037]

[0038] in, Indicates a channel connection operation. This represents the second layer feature map. This represents the third layer feature map after upsampling. Next, a clean visual concept library is constructed using a visual concept matching module, and confounding factors and purity factors are matched. Then, the input feature map is divided into 16×16 region feature blocks, and a recombination weight controller is used to obtain the recombination weights for each region. .

[0039] Next, this invention uses region recombination units to purify region-level visual clutter factors and reconstruct feature maps. First, region recombination feature blocks are constructed:

[0040]

[0041] in, Indicates the first Original feature blocks of each region Indicates the first Reorganization weights for each region This represents a lightweight multilayer perceptron structure containing three fully connected layers and a ReLU activation function, used to map the high-level semantic concept vectors output by CLIP to the image feature space. Indicates the first The visual concept after purification of the feature blocks in each region Indicates the first The original visual concept of each region feature block is then processed by adding the differences between the original feature blocks and the pure and mixed visual features to obtain the reconstructed feature blocks. By piecing together the original spatial locations, a complete reconstructed feature map of the region is obtained. :

[0042]

[0043] in, This indicates the first unit after being reorganized by the regional reorganization unit. Each feature block.

[0044] To eliminate interference from global visual clutter factors such as overall style and lighting, thereby improving the model's generalization ability, this invention designs a global reconstruction unit sequentially based on the region reconstruction unit. In this embodiment, the reconstructed feature map obtained by the region reconstruction unit is first downsampled to 1 / 16 of the input image: .in, Indicates step size is downsampling operation, This represents the complete feature map after region reorganization. Next, [the following will be done]... High-level feature map , The input feature map of the global recombination unit is obtained by concatenating the features and performing feature reduction and feature fusion using a 1×1 convolutional layer:

[0045]

[0046] in, This represents the convolution operation. Indicates a channel connection operation. This represents the reconstructed feature map of the downsampled region. This represents the fourth layer feature map. This represents the fifth layer feature map after upsampling.

[0047] Next, this embodiment uses a visual concept matching module to identify the global scene category of the image and its potential global visual bias at the whole image level, and constructs mixed-clean concept pairs, using cosine distance to calculate the global recombination weight:

[0048]

[0049] in, This represents the Sigmoid activation function. This represents the trainable weight parameters. Represents the actual global concept vector. Indicates and A matching pure global concept vector, Represents the cosine distance. This represents the trainable bias parameters.

[0050] Finally, this embodiment constructs a global reconstructed feature map to eliminate the interference of global confounding factors on the model:

[0051]

[0052]

[0053] in, This represents the input feature map of the global recombination unit. Indicates the global reorganization weight. This represents a mapping layer based on a multilayer perceptron, responsible for mapping high-level global semantic concept vectors to the image feature space. This represents the visual concept after overall purification. Represents the original visual concept of the whole. This represents the outer product operation. Indicates size is A matrix of all 1s Indicates the height of the original input image. This represents the width of the original input image.

[0054] In this embodiment, the global reconstructed feature map output by the semantic feature reconstruction module is input into the Region Proposal Network (RPN). Coarse text box proposals located in the original image coordinate system are generated using preset multi-scale anchor boxes. RoIAlign is used to precisely extract fixed-size feature blocks from the global reconstructed feature map for each candidate box proposed by the RPN. Subsequently, equidistant linear interpolation is performed on the four edges of the coarse text box to obtain an ordered sequence of boundary points, each defined by its two-dimensional planar coordinates. The dual-network collaborative optimization module then optimizes this point sequence point by point.

[0055] like Figure 3 As shown, this invention proposes a dual-network collaborative optimization module to optimize noisy initial text candidate boxes to target text boxes. This module is constructed as an iterative optimization process driven collaboratively by a boundary point displacement prediction network and a system error prediction network. During the training phase, at the feature map scale, a large number of intermediate state samples of text boxes are constructed by performing point-by-point linear interpolation between the initial and target text boxes. This is used as the input to the dual approximator. The boundary point displacement prediction network uses this to learn the coordinate offset required to move from a given input textbox state to the target textbox state. The process of constructing intermediate textbox state samples can be described as follows:

[0056]

[0057] in, Represents a continuous-time variable. This represents the initial state of the text box in the feature map coordinate system. This represents the state of manually annotated text boxes at the feature map scale. Each text box state is defined by the same ordered sequence of two-dimensional coordinate boundary points.

[0058] During the training phase, the coordinate offsets predicted by the model are compared with the target offsets. The boundary point displacement prediction network updates the model by minimizing the following loss function:

[0059]

[0060] in, express The square of the norm, Represents a continuous-time variable. This indicates the state of manually annotated text boxes at the feature map scale. This represents the initial state of the text box in the feature map coordinate system. This indicates that when the time variable takes the value The coordinate offset predicted by the boundary point displacement prediction network. The learnable parameters of the boundary point displacement prediction network are represented. Indicates by Towards The intermediate state of the text box is obtained through linear interpolation of point-by-point coordinates during the gradation process. The feature map representing the output of the global recombination unit. This represents a fixed-size feature block extracted by RoIAlign from the globally reconstructed feature map based on the candidate box coordinates output by the region proposal network. The states of the text boxes are all defined by the same ordered sequence of two-dimensional coordinate boundary points.

[0061] Next, the systematic error of the upstream model is eliminated by independently training a systematic error prediction network. The loss function of the systematic error prediction network is as follows:

[0062]

[0063] in, express The square of the norm, This represents the target value of the system error for the current text attribute. This indicates that when the time variable takes the value The system error predicted by the system error prediction network is... This represents the learnable parameters of the system error prediction network. This represents the initial state of the text box in the feature map coordinate system. Indicates by The intermediate states of the text box are obtained through linear interpolation of point-by-point coordinates during the gradual transition of the manually annotated text box state to the feature map scale. The feature map representing the output of the global recombination unit. This represents a fixed-size feature block extracted by RoIAlign from the global reconstructed feature map based on the coordinates of the candidate boxes output by the region proposal network. This represents a continuous-time variable. Systematic error prediction networks can adaptively compensate for individual differences when fitting the average systematic error of similar texts.

[0064] During the inference phase, the initial text candidate boxes output by the region proposal network are used as a starting point for iterative optimization at the feature map scale. The text boxes are updated by subtracting the systematic error from the systematic error prediction network based on the coordinate offset predicted by the boundary point displacement prediction network. After a few iterations, the final coordinates are mapped back to the original scale output.

[0065] The trained model is used to perform a real-world text detection task in natural scenes.

[0066] To verify the effectiveness of the method of the present invention in the task of natural scene text detection, the results of the natural scene text detection by the method are visualized, as shown below. Figure 4 As shown in the image, this visualization intuitively presents the practical effectiveness of the method of this invention in natural scene text detection tasks.

[0067] The method proposed in this invention is not only applicable to general natural scene text detection tasks, but also has wide applicability in areas such as intelligent transportation and retail security. For example, by accurately detecting text areas such as road signs and license plates, it can assist in traffic sign recognition and illegal license plate capture, providing data support for traffic management; by detecting and recognizing product labels, it can assist in completing product inventory checks and price tag information verification, improving retail operational efficiency.

[0068] It should be noted that the terms "comprising," "including," or other similar terms used in this specification are intended to indicate non-exclusive inclusion. That is, a process, method, article, or apparatus that comprises a series of elements includes not only the elements expressly listed, but may also include other elements not expressly listed or elements inherent to the process, method, article, or apparatus.

[0069] Furthermore, the execution order of the steps in the method described in this invention should not be limited to the order shown or described in the specification. Without conflict, related steps may be executed in a substantially simultaneous or reverse order according to their functions, and technical features described in different embodiments may also be reasonably combined.

[0070] The specific embodiments of the present invention have been described above with reference to the accompanying drawings. However, these embodiments are merely illustrative and are not intended to limit the scope of protection of the present invention. Under the guidance of the essence of this invention, those skilled in the art can make various changes or modifications to the present invention without departing from the spirit and scope of the claims, and all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. A method for detecting text in natural scenes based on hierarchical reorganization of semantic features, characterized in that, Includes the following steps: Visual concept matching steps: Visual concept clusters are constructed using CLIP encoder and clustering algorithm. Based on text co-occurrence rate, mixed concepts and pure concepts are further divided. Finally, mixed-pure concept matching pairs are obtained based on the similarity between concepts, forming a pure visual concept library. The steps for determining the reorganization weights are as follows: Using the reorganization weight controller, the regional reorganization weights are obtained by combining the difference in concept purity and the estimated text saliency, and the global reorganization weights are obtained based on the cosine distance. Semantic feature reorganization steps: The semantic feature reorganization module purifies visual clutter factors at both the regional and global levels to obtain reorganized clean feature maps. Specifically, FPN is used to extract multi-scale features from natural scene images to generate multi-level feature maps. The stitched low-level feature maps are matched in a clean visual concept library. The reorganization weight controller calculates the regional reorganization weight of the regional feature blocks based on the difference between the matched local clean visual features and the local cluttered visual features. The regional reorganization unit adds the difference to the original feature blocks according to the regional reorganization weight to obtain the regional reorganized feature map. The regional reorganized feature map is stitched with the high-level feature map, and a 1×1 convolutional layer is used to achieve feature fusion and feature dimensionality reduction to obtain the input feature map. The input feature map is then globally matched in a clean visual concept library. The global reorganization weight is obtained using cosine distance based on the matched clean visual features. The global reorganization unit adds the same adjustment vector at all spatial locations of the feature map to obtain the globally reorganized feature map. Dual-network collaborative optimization steps: Using a bidirectional approximator, predict the correction amount used to correct the initial text candidate boxes, and iteratively update the text detection boxes.

2. The method according to claim 1, characterized in that, The visual concept matching step specifically includes: The CLIP encoder is used to extract high-level semantic concept vectors and construct the overall feature matrix. The K-Means++ algorithm is used to cluster the feature vectors to obtain multiple visual concept clusters; The co-occurrence rate of visual concepts and text is statistically analyzed, and a threshold is set to further divide mixed concepts into pure concepts. Mixed-pure concept matching pairs are obtained based on similarity.

3. The method according to claim 1, characterized in that, The specific steps for determining the reorganization weights include: adaptively determining regional reorganization weights based on differences in concept purity and text saliency; determining global reorganization weights based on cosine distance; and so on. The region recombination weights of each region feature block are represented as follows: ， ， in, This represents the Sigmoid activation function. , All represent trainable positive weight parameters. Indicates the first The difference in conceptual purity of each region's feature blocks Represents the concept purity function. Indicates the first The visual concept after purification of the feature blocks in each region Indicates the first The original visual concept of each region's feature blocks. Indicates the first Preliminary estimates of the text saliency of feature blocks in each region This represents the trainable bias parameters.

4. The method according to claim 1, characterized in that, The dual-network collaborative optimization steps specifically include: Boundary point displacement prediction network is used to predict the coordinate offset required from a given input text box state to a target text box state. System error prediction network is used to predict the inherent and consistent system errors of the upstream model caused by model bias and imbalanced training data. By combining coordinate offset and system error, the initial text candidate box is jointly corrected, and a high-precision text box is finally obtained.

5. The method according to claim 4, characterized in that, At the feature map scale, intermediate text box state samples between the initial text box and the target text box are obtained through point-by-point linear interpolation of coordinates. ， in, Represents a continuous-time variable. This represents the initial state of the text box in the feature map coordinate system. This indicates the state of manually annotated text boxes at the feature map scale.

Images