Character recognition method based on efficient fusion encoder and text center curve prediction

Abstract

The invention relates to the field of computer vision and artificial intelligence, in particular to a character recognition method based on efficient fusion encoder and text center curve prediction. According to the method, image features are obtained through a feature extraction module, high-semantic information is captured through an efficient fusion encoder, and meanwhile redundant calculation of low-level features is reduced. In combination with a feature modeling module, the method can accurately predict boundary points and text center curves of text instances, and text instance classification, character classification, center curve point prediction and boundary point prediction are completed at the same time through a parallel prediction mechanism. And finally, a decoding module is combined to process a character classification result, and text content is generated. Experimental results show that the method has high recognition precision, good real-time performance and high robustness in complex scenes, and is suitable for various application scenes of character detection and recognition.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and artificial intelligence, and in particular to a text recognition method based on an efficient fusion encoder and text center curve prediction. Background Art

[0002] Text recognition is a key research area in computer vision, widely used in fields such as map navigation, autonomous driving, and image retrieval. End-to-end text recognition methods typically involve two key steps: text detection and text recognition. Text detection identifies text regions in an image, while text recognition extracts character information from the detected regions. Currently, mainstream text recognition methods can be categorized into three types: region-of-interest (ROI)-based methods, segmentation-based methods, and Transformer-based methods. ROI-based methods extract features from the image's ROI through pooling operations and pass these features to the recognition module for further processing. However, these methods rely on intermediate region connectors to combine detection and recognition tasks, which can be inefficient when dealing with irregularly laid out or densely packed text. Segmentation-based methods detect text regions and characters through pixel-level segmentation, enabling recognition of text of arbitrary shapes. While segmentation-based methods can effectively handle complex text shapes, they often rely on post-processing steps (such as character-level grouping or connection rules) to assemble multiple characters into complete text. In dense or overlapping text scenarios, these post-processing steps can introduce errors, affecting the final recognition performance. Transformer-based methods employ an encoder-decoder architecture to jointly handle text detection and recognition tasks within a single model. Through a dense self-attention mechanism, the Transformer model strengthens the interaction between text detection and recognition, thereby improving overall performance. However, the Transformer's attention mechanism is computationally intensive, potentially leading to increased computational overhead and excessive resource consumption, limiting its scalability in real-time applications. Summary of the Invention

[0003] The purpose of the present invention is to provide a text recognition method based on an efficient fusion encoder and text center curve prediction. The method aims to reduce computational complexity and improve inference speed while maintaining high recognition accuracy and robustness. It can effectively recognize complex and diverse texts in various scenarios and has good practicality and application value.

[0004] To achieve the above objectives, the present invention provides a text recognition method based on an efficient fusion encoder and text center curve prediction, which specifically includes the following steps:

[0005] Step 1: Image preprocessing: Obtain the images to be processed. The images can come from cameras, scanning devices, etc. to form an input image set. First, adjust the image size to ensure that the image size is divisible by 32 while maintaining the original image ratio. To enhance the generalization ability of the model, data augmentation methods are used to process the training data, including but not limited to random rotation, instance-aware random cropping, random image resizing, and color jittering. This improves the model's robustness and adaptability to text instances in different scenarios.

[0006] Step 2: Image feature extraction, build a text recognition network, input the text image to be processed into the network, and extract multi-scale feature information.

[0007] Step 3: Use an efficient fusion encoder for multi-scale feature fusion. An efficient fusion encoder is used to capture high-semantic relationships through the attention-based intra-scale feature interaction module (AIFI), and dynamically adjust feature weights through the bidirectional cross-scale fusion module (BCFM) combined with the efficient multi-scale attention mechanism (EMA) to achieve efficient extraction and fusion of multi-scale features, thereby improving the performance of text detection and recognition.

[0008] Step 4: Predict Bezier control points and text center curves. By combining an efficient fusion encoder with a three-layer MLP network, Bezier control points and text center curves are predicted, and high-confidence detection suggestions are screened to achieve accurate positioning of text detection.

[0009] Step 5: Multi-task prediction: adopt a multi-task prediction mechanism and use parallel prediction heads to complete text classification, character classification, center curve prediction and boundary point prediction to accurately locate and identify text instances.

[0010] Step 6: CTC decoding and result generation: The connectionist temporal classification (CTC) decoder is used to calculate the unaligned path probability of the character classification results, remove redundant characters, generate text content, and combine post-processing mechanisms to improve the accuracy and reliability of the recognition results.

[0011] Step 7: Model training. A two-stage training strategy is used. First, pre-training is performed on multiple datasets, followed by fine-tuning on the Total-Text dataset. The Adam optimizer and dynamic learning rate adjustment mechanism are combined to ensure the stability and efficiency of model training.

[0012] Compared with the prior art, the present invention has significant advantages and technical effects:

[0013] The present invention realizes multi-scale feature fusion through an efficient fusion encoder, captures high-semantic information, and reduces the redundant calculation of low-level features. Combined with a multi-layer perceptron (MLP), the present method can accurately predict the boundary points and text center curves of text instances. Through parallel prediction heads, the present invention can solve multiple subtasks at the same time, including text instance classification, character classification, center curve point prediction, and boundary point prediction. Finally, the CTC decoder is used to process the character classification results to generate text content. Experimental results show that the method of the present invention has high recognition accuracy and robustness, can effectively recognize text in various complex scenes, and exhibits excellent practicality and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following is a brief description of the accompanying drawings. It should be noted that the following drawings are only used to illustrate some embodiments of the present invention. People familiar with the art can still design other types of drawings based on these drawings without the need for innovative work. The contents of the drawings are as follows:

[0015] Figure 1 A flowchart of a text recognition method based on an efficient fusion encoder and text center curve prediction in an embodiment of the present invention is shown;

[0016] Figure 2 The following is a schematic diagram showing the operation of an efficient fusion encoder for multi-scale feature extraction and fusion in an embodiment of the present invention;

[0017] Figure 3 A schematic diagram of bidirectional cross-scale feature fusion in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0018] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the present invention is not limited thereto.

[0019] Example:

[0020] Reference Figure 1 A text recognition method based on an efficient fusion encoder and text center curve prediction includes the following steps:

[0021] (1) Obtaining the text image to be processed from an image acquisition device (such as a camera, scanner, etc.);

[0022] (2) Image feature extraction, refer to Figure 2For the input text image to be processed, it first passes through the ResNet50 network to extract multi-scale feature maps, including the output features of the last three stages S3, S4 and S5. The extracted feature maps are then sent to the efficient fusion encoder for processing;

[0023] (3) Using efficient fusion encoder for multi-scale feature fusion;

[0024] Further, refer to Figure 3 , a highly efficient fusion encoder is used for multi-scale feature fusion. The encoder includes an internal scale feature interaction module based on the attention mechanism and a bidirectional cross-scale fusion module. First, feature maps are extracted from the last three stages (S3, S4, and S5) of the ResNet50 network. The internal scale feature interaction module is only applied to the S5 feature map containing high semantic information to capture the semantic relationship of text instances and reduce the redundant calculation of low-level features. It is calculated using the following formula:

[0025] Q=K=V=Flatten(S5)

[0026] F5=Reshape(AIFI(Q,K,V))

[0027] Among them, Q, K, and V are all obtained by flattening the feature map S5. AIFI represents the intra-scale feature interaction operation, which uses the self-attention mechanism. Reshape is used to adjust the size of the output features to adapt to subsequent tasks. Then, a bidirectional cross-scale fusion module performs top-down global context capture and bottom-up local detail extraction. This process, combined with an efficient multi-scale attention mechanism (EMA), dynamically adjusts feature weights to achieve efficient fusion of multi-scale features, enhancing the network's capabilities in multi-scale text detection and recognition tasks. Specifically, the bidirectional cross-scale fusion module first maps high-level feature maps to the same number of channels through 1×1 upsampling convolutions, fuses them with low-level feature maps, and applies the EMA attention mechanism to optimize feature weights, enhancing the expressive power of multi-scale features. Simultaneously, a 3×3 downsampling convolution is used to reduce the resolution of the low-level feature maps, which are then fused with the high-level feature maps. The EMA attention mechanism is again applied to optimize feature weights, thereby improving the discriminability and fusion effect of multi-scale features.

[0028] (4) Bezier control points and text center curve prediction;

[0029] Furthermore, in feature fusion, a 3-layer multi-layer perceptron (MLP) is used to predict the position of each pixel and obtain the offset of the 4 Bezier control points, thereby fitting the central curve of the text. Bezier curve is a mathematical model widely used in computer graphics and image processing. The shape of a smooth curve is defined by control points, and the position of the control points determines the specific trend of the curve. Specifically, given the normalized coordinates of a pixel point in and Represents the normalized coordinates of the pixel in the x and y directions respectively. Normalization refers to converting the coordinates of the pixel to a standard range (such as [0,1]). Based on the normalized coordinates, the Bezier control point set corresponding to the pixel can be calculated Calculated by the following formula:

[0030]

[0031] in and is the offset predicted by MLP, σ is the Sigmoid function, which is used to normalize the coordinates of the control points to a reasonable range, σ -1 This is the inverse operation of the activation function, mapping the normalized pixel coordinates back to the original coordinate system. Then, the predicted control points and their corresponding center curve suggestions and corresponding confidence scores are input into the MLP network to further obtain multiple corrected offsets. A linear layer is used to classify whether text exists, and the top 100 are selected as text detection suggestions to complete the detection of text instances.

[0032] (5) Multi-task prediction;

[0033] After obtaining the query vector, multiple subtasks are solved simultaneously through parallel prediction heads, including text instance classification, character classification, center curve point prediction, and boundary point prediction. Specifically, text instance classification uses linear projection to achieve binary classification of text and background. During the inference phase, the average classification score of all sampled points on the center curve is calculated as the confidence score of the text instance; character classification uses linear projection to predict the character category of uniformly sampled query points on the center curve, determining the character category corresponding to each point, including the background category; center curve point prediction inputs the explicit coordinates of the reference point into the three-layer MLP head, predicting the coordinate offset from the reference point to the true center curve point to fit the center curve of the text; boundary point prediction uses the three-layer MLP head to predict the coordinate offset from the reference point to the true points of the upper and lower boundary curves, completing the precise description of the text instance boundary.

[0034] (6) CTC decoding and result generation;

[0035] The character classification results output by the decoder are fed into the Connectionist Temporal Classification (CTC) decoder, which then calculates the probability of the character sequence and generates a text sequence that best matches the input features. CTC is a decoding method commonly used to address the problem of inconsistent lengths between input and output sequences. By introducing "whitespace" and a dynamic path aggregation mechanism, it allows for flexible alignment of input and output sequences without explicit alignment annotations. The formula is as follows:

[0036]

[0037] Among them, P(Y|X) represents the conditional probability of output sequence Y given the input sequence X. This means finding the sequence with the highest conditional probability among all possible output sequences Y. CTC decoding aggregates the probabilities of unaligned paths, removes duplicate characters and whitespace, and outputs the final character sequence. Furthermore, a post-processing mechanism combines the geometric position information and recognition scores of the character classification results to correct the text content, further improving recognition accuracy and reliability.

[0038] (7) Model training;

[0039] Initialization is performed using pre-trained weights from the ImageNet dataset. To enhance the network's ability to process text in scenes of arbitrary shapes, pre-training is first performed on the SynthText150K, ICDAR 2013, ICDAR 2015, and CTW1500 datasets. On this basis, the model is fine-tuned on the Total-Text training set and evaluated on the Total-Text test set to verify the model's detection accuracy and inference speed in text detection and recognition tasks. The model is iterated 500,000 times in total. During training, the Adam optimizer is used for parameter optimization, and the initial learning rate is set to 1×10 -4 ,After 420,000 iterations, a learning rate decay coefficient of 0.95 is used for dynamic adjustment to ensure the stability and efficiency of training.

[0040] (8) Real-time reasoning and result output;

[0041] To validate the performance of the proposed algorithm in text detection and recognition tasks, an evaluation was conducted on the Total-Text dataset. Experimental results show that the proposed algorithm achieves 92.3% precision, 81.8% recall, and an F1 score of 86.7% in text detection tasks. The F1 score is a comprehensive measure of precision and recall, effectively reflecting the overall performance of the algorithm. In text recognition tasks, the F1 score reached 84% and the inference speed reached 16.5 frames per second, fully demonstrating the high accuracy and real-time performance of the proposed algorithm in text detection and recognition.

[0042] The above disclosure is only a preferred embodiment of the present invention, and certainly cannot be used to limit the scope of the rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A text recognition method based on efficient fusion encoder and text center curve prediction, characterized in that: The following steps are involved: Step 1: Image preprocessing, obtaining the image to be processed, resizing the image, and performing data enhancement processing; Step 2: Image feature extraction, inputting the image into a text recognition network to extract multi-scale feature information; Step 3: using an efficient fusion encoder to perform multi-scale feature fusion, fusing the multi-scale features of the image through the efficient fusion encoder to generate an optimized feature representation; Step 4: Bezier control point and text center curve prediction, combining the efficient fusion encoder and the multi-layer perceptron network to predict Bezier control points and text center curves, and screening high confidence detection results to achieve accurate positioning of the text area; Step 5: Multi-task prediction: Through the parallel prediction mechanism, text classification, character classification, center curve prediction and boundary point prediction are completed to achieve text positioning and recognition; Step 6: CTC decoding and result generation: The CTC decoder processes the character classification results, removes redundant characters, generates text content, and combines the post-processing mechanism to optimize the recognition accuracy. Step 7: Model training, using a two-stage training strategy, including pre-training on multiple datasets and fine-tuning on the target dataset to improve the generalization performance and recognition accuracy of the model.

2. The text recognition method based on efficient fusion encoder and text center curve prediction according to claim 1 is characterized in that: In the step (3), an efficient fusion encoder is used to perform multi-scale feature fusion, and the encoder includes an attention-based intra-scale feature interaction module (Attention-based Intra-scale Feature Interaction, AIFI) and a bidirectional cross-scale fusion module (Bidirectional Cross-scale Fusion Module, BCFM). The AIFI module is used to model the feature relationship of the feature map of high semantic information, and the BCFM module is used to realize top-down global context information capture and bottom-up local detail extraction, and dynamically optimize the feature weights in combination with the multi-scale attention mechanism, so as to realize efficient fusion of multi-scale features.

3. The text recognition method based on efficient fusion encoder and text center curve prediction according to claim 2 is characterized in that: In the step (4), the offset of the Bezier control point is predicted by using a multi-layer perceptron (MLP) network for the fused feature map to fit the text center curve, and high confidence detection results are selected to achieve accurate positioning of the text.

4. The text recognition method based on efficient fusion encoder and text center curve prediction according to claim 3 is characterized in that: In the step (5), a multi-task prediction mechanism is adopted to simultaneously complete text classification, character classification, center curve prediction and boundary point prediction through parallel prediction heads, thereby achieving accurate recognition of text instances.

5. The method for character recognition based on efficient fusion encoder and text center curve prediction according to claim 4, characterized in that: In the step (6), the character classification results are subjected to non-aligned path probability calculation by a connectionist temporal classification (CTC) decoder, redundant characters are removed to generate text content, and the accuracy and reliability of the recognition results are optimized in combination with a post-processing mechanism.

6. The method for character recognition based on efficient fusion encoder and text center curve prediction according to claim 5, characterized in that: In step (7), a two-stage training strategy is adopted. First, the pre-trained weights of the ImageNet dataset are used for initialization, and pre-training is performed on the SynthText150K, ICDAR2013, ICDAR2015, and CTW1500 datasets. On this basis, the model is fine-tuned on the Total-Text training set and evaluated on the Total-Text test set. The model is iterated 500,000 times in total, and the Adam optimizer is used for parameter optimization. The initial learning rate is set to 1×10 -4 , after 420,000 iterations, it is dynamically adjusted with a decay coefficient of 0.95 to ensure the stability and efficiency of training.

Images