Double-stage double-branch handwritten Chinese formula mixed arrangement image recognition method

Abstract

The invention relates to the technical field of computer vision and image recognition, and discloses a double-stage double-branch handwritten Chinese formula mixed arrangement image recognition method, which comprises the following steps of: performing accurate detection and type classification on all characters in an input image by utilizing an improved YOLOv8s target detection network, and obtaining bounding boxes and initial category information of the characters; respectively inputting the cut image into a handwritten Chinese recognition branch and a handwritten mathematical formula recognition branch according to character categories, and performing feature extraction and semantic analysis by a specific pre-trained neural network; sorting and rearranging the recognition results of the double branches according to a natural reading sequence based on the character space position; according to the method, secondary verification based on multi-modal large model fine tuning is introduced, context-aware anomaly detection and correction are carried out on an identification result, and the method has good universality and expansibility and can be widely applied to scenes such as examination paper marking, learning assistance and literature digitization in the education field.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and image recognition, and particularly relates to a two-stage double-branch handwritten Chinese formula mixed arrangement image recognition method. BACKGROUND

[0002] The mixed arrangement document of handwritten Chinese and mathematical formula is a common information carrier in educational assessment (such as test papers, homework), scientific research notes and historical documents. Realizing automatic and accurate recognition of such mixed arrangement images has important value for educational intelligence (such as automatic marking, learning analysis) and literature digitization. However, this task faces many technical challenges:

[0003] (1) Complexity of character type and layout: the image contains both freehand written Chinese characters and mathematical formulas with two-dimensional spatial structure, and there are significant differences in shape, size and spatial relationship between the two, and the layout is free, which may exist in various forms such as mixed arrangement within a line and formula block embedding.

[0004] (2) Difficulty of detection and classification: handwritten characters often have problems such as different sizes, stroke connection, and tilting deformation, making it difficult for traditional detection models to accurately locate each character and correctly classify it as "Chinese character" or "formula symbol".

[0005] (3) Heterogeneity of recognition model: Chinese recognition is usually considered as a sequence labeling problem, while mathematical formula recognition needs to understand its two-dimensional grammar structure and convert it into a structured markup language (such as LaTeX format), and a single recognition model cannot handle both tasks.

[0006] (4) Context association and error propagation: the semantic understanding of mixed arrangement text depends on the context relationship between Chinese and formula. In the phased recognition process, the errors of the previous modules (such as detection, classification) will be propagated to the subsequent recognition modules, affecting the overall accuracy of the final result.

[0007] At present, the existing handwritten document recognition methods are mostly focused on single language or pure mathematical formula, and lack effective processing solutions for mixed arrangement of Chinese and formula. Some methods try to use end-to-end models, but they lack generalization ability when facing complex mixed arrangement layout. Therefore, there is an urgent need for a recognition method that can cooperatively process two types of heterogeneous content and improve robustness through subsequent verification. SUMMARY

[0008] In view of the technical problems of single recognition content, insufficient generalization ability and insufficient robustness in the prior art, the application provides a two-stage double-branch handwritten Chinese formula mixed arrangement image recognition method, and the technical scheme is as follows:

[0009] Step 1: Perform end-to-end character-level object detection on the input image of handwritten Chinese formulas, and output the bounding box coordinates and category label of each character;

[0010] Step 2: Based on the above category labels, input the cropped character images into the handwritten Chinese character recognition branch and the handwritten mathematical formula recognition branch respectively, and use the optimized ResNet18 network and encoder-decoder architecture for recognition respectively;

[0011] Step 3: Based on the spatial relationship of the characters in the original image, sort and splice the recognition result sequence of the two branches in the natural reading order to form a preliminary mixed text;

[0012] Step 4: Use the fine-tuned visual-language model to perform cross-modal matching and understanding of the original image and the preliminary mixed text, detect and correct recognition errors, and output the final optimized recognition text.

[0013] Furthermore, step 1 performs character detection and classification based on an improved YOLOv8s network, introduces an attention mechanism into its neck network, and optimizes the anchor box size and loss function.

[0014] Furthermore, the anchor frame dimensions are specifically designed as follows: using clustering to design the anchor frame dimensions, the aspect ratio is designed to be 0.5-1.5 for small-sized characters, with a size range of 8×8-24×24; the aspect ratio is designed to be 1.0-2.5 for connected characters, with a size range of 24×32-48×64.

[0015] The loss function adopts a combination of weighted classification loss, localization loss, and confidence loss, with the specific formula as follows:

[0016]

[0017] in, Total loss; Focal Loss is used for classification. To determine the loss, CIoU is used; For confidence loss, BCEWithLogitsLoss is used; , , These are the weighting coefficients for the three loss terms.

[0018] Furthermore, in step 2, the handwritten Chinese character recognition branch is used to recognize Chinese characters in the character image, and a ResNet18 network pre-trained on the CASIA-HWDB handwritten Chinese character dataset is used.

[0019] Furthermore, the specific operation of the handwritten Chinese character recognition branch is as follows:

[0020] 2.1.1 During the data preparation stage, the original images of the CASIA-HWDB handwritten Chinese character dataset were transformed and expanded, including appearance perturbation, adjustment of image contrast and brightness, addition of noise, application of Gaussian blur and image sharpening filtering, and geometric transformation, to simulate the imaging differences and writing style changes in real scenes.

[0021] 2.1.2. ResNet18 is used as the backbone network, and its weights, which are pre-trained on the CASIA-HWDB handwritten Chinese character dataset, are used to focus on the stroke and structural characteristics of handwritten Chinese characters.

[0022] 2.1.3 In the ResNet18 framework, the channel attention module and the spatial attention module are integrated sequentially. The channel attention module adaptively recalibrates the weights of each feature channel to highlight important features; the spatial attention module learns the spatial importance distribution of the feature map and focuses on the key stroke regions of the character.

[0023] Furthermore, in step 2, the handwritten mathematical formula recognition branch is used to recognize mathematical formulas in character images and convert them into a structured sequence format. It adopts an encoder-decoder architecture based on an attention mechanism and introduces symbol counting as an auxiliary task.

[0024] Furthermore, the specific operation of the handwritten mathematical formula recognition branch is as follows:

[0025] 2.2.1 DenseNet, pre-trained on ImageNet, is used as the encoder backbone network. DenseNet adopts a dense connection mechanism to alleviate the gradient vanishing problem in deep networks and extract two-dimensional feature maps containing rich multi-scale contextual information. These two-dimensional feature maps serve as the common input to the subsequent decoder and multi-scale counting module.

[0026] 2.2.2. A weakly supervised multi-scale counting module is introduced. This module automatically learns and predicts the frequency of each type of symbol in the image using only sequence labeling. Specifically:

[0027] 1) Use parallel convolutional branches to process the two-dimensional feature map output by the encoder;

[0028] 2) SENet introduces a channel attention mechanism after each parallel convolution branch to adaptively recalibrate the feature channel responses;

[0029] 3) Through Convolution reduces the number of feature channels to the number of symbol classes. And generated by Sigmoid A pseudo-density map of the same size as the input 2D feature map. Reflecting the first The probability of the class symbol appearing in space is obtained by analyzing each pseudo-density map. Perform a global summation to obtain the prediction count for the symbol;

[0030] 4) Indirect supervision is achieved by comparing predicted counts with the actual sequence, enabling symbol counting without location point labeling;

[0031] 2.2.3 This branch uses a Transformer structure decoder, which gradually focuses on different regions of the feature map through an attention mechanism and generates LaTeX sequences autoregressively.

[0032] Furthermore, step 3 specifically involves the following steps:

[0033] 3.1 Sort the coordinates according to the center of the bounding box coordinates of all characters, following the reading order from left to right and from top to bottom;

[0034] 3.2 Calculate the intersection-union ratio of adjacent coordinate frames, and parse and filter characters with overlapping areas greater than the threshold. Among them, multiple consecutive characters identified as mathematical formulas are treated as a whole unit for sorting and filtering.

[0035] 3.3. The sorted and filtered Chinese recognition results are combined with the formula recognition results to form a continuous preliminary mixed text that conforms to human reading habits.

[0036] Furthermore, in step 4, the generalized 2-VL-2B-Instruct is selected as the base model, and the LoRA method is used to fine-tune the model. LoRA adds low-rank adapters to the linear bypass in the original Transformer layer, trains only these newly added adapters with a very small number of parameters, and does not update all model weights, so as to achieve efficient and stable model adaptation.

[0037] Furthermore, step 4 specifically involves the following steps:

[0038] 4.1 Construct a fine-tuned dataset adapted to the specific task of image recognition of handwritten Chinese formula mixed text. Each sample in the fine-tuned dataset is a triplet containing the original image, the initial mixed text, and the target text.

[0039] 4.2 In the inference stage, the original image to be verified and the preliminary mixed text generated in step 3 are input into this fine-tuned visual-language large model. The cross-modal alignment capability of this large model is used to deeply understand the semantic consistency between the image content and the text description. Finally, the anomalies in the preliminary text are identified, including symbol repetition, missing key characters, misidentification of context characters, and formula structure errors. The accurate recognized text is obtained through correction and optimization.

[0040] Beneficial effects:

[0041] This method has good versatility and scalability, and can be widely applied to scenarios such as exam marking, learning assistance, and document digitization in the education field. This invention can efficiently and accurately complete end-to-end recognition of mixed images of handwritten Chinese formulas, and significantly improve the accuracy of the results by utilizing the semantic understanding capabilities of a large model in the final stage. Attached Figure Description

[0042] Figure 1 The overall flowchart of the two-stage, two-branch image recognition method for mixed handwritten Chinese formulas;

[0043] Figure 2 Architecture diagram of the multi-scale counting module in the handwritten mathematical formula recognition branch;

[0044] Figure 3 This is a schematic diagram illustrating the principle of secondary verification based on a fine-tuned visual-language model. Detailed Implementation

[0045] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0046] The embodiments of the present invention are implemented using the PyTorch deep learning framework on a computer system equipped with an NVIDIA GPU.

[0047] like Figure 1 As shown, the two-stage, two-branch image recognition method for mixed handwritten Chinese formulas includes the following steps:

[0048] Step 1: Character detection and classification based on improved YOLOv8s.

[0049] 1.1 Data preparation: Collect and label images of handwritten Chinese formulas in mixed layout. The labeling information includes the bounding box and category ("Chinese character" or "mathematical formula") of each character.

[0050] 1.2 Model Training: The image was used to train an improved YOLOv8s model. The improvements included integrating a lightweight attention mechanism (CBAM module) into the neck network of the model, redesigning the anchor box size using K-means clustering based on the aspect ratio statistics of the characters in the training set, and adjusting the loss function. Specifically:

[0051] 1.2.1 Anchor frame size optimization: Six anchor frame sizes were designed using clustering: 8×8, 12×12, and 24×24 for small characters (including square and narrow-height types); and 24×32, 48×32, and 64×48 for connected characters (including single-character connected, double-character horizontally connected, and multi-character connected).

[0052] 1.2.2 Loss Function Optimization: The loss function formula is as follows:

[0053]

[0054] in, Total loss; Focal Loss is used for classification. To determine the loss, CIoU is used; For confidence loss, BCEWithLogitsLoss is used; , , These are the weighting coefficients for the three loss terms.

[0055] 1.3 Inference Application: After training, inference is performed on the image of handwritten Chinese formulas, and the coordinates and categories of all characters are output.

[0056] Step 2: Dual-branch recognition.

[0057] 2.1 Handwritten Chinese character recognition branch.

[0058] 2.1.1 Data Augmentation: During the training phase, various data augmentations are applied to the input Chinese character images in real time, including randomly adjusting contrast and brightness within ±30%, adding salt and pepper noise and Gaussian noise, applying random angle (-15° to +15°) rotation and random scaling (0.8x to 1.2x).

[0059] 2.1.2 Model Training: ResNet18 weights pre-trained on the CASIA-HWDB dataset were loaded and fine-tuned on a handwritten Chinese character dataset. Channel attention (SE module) and spatial attention (spatial attention submodule in CBAM) were inserted after specific layers of ResNet18. The model was trained using the cross-entropy loss function and the Adam optimizer.

[0060] 2.2 Recognition of handwritten mathematical formulas.

[0061] 2.2.1 Model Construction: The encoder uses DenseNet pre-trained on ImageNet to output feature maps. For example... Figure 2 As shown, the multi-scale counting module uses and Two parallel convolutional branches process the feature map, followed by an SE module, ultimately generating several pseudo-density maps for the mathematical symbol categories and summing them to obtain a counting vector. The decoder is a Transformer decoder.

[0062] 2.2.2 Joint Training: The total loss function for the handwritten mathematical formula recognition branch is the classification loss (cross-entropy loss) and the counting loss (smoothing loss). The model is a weighted sum of losses, with weights set to 1.0. The model is trained end-to-end using the Adadelta optimizer.

[0063] Step 3: Reorganize and rearrange the recognition results.

[0064] 3.1 Obtain the coordinates of the center point of all character bounding boxes. .

[0065] 3.2 The sorting algorithm first follows the... The coordinates are sorted in ascending order to determine the text lines, within the same line ( (If the difference is less than 0.5 times the average character height) then press Sort the coordinates in ascending order.

[0066] 3.3 For adjacent characters with an IoU greater than 0.5, it is considered that there is a duplicate recognition, and the characters are filtered, optimized or merged into a whole.

[0067] 3.4. Following the reading order of "from top to bottom and from left to right", the Chinese character recognition results and mathematical formula recognition results (LaTeX sequence) are concatenated into a preliminary mixed text.

[0068] Step 4, as follows Figure 3 As shown, secondary verification is based on fine-tuning of the visual-language large model.

[0069] 4.1 Dataset Construction: Approximately 10,000 fine-tuning instructions were constructed. Each sample format is as follows:

[0070] {

[0071] Instruction: "Please correct the following errors in the recognized text:",

[0072] "Input": <Original Image> <Preliminary Text Recognition>,

[0073] "Output": <Corrected standard text>

[0074] }

[0075] 4.2 Model Fine-tuning: The Tongyi 1000 Questions 2-VL-2B-Instruct was used as the base model. The LoRA fine-tuning method was employed, adding adapters to the Q, K, V, and O projection layers of the model, with a rank of 64 and an alpha value of 16. The AdamW optimizer was used, with a learning rate of 3e-4 and a batch size of 8, for training for 10 epochs.

[0076] 4.3 Reasoning and Error Correction: During reasoning, the original image and the preliminary mixed text output in step 3 are concatenated according to the instruction template and input into the fine-tuned model. The model will automatically generate and return the corrected final text.

[0077] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A two-stage double-branch handwritten Chinese formula mixed arrangement image recognition method, characterized in that Comprising the following steps: Step 1, end-to-end character-level target detection is performed on the input handwritten Chinese formula mixed layout image, and the bounding box coordinates and class labels of each character are output; Step 2, according to the above class labels, the cropped character images are input into the handwritten Chinese character recognition branch and the handwritten mathematical formula recognition branch respectively, and the optimized ResNet18 network and the encoder-decoder architecture are used for recognition respectively; Step 3, according to the spatial position relationship of the characters in the original image, the recognition result sequences of the double branches are sorted and spliced in the natural reading order to form the preliminary mixed layout text; Step 4, the original image and the preliminary mixed layout text are matched and understood across modalities by using the fine-tuned visual-language large model, and the recognition errors are detected and corrected to output the final optimized recognition text.

2. The two-stage and double-branch handwritten Chinese formula mixed arrangement image recognition method of claim 1, wherein: The step 1 performs character detection and classification based on the improved YOLOv8s network, introduces an attention mechanism in the neck network, and optimizes the anchor box size and the loss function.

3. The two-stage and double-branch handwritten Chinese formula mixed arrangement image recognition method of claim 2, wherein: The anchor box size is specifically: the anchor box size is designed by clustering, the width-height ratio is 0.5-1.5 for small size characters, and the size range is 8x8-24x24; the width-height ratio is 1.0-2.5 for connected characters, and the size range is 24x32-48x64; The loss function adopts a combination form of weighted classification loss+positioning loss+confidence loss, and the specific formula is: where, is the total loss; is the classification loss, using FocalLoss; is the localization loss, using CIoU; is the confidence loss, using BCEWithLogitsLoss; , , are the weight coefficients of the three loss terms.

4. The method for recognizing mixed handwritten Chinese formulas in a two-stage, two-branch manner as described in claim 1, characterized in that: The handwritten Chinese character recognition branch in step 2 is used to recognize Chinese characters in character images, and a ResNet18 network pre-trained on the CASIA-HWDB handwritten Chinese character dataset is used.

5. The two-stage and double-branch handwritten Chinese formula mixed arrangement image recognition method of claim 4, wherein: The specific operation of the handwritten Chinese character recognition branch is: 2.1.1, in the data preparation stage, the original images of the CASIA-HWDB handwritten Chinese character dataset are transformed and expanded, including appearance disturbance, adjusting image contrast and brightness, adding noise, applying Gaussian blur and image sharpening filter, geometric transformation, simulating imaging differences and writing style changes in real scenes; 2.1.2, ResNet18 is used as the backbone network, and the pre-trained weights on the CASIA-HWDB handwritten Chinese character dataset are used to focus on the stroke and structure characteristics of handwritten Chinese characters; 2.1.3, channel attention modules and spatial attention modules are integrated in the framework of ResNet18 in sequence, the channel attention module adaptively recalibrates the weight of each feature channel, and highlights important features; The spatial attention module learns the spatial importance distribution of the feature map and focuses on the key stroke area of the character.

6. The method for recognizing mixed handwritten Chinese formulas in a two-stage, two-branch manner as described in claim 1, characterized in that: The handwritten mathematical formula recognition branch in step 2 is used to recognize mathematical formulas in character images and convert them into structured sequence format, and an encoder-decoder architecture based on attention mechanism is used, and symbol counting is introduced as an auxiliary task.

7. The two-stage and double-branch handwritten Chinese formula mixed arrangement image recognition method of claim 6, wherein: The specific operation of the handwritten mathematical formula recognition branch is: 2.2.1, DenseNet pre-trained on ImageNet is used as the encoder backbone network. DenseNet uses a dense connection mechanism to alleviate the gradient vanishing problem in deep networks and extract two-dimensional feature maps containing rich multi-scale context information, which are used as the common input of the subsequent decoder and multi-scale counting module; 2.2.2, a weakly supervised multi-scale counting module is introduced. This module only uses sequence labeling to automatically learn and predict the occurrence frequency of each type of symbol in the image, specifically: 1) use parallel convolution branches to process the two-dimensional feature maps output by the encoder; 2) introduce channel attention mechanism SENet after each parallel convolution branch to adaptively recalibrate the feature channel response; 3) by convolution reduces the number of feature channels to the number of symbol classes and through Sigmoid to generate pseudo density maps with the same size as the input two-dimensional feature maps , reflecting the probability of the occurrence of the class symbol in space, and by performing global summation on each pseudo density map , the predicted count of the symbol is obtained; 4) indirectly supervise by comparing the predicted count with the real sequence to count the symbols without position point labeling; 2.2.3, this branch uses a Transformer structure decoder to gradually focus on different regions of the feature map through attention mechanism and generates LaTeX sequence in an autoregressive manner.

8. The two-stage and double-branch handwritten Chinese formula mixed arrangement image recognition method of claim 1, wherein: The step 3 is specifically operated as: 3.1, according to the center of the bounding box coordinates of all characters, the coordinate points are sorted in the reading order from left to right and from top to bottom; 3.2, calculate the intersection-over-union of adjacent position coordinates, and filter the characters with overlapping area greater than a threshold, wherein the multiple continuous characters identified as mathematical formulas are treated as a whole unit to participate in sorting and filtering; 3.3, splice the sorted and filtered Chinese recognition result and formula recognition result to form a continuous preliminary mixed layout text that conforms to human reading habits.

9. The two-stage and double-branch handwritten Chinese formula mixed arrangement image recognition method of claim 1, wherein: The step 4 selects the general-purpose Qiyi 2-VL-2B-Instruct as the base model and fine-tunes the model using the LoRA method. LoRA adds a low-rank adapter in the linear bypass of the original Transformer layer, only trains these newly added adapters with minimal parameters, and does not update all model weights for efficient and stable model adaptation.

10. The method for recognizing mixed handwritten Chinese formulas in a two-stage, two-branch manner as described in claim 1, characterized in that: The step 4 is specifically operated as: 4.1, construct a fine-tuning dataset for the specific task of handwritten Chinese formula mixed layout image recognition. Each sample in the fine-tuning dataset is a triple, containing the original image, the preliminary mixed layout text, and the target text; 4.2, in the inference stage, input the original image to be verified and the preliminary mixed layout text generated in step 3 into the fine-tuned visual-language large model. Utilize the cross-modal alignment capability of the large model to deeply understand the semantic consistency between image content and text description, and finally identify the abnormalities in the preliminary text, including symbol duplication, key character omission, context character misrecognition, and formula structure error. Through correction and optimization, the accurate recognition text is obtained.

Images