Optical character recognition method and system for mixing handwritten form and printed form

Abstract

The invention discloses an optical character recognition method and system for mixing a handwritten form and a printed form, relates to the technical field of computer vision and document analysis, and adopts a handwritten form OCR (Optical Character Recognition) model to carry out handwritten form area detection on an input document image, and the input document image is divided into a printed body internal handwritten form, an independent handwritten paragraph and a table handwritten form. Through a two-channel model architecture and a fine-grained classification strategy, the problems of low detection precision and poor model generalization are solved, and efficient and lightweight mixed text analysis is realized. The improved YOLOv12-L model is based on spatial layout and writing feature classification, can cope with diversified document scenes, and reduces recognition failures caused by scene differences. Through classification optimization, the core problems of false detection, semantic fracture of complex typesetting, table interference and the like in a mixed scene can be solved in a targeted manner, and scene self-adaption of end-to-end handwriting recognition is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and document analysis, and particularly relates to an optical character recognition method and system for mixed handwriting and printed matter. BACKGROUND

[0002] The OCR technology realizes image-to-text conversion through computer vision, but in the mixed handwriting and printed matter scene, the following core challenges are faced:

[0003] Pattern mixing interference: printed matter and handwriting are superimposed in the same area (such as the contract signature column), and traditional models have difficulty in distinguishing features.

[0004] Writing style difference: handwriting has diversity such as continuous writing and cursive writing, and printed matter needs to deal with fixed formats and special symbols (such as formulas and tables).

[0005] Low resource adaptability: the labeled data in specific fields (such as law and medicine) is scarce, and dynamic updating needs to be handled.

[0006] The prior art has the following defects:

[0007] Insufficient detection accuracy: traditional methods (such as FasterR-CNN) have high missing detection rate in mixed areas, which leads to accumulation of subsequent recognition errors.

[0008] Poor model generalization: a single model is difficult to adapt to the morphological differences between printed matter and handwriting, for example, the handwritten digit "7" is easily confused with the printed digit "1".

[0009] High computational complexity: large visual language models (LVLM) have large parameter quantities, which are difficult to deploy to embedded devices.

[0010] Insufficient handwriting recognition accuracy: traditional text recognition methods have low handwriting recognition accuracy.

[0011] Therefore, it is urgent to design an optical character recognition method and system for mixed handwriting and printed matter to solve the optical character recognition needs in the mixed handwriting and printed matter scene. SUMMARY

[0012] To solve the above technical problems, the present application provides an optical character recognition method and system for mixed handwriting and printed matter. The following technical solutions are adopted:

[0013] An optical character recognition method for mixed handwriting and printed matter, comprising the following steps:

[0014] Step 1, using a handwriting OCR model to detect handwriting regions in the input document image, and dividing into printed matter, inline handwriting, independent handwriting paragraphs, and table handwriting;

[0015] Step 2: Mask the area detected by the handwritten OCR model with the background color of the image, and then transmit it to the printed OCR model;

[0016] Step 3: The printed text OCR model outputs the printed text recognition results;

[0017] Step 4: Combine the printed text recognition results with the handwritten text recognition results, and concatenate them according to coordinate positions to generate the final output text.

[0018] By adopting the above technical solutions, through a dual-channel model architecture and a fine-grained classification strategy, the problems of low detection accuracy and poor model generalization are solved, achieving efficient and lightweight mixed text parsing.

[0019] Optionally, the handwritten OCR model is an improved YOLOv12-L model, which classifies handwritten text in input document images based on the differences in spatial layout of handwritten content and scene, writing features, and model optimization objectives.

[0020] Optionally, inline_hw in printed text refers to the overlap between handwritten and printed text;

[0021] Isolated_hw refers to a long sequence of handwritten text that is freely formatted.

[0022] The table_hw handwriting style refers to small-sized handwritten characters constrained by a table.

[0023] By adopting the above technical solution, handwritten and printed characters are processed by independent OCR models, avoiding the need for a single model to adapt to two very different character types at the same time (such as the randomness of handwritten characters and the standardization of printed characters), thereby improving their respective recognition accuracy.

[0024] Eliminating cross-interference: By masking the handwritten area with a background color before inputting it into the printed model, interference from handwritten text on printed text recognition can be avoided (e.g., overlapping inline_hw may cause printed characters to break). Similarly, the handwritten text model focuses on three subdivided regions to reduce interference from the printed text background.

[0025] Fine-grained classification optimization: Handwritten characters are divided into three categories: inline_hw, isolated_hw, and table_hw. Model parameters are optimized for different scenarios (such as overlapping layouts, long sequences of free writing, and small characters under table constraints) to improve the localization accuracy and classification accuracy of handwritten character detection (such as avoiding misclassifying handwritten characters in tables as independent paragraphs).

[0026] The improved YOLOv12-L model is based on spatial layout and handwriting feature classification, which can handle diverse document scenarios (such as handwritten annotations in academic papers, numbers filled in tables, and freely handwritten notes), reducing recognition failures caused by scenario differences.

[0027] We have optimized the special attributes of three types of handwritten characters to solve the problem of poor recognition performance of traditional models for non-standard layout text.

[0028] Because inline_hw has strong overlap with printed text and irregular handwriting, the model needs to prioritize learning the detection of the covered area and the ability to resist interference.

[0029] Due to the free-form layout and ligatures in isolated_hw, the robustness of long sequence context modeling and character segmentation needs to be strengthened.

[0030] The character size in table_hw is limited by table constraints, so the positioning accuracy of small characters needs to be improved.

[0031] Through classification optimization, core issues such as false detections in mixed scenarios, semantic breaks in complex layouts, and table interference can be addressed in a targeted manner, enabling end-to-end handwriting recognition to adapt to different scenarios.

[0032] In step 2, the detected handwritten area is replaced with the background color using the handwritten mask mechanism to generate a masked image, which is then input into the printed text OCR model.

[0033] Optionally, the handwritten mask mechanism eliminates printed text interference by filling in the background color, and its mathematical expression is as follows:

[0034]

[0035] Among them I bg I(x, y) represents the average background color, and I(x, y) represents the original pixel value.

[0036] By adopting the above technical solutions, the handwritten model based on YOLOv12-L has efficient detection characteristics. Combined with mask operations to reduce invalid area processing, it further improves inference speed and meets the requirements of real-time or near-real-time parsing.

[0037] Optionally, in step 3, the printed text OCR model calls the detection and recognition module of PP-OCRv4 to output the printed text recognition result.

[0038] By adopting the above technical solutions, PP-OCRv4 is an end-to-end OCR system, comprising three core modules: text detection, orientation classification, and text recognition. Its modular design allows for independent use of each component in mixed scenarios.

[0039] Detection model: The improved DB algorithm (based on PPLCNetV3 backbone network) is adopted. The detection accuracy is optimized by PFHead multi-branch fusion structure and dynamic ShrinkRatio strategy. It is particularly good at handling complex layouts and small text sizes.

[0040] Orientation classifier: used to correct the orientation of tilted text boxes (0° / 180° / 90°) to ensure that the input to the recognition model is horizontal text.

[0041] Recognition Model: Based on the SVTR_LCNetV3 architecture, combined with the Lite-Neck lightweight neck network and the GTC-NRTR attention guidance branch, it supports multi-scale training and data mining (DF) strategies.

[0042] The printed text OCR model only calls the detection and recognition modules of PP-OCRv4, skipping the orientation classifier (because printed text has strong regularity and does not require frequent rotation). This choice retains the high-precision detection capability of PP-OCRv4 while simplifying the process and reducing computational overhead.

[0043] Optionally, the improved YOLOv12 introduces a region attention mechanism, which reduces the secondary complexity of self-attention by dividing the space into regions, while preserving a large receptive field. The complexity of the region attention mechanism is θ. AreaAttn The formula is:

[0044]

[0045] Where n is the feature map size, k is the number of region divisions, and d is the feature dimension. k·d represents the local attention calculation of pixels within the region. 2 This represents the computation of global attention across regions. When k < n, the complexity of the regional attention mechanism is θ. AreaAttn From the traditional θ(n) 2 d) dropped to

[0046] The improved YOLOv12-L model employs the R-ELAN module, which optimizes gradient flow and enhances multi-scale feature representation through residual shortcuts and hierarchical feature aggregation strategies. The feature fusion process of the R-ELAN module is modeled as follows:

[0047] F out =(W1·F high +W2·F mid +W3·F low );

[0048] Where F high For high-level semantic features, F mid Mid-layer detail features, F lowW1 represents the low-level edge features; W2 and W3 are the learnable weights of the high-level semantic features, mid-level detail features, and low-level edge features, respectively.

[0049] By adopting the above technical solutions, the region attention mechanism significantly reduces computational overhead, and is especially suitable for detecting small-sized characters in handwritten text (such as table_hw), thereby improving inference speed.

[0050] The R-ELAN module enhances robust perception of handwritten cursive features (such as long sequence context of isolated_hw) and characters under table constraints (such as dense typesetting of table_hw).

[0051] Optionally, the improved YOLOv12-L model employs Flash Attention, which reduces GPU memory usage and accelerates attention computation through block-based computation and memory compression techniques. The memory access efficiency of Flash Attention is quantified as follows:

[0052]

[0053] Where T Flash This is the total computation time after optimization using Flash Attention; α<1 represents the time compression factor, T baseline The basic computation time β for the original attention mechanism is a memory optimization factor, and M is the base time. baseline M represents the memory usage of the original attention mechanism. flash This represents the video memory usage after optimization using Flash Attention.

[0054] By adopting the above technical solutions, Flash Attention optimization enables the model to maintain real-time inference capabilities when processing high-resolution mixed text (such as overlapping areas of printed and handwritten text inline_hw).

[0055] Optionally, for the differentiated detection requirements of three categories—integrated handwritten text, independent handwritten paragraphs, and table-based handwritten text—the improved YOLOv12-L model adopts a multi-task loss function. The expression for the multi-task loss function is as follows:

[0056] L total =λ1L det +λ2L cls +λ3L reg ;

[0057] Where L det To mitigate region detection loss, the focus is on locating small-sized characters in the handwritten table text (table_hw).

[0058] L clsTo classify the loss, three types of handwriting were distinguished;

[0059] L reg To improve the regression loss, the segmentation boundary of the contiguous stroke interval is optimized for the isolated handwritten paragraphs (isolated_hw).

[0060] λ1, λ2, and λ3 are task weights, which are dynamically adjusted to balance the optimization objectives for different scenarios.

[0061] By adopting the above technical solution, the region detection loss is more sensitive to small-sized characters in `table_hw`. By minimizing the overlap error between the predicted bounding box and the ground truth bounding box, it effectively solves the problems of missed detection and location offset of handwritten characters in tables. It avoids overshadowing small targets: traditional single-task loss may ignore small targets due to the high weight of large-sized text. The multi-task design treats the location of `table_hw` as an independent optimization objective, ensuring its detection priority.

[0062] Enhanced handwriting type classification capabilities and fine-grained scene differentiation: Classification loss (such as cross-entropy) forces the model to learn the feature differences of inline_hw (overlapping layout), isolated_hw (free paragraph), and table_hw (table constraints), reducing type misclassification. Semantic information enhancement: Through classification supervision, the model can capture writing features in different scenarios, improving its ability to understand complex mixed scenarios.

[0063] Optimize the segmentation of cursive handwritten text (isolated_hw scenario) with precise boundary regression: The regression loss (such as SmoothL1) is designed for the cursive characteristics of isolated_hw. By optimizing the segmentation boundaries between characters or words, it reduces the misjudgment of character adhesion caused by cursive writing and improves the recognition integrity of long sequence handwritten text.

[0064] Precise segmentation of the cursive writing area can reduce contextual interference (such as the similar cursive forms of "4" and "9"), making it especially suitable for freehand writing of notes, drafts, and other scenarios.

[0065] An optical character recognition system that combines handwritten and printed characters is provided to implement an optical character recognition method that combines handwritten and printed characters. The system includes a memory, a processor, and a display. The memory stores a document image to be processed and a recognition program designed using the optical character recognition method that combines handwritten and printed characters. The processor is communicatively connected to the memory, inputs the document image to be processed into the recognition program, and outputs the recognition result. The display is communicatively connected to the processor and is used to display the recognition result.

[0066] In summary, the present invention has at least one of the following beneficial technical effects:

[0067] This invention provides a method and system for optical character recognition that combines handwritten and printed text. Through a dual-channel model architecture and fine-grained classification strategy, it addresses the problems of low detection accuracy and poor model generalization, achieving efficient and lightweight mixed text parsing. The improved YOLOv12-L model, based on spatial layout and handwriting feature classification, can handle diverse document scenarios and reduce recognition failures caused by scene differences. Through classification optimization, it specifically addresses core issues such as false detections in mixed scenarios, semantic breaks in complex layouts, and table interference, achieving end-to-end scene adaptation for handwriting recognition. Attached Figure Description

[0068] Figure 1 This is a flowchart illustrating an optical character recognition method that combines handwritten and printed text according to the present invention.

[0069] Figure 2 This is a schematic diagram of evidence materials including printed and handwritten text in a specific embodiment of the present invention;

[0070] Figure 3 This is a schematic diagram of an embodiment of the present invention where the handwritten area is masked, and only the printed area is preserved.

[0071] Figure 4 This is a schematic diagram illustrating the output of plain text data in a specific embodiment of the present invention. Detailed Implementation

[0072] The present invention will be further described in detail below with reference to the accompanying drawings.

[0073] This invention discloses an optical character recognition method and system that combines handwritten and printed characters.

[0074] Reference Figures 1-4 Example 1: An optical character recognition method that combines handwritten and printed characters, comprising the following steps:

[0075] Step 1: Use the handwritten OCR model to detect handwritten regions in the input document image and classify them into inline handwritten text in printed text, independent handwritten paragraphs, and table handwritten text.

[0076] Step 2: Mask the area detected by the handwritten OCR model with the background color of the image, and then transmit it to the printed OCR model;

[0077] Step 3: The printed text OCR model outputs the printed text recognition results;

[0078] Step 4: Combine the printed text recognition results with the handwritten text recognition results, and concatenate them according to coordinate positions to generate the final output text.

[0079] By employing a dual-channel model architecture and a fine-grained classification strategy, the problems of low detection accuracy and poor model generalization are addressed, achieving efficient and lightweight mixed text parsing.

[0080] Example 2: The handwritten OCR model is an improved YOLOv12-L model, which classifies handwritten text in input document images based on the differences in spatial layout of handwritten content and scene, writing features, and model optimization objectives.

[0081] Example 3, inline_hw in printed text refers to the overlap between handwritten text and printed text;

[0082] Isolated_hw refers to a long sequence of handwritten text that is freely formatted.

[0083] The table_hw handwriting style refers to small-sized handwritten characters constrained by a table.

[0084] Handwritten and printed characters are processed using independent OCR models to avoid a single model simultaneously adapting to two very different character types (such as the randomness of handwritten characters and the standardization of printed characters), thereby improving their respective recognition accuracy.

[0085] Eliminating cross-interference: By masking the handwritten area with a background color before inputting it into the printed model, interference from handwritten text on printed text recognition can be avoided (e.g., overlapping inline_hw may cause printed characters to break). Similarly, the handwritten text model focuses on three subdivided regions to reduce interference from the printed text background.

[0086] Fine-grained classification optimization: Handwritten characters are divided into three categories: inline_hw, isolated_hw, and table_hw. Model parameters are optimized for different scenarios (such as overlapping layouts, long sequences of free writing, and small characters under table constraints) to improve the localization accuracy and classification accuracy of handwritten character detection (such as avoiding misclassifying handwritten characters in tables as independent paragraphs).

[0087] The improved YOLOv12-L model is based on spatial layout and handwriting feature classification, which can handle diverse document scenarios (such as handwritten annotations in academic papers, numbers filled in tables, and freely handwritten notes), reducing recognition failures caused by scenario differences.

[0088] We have optimized the special attributes of three types of handwritten characters to solve the problem of poor recognition performance of traditional models for non-standard layout text.

[0089] Because inline_hw has strong overlap with printed text and irregular handwriting, the model needs to prioritize learning the detection of the covered area and the ability to resist interference.

[0090] Due to the free-form layout and ligatures in isolated_hw, the robustness of long sequence context modeling and character segmentation needs to be strengthened.

[0091] The character size in table_hw is limited by table constraints, so the positioning accuracy of small characters needs to be improved.

[0092] Through classification optimization, core issues such as false detections in mixed scenarios, semantic breaks in complex layouts, and table interference can be addressed in a targeted manner, enabling end-to-end handwriting recognition to adapt to different scenarios.

[0093] In Example 4, in step 2, the detected handwritten area is replaced with the background color using the handwritten mask mechanism to generate a masked image, which is then input into the printed OCR model.

[0094] Example 5: The handwritten mask mechanism eliminates printed text interference by filling in background color. Its mathematical expression is as follows:

[0095]

[0096] Among them I bg I(x, y) represents the average background color, and I(x, y) represents the original pixel value.

[0097] The handwritten digit model based on YOLOv12-L has efficient detection capabilities. Combined with mask operations to reduce invalid regions, it further improves inference speed and meets the requirements for real-time or near-real-time parsing.

[0098] In Example 6, in step 3, the printed text OCR model calls the detection and recognition module of PP-OCRv4 to output the printed text recognition result.

[0099] PP-OCRv4 is an end-to-end OCR system comprising three core modules: text detection, orientation classification, and text recognition. Its modular design allows for independent use of each component in mixed-scene environments.

[0100] Detection model: The improved DB algorithm (based on PPLCNetV3 backbone network) is adopted. The detection accuracy is optimized by PFHead multi-branch fusion structure and dynamic ShrinkRatio strategy. It is particularly good at handling complex layouts and small text sizes.

[0101] Orientation classifier: used to correct the orientation of tilted text boxes (0° / 180° / 90°) to ensure that the input to the recognition model is horizontal text.

[0102] Recognition Model: Based on the SVTR_LCNetV3 architecture, combined with the Lite-Neck lightweight neck network and the GTC-NRTR attention guidance branch, it supports multi-scale training and data mining (DF) strategies.

[0103] The printed text OCR model only calls the detection and recognition modules of PP-OCRv4, skipping the orientation classifier (because printed text has strong regularity and does not require frequent rotation). This choice retains the high-precision detection capability of PP-OCRv4 while simplifying the process and reducing computational overhead.

[0104] Example 7: The improved YOLOv12 introduces a region attention mechanism, which reduces the secondary complexity of self-attention by dividing spatial regions, while preserving a large receptive field. The complexity θ of the region attention mechanism is reduced. AreaAttn The formula is:

[0105]

[0106] Where n is the feature map size, k is the number of region divisions, and d is the feature dimension. k·d represents the local attention calculation of pixels within the region. 2 This represents the computation of global attention across regions. When k < n, the complexity of the regional attention mechanism is θ. AreaAttn From the traditional θ(n) 2 d) dropped to

[0107] The improved YOLOv12-L model employs the R-ELAN module, which optimizes gradient flow and enhances multi-scale feature representation through residual shortcuts and hierarchical feature aggregation strategies. The feature fusion process of the R-ELAN module is modeled as follows:

[0108] F out =(W1·F high +W2·F mid +W3·F low );

[0109] Where F high For high-level semantic features, F mid Mid-layer detail features, F low W1 represents the low-level edge features; W2 and W3 are the learnable weights of the high-level semantic features, mid-level detail features, and low-level edge features, respectively.

[0110] Region attention mechanisms significantly reduce computational overhead, and are especially suitable for detecting small characters in handwritten text (such as table_hw), thereby improving inference speed.

[0111] The R-ELAN module enhances robust perception of handwritten cursive features (such as long sequence context of isolated_hw) and characters under table constraints (such as dense typesetting of table_hw).

[0112] Example 8: The improved YOLOv12-L model uses Flash Attention to reduce GPU memory usage and accelerate attention operations through block computation and memory compression techniques. The memory access efficiency of Flash Attention is quantified as follows:

[0113]

[0114] Where T Flash This is the total computation time after optimization using Flash Attention; α<1 represents the time compression factor, T baseline The basic computation time β for the original attention mechanism is a memory optimization factor, and M is the base time. baseline M represents the memory usage of the original attention mechanism. flash This represents the memory usage after optimization using FlashAttention.

[0115] By adopting the above technical solutions, FlashAttention optimization enables the model to maintain real-time inference capabilities when processing high-resolution mixed text (such as overlapping areas of printed and handwritten text inline_hw).

[0116] Example 9: Addressing the differentiated detection requirements for three categories of handwritten text—integrated handwritten text, independent handwritten paragraphs, and table-based handwritten text—the improved YOLOv12-L model employs a multi-task loss function. The expression for the multi-task loss function is as follows:

[0117] L total =λ1L det +λ2L cls +λ3L reg ;

[0118] Where L det To mitigate region detection loss, the focus is on locating small-sized characters in the handwritten table text (table_hw).

[0119] L cls To classify the loss, three types of handwriting were distinguished;

[0120] L reg To improve the regression loss, the segmentation boundary of the contiguous stroke interval is optimized for the isolated handwritten paragraphs (isolated_hw).

[0121] λ1, λ2, and λ3 are task weights, which are dynamically adjusted to balance the optimization objectives for different scenarios.

[0122] Region detection loss is more sensitive to small characters in table_hw. By minimizing the overlap error between the predicted bounding box and the ground truth bounding box, it effectively solves the problems of missed detection and location offset of handwritten characters in tables. It avoids overshadowing small targets: traditional single-task loss may ignore small targets due to the high weight of large text. The multi-task design treats the location of table_hw as an independent optimization objective, ensuring its detection priority.

[0123] Enhanced handwriting type classification capabilities and fine-grained scene differentiation: Classification loss (such as cross-entropy) forces the model to learn the feature differences of inline_hw (overlapping layout), isolated_hw (free paragraph), and table_hw (table constraints), reducing type misclassification. Semantic information enhancement: Through classification supervision, the model can capture writing features in different scenarios, improving its ability to understand complex mixed scenarios.

[0124] Optimize the segmentation of cursive handwritten text (isolated_hw scenario) with precise boundary regression: The regression loss (such as SmoothL1) is designed for the cursive characteristics of isolated_hw. By optimizing the segmentation boundaries between characters or words, it reduces the misjudgment of character adhesion caused by cursive writing and improves the recognition integrity of long sequence handwritten text.

[0125] Precise segmentation of the cursive writing area can reduce contextual interference (such as the similar cursive forms of "4" and "9"), making it especially suitable for freehand writing of notes, drafts, and other scenarios.

[0126] Example 10: A hybrid handwritten and printed optical character recognition system for implementing a hybrid handwritten and printed optical character recognition method. The system includes a memory, a processor, and a display. The memory stores a document image to be processed and a recognition program designed using a hybrid handwritten and printed optical character recognition method. The processor is communicatively connected to the memory, inputs the document image to be processed into the recognition program, and outputs the recognition result. The display is communicatively connected to the processor and is used to display the recognition result.

[0127] The following specific embodiments illustrate the implementation principle of the present invention:

[0128] 1. Handwritten digit detection data preparation and model training:

[0129] Dataset Construction:

[0130] Collect over 1000 images of documents containing a mix of printed and handwritten text, labeled as printed, inline_hw (inline handwritten text within printed text), isolated_hw (isolated handwritten paragraphs), and table_hw (handwritten text within tables).

[0131] YOLOv12-L Training:

[0132] Training parameters: Initial learning rate 0.001, large input size (960×960), full-process Mosaic enhancement and cosine annealing learning rate scheduling, high localization weight (box=7.5) and DFL loss to optimize detection accuracy, and early stopping mechanism (patience=70) to prevent overfitting, giving full play to the advantages of pre-trained model.

[0133] Data Augmentation: Data augmentation employs full-process Mosaic stitching (with the last 10 rounds off) combined with Mixup (10% probability), RandAugment automatic enhancement, and HSV color perturbation, along with 50% horizontal flipping and ±10% translation scaling, to comprehensively improve multi-scale target detection capabilities and robustness in complex scenes.

[0134] Training results:

[0135] Performance metrics: mAP@50 (mean accuracy) reaches 0.90332.

[0136] Handwritten Script Recognition Data Preparation and Model Training

[0137] Dataset Construction:

[0138] Based on the handwriting detection model described above, more than 40,000 handwriting region images were detected and labeled with the corresponding handwriting content.

[0139] Training with ch_PP-OCRv4_rec_server_train:

[0140] Based on the SVTR_HGNet multi-task architecture, this paper adopts the PPHGNet_small backbone network and a hybrid loss of CTCLoss + NTRLoss, and is optimized for complex handwritten character recognition tasks. The core strategies include:

[0141] 1. Training schedule: 200 epochs of training, using Adam optimizer and cosine annealing learning rate (initial 0.0001), with 5 epochs of warm-up and L2 regularization (coefficient 3e-5) to suppress overfitting;

[0142] 2. Multi-scale adaptation: The input scale is dynamically adjusted by MultiScaleSampler (width 320 combined with height 32 / 48 / 64), and RecConAug (50% probability context enhancement) and RecAug geometric transformation are combined to improve the robustness of the model to deformed text;

[0143] 3. Efficient training: The validation set uses norm_edit_dis (normalized edit distance) as the core metric to achieve a balanced optimization of accuracy and speed.

[0144] The training results are shown in Table 1:

[0145] Table 1

[0146] Evaluation Index Before Training After Training Promotion Range Relative Promotion Rate Identification Accuracy 27％ 57％ +30％ 111.1％ Normalized Edit Distance 54％ 89％ +35％ 64.8％

[0147] Arbitration material analysis case based on the model trained above:

[0148] Input: Evidence materials including printed and handwritten text. For example... Figure 2 As shown;

[0149] Processing flow:

[0150] YOLOv12-L detected handwritten areas such as "signature" and "evidence name".

[0151] The handwritten region is processed by calling the handwritten model ch_PP-OCRv4_rec_server_train to obtain the text data of the handwritten region.

[0152] Masking the handwritten area, only preserving the printed area, the image is fed into the printed text detection and recognition model to obtain the text data for the printed area. For example... Figure 3 As shown;

[0153] Output: The text data from the two regions is concatenated according to coordinate order to obtain plain text data, which is then output, such as... Figure 4 As shown.

[0154] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for optical character recognition that combines handwritten and printed characters, characterized in that, Includes the following steps: Step 1: Use the handwritten OCR model to detect handwritten regions in the input document image and classify them into inline handwritten text in printed text, independent handwritten paragraphs, and table handwritten text. Step 2: Mask the area detected by the handwritten OCR model with the background color of the image, and then transmit it to the printed OCR model; Step 3: The printed text OCR model outputs the printed text recognition results; Step 4: Combine the printed text recognition results with the handwritten text recognition results, and concatenate them according to coordinate positions to generate the final output text.

2. The optical character recognition method combining handwritten and printed text according to claim 1, characterized in that, The handwritten OCR model is an improved YOLOv12-L model that classifies handwritten text in input document images based on the differences in spatial layout of handwritten content and scene, writing features, and model optimization objectives.

3. The optical character recognition method combining handwritten and printed text according to claim 1, characterized in that, Inline_hw in printed text refers to the overlap between handwritten and printed text. Isolated_hw refers to a long sequence of handwritten text that is freely formatted. The table_hw handwriting style refers to small-sized handwritten characters constrained by a table.

4. The optical character recognition method combining handwritten and printed text according to claim 1, characterized in that, In step 2, the detected handwritten area is replaced with the background color using the handwritten mask mechanism to generate a masked image, which is then input into the printed text OCR model.

5. The optical character recognition method combining handwritten and printed text according to claim 4, characterized in that, The handwritten text mask mechanism eliminates printed text interference by filling in background color; its mathematical expression is as follows: Among them I bg I(x, y) represents the average background color, and I(x, y) represents the original pixel value.

6. The optical character recognition method combining handwritten and printed text according to claim 4, characterized in that, In step 3, the printed text OCR model calls the detection and recognition module of PP-OCRv4 to output the printed text recognition results.

7. The optical character recognition method combining handwritten and printed characters according to claim 2, characterized in that, The improved YOLOv12 introduces a region attention mechanism, which reduces the secondary complexity of self-attention by dividing the space into regions, while preserving a large receptive field. This reduces the complexity of the region attention mechanism. The formula is: Where n is the feature map size, k is the number of region divisions, and d is the feature dimension. k·d represents the local attention calculation of pixels within the region. 2 This represents the computation of global attention across regions. When k < n, the complexity of the regional attention mechanism is... From traditional Down to 8. The optical character recognition method combining handwritten and printed text according to claim 2, characterized in that, The improved YOLOv12-L model employs the R-ELAN module, which optimizes gradient flow and enhances multi-scale feature representation through residual shortcuts and hierarchical feature aggregation strategies. The feature fusion process of the R-ELAN module is modeled as follows: F out =(W1·F high +W2·F mid +W3·F low ); Where F high For high-level semantic features, F mid Mid-layer detail features, F low W1 represents the low-level edge features; W2 and W3 are the learnable weights of the high-level semantic features, mid-level detail features, and low-level edge features, respectively. The improved YOLOv12-L model uses Flash Attention to reduce GPU memory usage and accelerate attention operations through block-based computation and memory compression techniques. The memory access efficiency of Flash Attention is quantified as follows: Where T Flash This is the total computation time after optimization using Flash Attention; α<1 represents the time compression factor, T baseline The basic computation time β for the original attention mechanism is a memory optimization factor, and M is the base time. baseline M represents the memory usage of the original attention mechanism. flash This represents the memory usage after optimization using FlashAttention.

9. The optical character recognition method combining handwritten and printed text according to claim 2, characterized in that, To address the differentiated detection requirements for three categories of handwritten text—integrated handwritten text, independent handwritten paragraphs, and table-based handwritten text—the improved YOLOv12-L model employs a multi-task loss function. The expression for the multi-task loss function is as follows: L total =λ1L det +λ2L cls +λ3L reg ; Where L det To mitigate region detection loss, the focus is on locating small-sized characters in the handwritten table text (table_hw). L cls To classify the loss, three types of handwriting were distinguished; L reg To improve the regression loss, the segmentation boundary of the contiguous stroke interval is optimized for the isolated handwritten paragraphs (isolated_hw). λ1, λ2, and λ3 are task weights, which are dynamically adjusted to balance the optimization objectives for different scenarios.

10. An optical character recognition system that combines handwritten and printed characters, characterized in that: The system for implementing the optical character recognition method for hybrid handwriting and printed text as described in any one of claims 1-9 includes a memory, a processor, and a display. The memory stores a document image to be processed and a recognition program designed using the optical character recognition method for hybrid handwriting and printed text as described in any one of claims 1-9. The processor is communicatively connected to the memory, inputs the document image to be processed into the recognition program, and outputs the recognition result. The display is communicatively connected to the processor and is used to display the recognition result.

Images