An optical character recognition method and system that combines handwritten and printed characters

Abstract

This invention discloses an optical character recognition method and system that combines handwritten and printed text, relating to the fields of computer vision and document analysis technology. It employs a handwritten OCR model to detect handwritten regions in input document images, categorizing them into inline handwritten text within printed text, independent handwritten paragraphs, and table-based handwritten text. This invention addresses the issues of low detection accuracy and poor model generalization through a dual-channel model architecture and fine-grained classification strategy, achieving efficient and lightweight mixed text parsing. The improved YOLOv12-L model, based on spatial layout and handwriting feature classification, can handle diverse document scenarios and reduce recognition failures caused by scene differences. Through classification optimization, it specifically addresses core issues such as false detections in mixed scenarios, semantic breaks in complex layouts, and table interference, achieving end-to-end scene adaptation for handwriting recognition.

Description

Technical Field

[0001] This invention relates to the field of computer vision and document analysis technology, and in particular to an optical character recognition method and system that combines handwritten and printed characters. Background Technology

[0002] OCR technology uses computer vision to convert images into text, but it faces the following core challenges in scenarios involving a mix of handwritten and printed text:

[0003] Pattern mixing interference: Printed and handwritten text overlap in the same area (such as the signature bar of a contract), making it difficult for traditional models to distinguish features.

[0004] Differences in writing style: Handwriting can be diverse, including cursive and cursive scripts, while printed text requires handling fixed formats and special symbols (such as formulas and tables).

[0005] Low resource adaptability: Data annotation is scarce in specific fields (such as law and medical care), and dynamic update requirements need to be handled.

[0006] The existing technology has the following drawbacks:

[0007] Insufficient detection accuracy: Traditional methods (such as Faster R-CNN) have a high rate of missing detections in mixed regions, leading to the accumulation of subsequent identification errors.

[0008] Poor model generalization: A single model is difficult to adapt to the morphological differences between printed and handwritten characters at the same time. For example, the handwritten number "7" is easily confused with the printed number "1".

[0009] High computational complexity: Large visual language models (LVLMs) have a large number of parameters, making them difficult to deploy on embedded devices.

[0010] Insufficient accuracy in handwriting recognition: Traditional text recognition methods have low accuracy in handwriting recognition.

[0011] Therefore, there is an urgent need to design an optical character recognition method and system that combines handwritten and printed text to address the optical character recognition requirements in scenarios involving both handwritten and printed text. Summary of the Invention

[0012] To address the aforementioned technical problems, this invention provides a method and system for optical character recognition that combines handwritten and printed characters. The technical solution is as follows:

[0013] An optical character recognition method that combines handwritten and printed characters includes the following steps:

[0014] Step 1: Use the handwritten OCR model to detect handwritten regions in the input document image and classify them into inline handwritten text in printed text, independent handwritten paragraphs, and table handwritten text.

[0015] Step 2: Mask the area detected by the handwritten OCR model with the background color of the image, and then transmit it to the printed OCR model;

[0016] Step 3: The printed text OCR model outputs the printed text recognition results;

[0017] Step 4: Combine the printed text recognition results with the handwritten text recognition results, and concatenate them according to coordinate positions to generate the final output text.

[0018] By adopting the above technical solutions, through a dual-channel model architecture and a fine-grained classification strategy, the problems of low detection accuracy and poor model generalization are solved, achieving efficient and lightweight mixed text parsing.

[0019] Optionally, the handwritten OCR model is an improved YOLOv12-L model, which classifies handwritten text in input document images based on the differences in spatial layout of handwritten content and scene, writing features, and model optimization objectives.

[0020] Optionally, inline_hw in printed text refers to the overlap between handwritten and printed text;

[0021] Isolated_hw refers to a long sequence of handwritten text that is freely formatted.

[0022] The table_hw handwriting style refers to small-sized handwritten characters constrained by a table.

[0023] By adopting the above technical solution, handwritten and printed characters are processed by independent OCR models, avoiding the need for a single model to adapt to two very different character types at the same time (such as the randomness of handwritten characters and the standardization of printed characters), thereby improving their respective recognition accuracy.

[0024] Eliminating cross-interference: By masking the handwritten area with a background color before inputting it into the printed model, interference from handwritten text on printed text recognition can be avoided (e.g., overlapping inline_hw may cause printed characters to break). Similarly, the handwritten text model focuses on three subdivided regions to reduce interference from the printed text background.

[0025] Fine-grained classification optimization: Handwritten characters are divided into three categories: inline_hw, isolated_hw, and table_hw. Model parameters are optimized for different scenarios (such as overlapping layouts, long sequences of free writing, and small characters under table constraints) to improve the localization accuracy and classification accuracy of handwritten character detection (such as avoiding misclassifying handwritten characters in tables as independent paragraphs).

[0026] The improved YOLOv12-L model is based on spatial layout and handwriting feature classification, which can handle diverse document scenarios (such as handwritten annotations in academic papers, numbers filled in tables, and freely handwritten notes), reducing recognition failures caused by scenario differences.

[0027] We have optimized the special attributes of three types of handwritten characters to solve the problem of poor recognition performance of traditional models for non-standard layout text.

[0028] Because inline_hw has strong overlap with printed text and irregular handwriting, the model needs to prioritize learning the detection of the covered area and the ability to resist interference.

[0029] Due to the free-form layout and ligatures in isolated_hw, the robustness of long sequence context modeling and character segmentation needs to be strengthened.

[0030] The character size in table_hw is limited by table constraints, so the positioning accuracy of small characters needs to be improved.

[0031] Through classification optimization, core issues such as false detections in mixed scenarios, semantic breaks in complex layouts, and table interference can be addressed in a targeted manner, enabling end-to-end handwriting recognition to adapt to different scenarios.

[0032] In step 2, the detected handwritten area is replaced with the background color using the handwritten mask mechanism to generate a masked image, which is then input into the printed text OCR model.

[0033] Optionally, the handwritten mask mechanism eliminates printed text interference by filling in the background color, and its mathematical expression is as follows:

[0034]

[0035] Where I bg I(x, y) represents the average background color, and I(x, y) represents the original pixel value.

[0036] By adopting the above technical solutions, the handwritten model based on YOLOv12-L has efficient detection characteristics. Combined with mask operations to reduce invalid area processing, it further improves inference speed and meets the requirements of real-time or near-real-time parsing.

[0037] Optionally, in step 3, the printed text OCR model calls the detection and recognition module of PP-OCRv4 to output the printed text recognition result.

[0038] By adopting the above technical solutions, PP-OCRv4 is an end-to-end OCR system, comprising three core modules: text detection, orientation classification, and text recognition. Its modular design allows for independent use of each component in mixed scenarios.

[0039] Detection model: The improved DB algorithm (based on PPLCNetV3 backbone network) is adopted. The detection accuracy is optimized by PFHead multi-branch fusion structure and dynamic ShrinkRatio strategy. It is particularly good at handling complex layouts and small text sizes.

[0040] Orientation classifier: used to correct the orientation of tilted text boxes (0° / 180° / 90°) to ensure that the input to the recognition model is horizontal text.

[0041] Recognition Model: Based on the SVTR_LCNetV3 architecture, combined with the Lite-Neck lightweight neck network and the GTC-NRTR attention guidance branch, it supports multi-scale training and data mining (DF) strategies.

[0042] The printed text OCR model only calls the detection and recognition modules of PP-OCRv4, skipping the orientation classifier (because printed text has strong regularity and does not require frequent rotation). This choice retains the high-precision detection capability of PP-OCRv4 while simplifying the process and reducing computational overhead.

[0043] Optionally, the improved YOLOv12 introduces a region attention mechanism, which reduces the secondary complexity of self-attention by dividing the space into regions, while preserving a large receptive field. The complexity of the region attention mechanism is θ. AreaAttn The formula is:

[0044]

[0045] Where n is the feature map size, k is the number of region divisions, and d is the feature dimension. k·d represents the local attention calculation of pixels within the region. 2 This represents the computation of global attention across regions. When k < n, the complexity of the regional attention mechanism is θ. AreaAttn From the traditional θ(n) 2 d) dropped to

[0046] The improved YOLOv12-L model employs the R-ELAN module, which optimizes gradient flow and enhances multi-scale feature representation through residual shortcuts and hierarchical feature aggregation strategies. The feature fusion process of the R-ELAN module is modeled as follows:

[0047] F out =(W1·F high +W2·F mid +W3·F low );

[0048] Where F high For high-level semantic features, F mid Mid-layer detail features, F lowW1 represents the low-level edge features; W2 and W3 are the learnable weights of the high-level semantic features, mid-level detail features, and low-level edge features, respectively.

[0049] By adopting the above technical solutions, the region attention mechanism significantly reduces computational overhead, and is especially suitable for detecting small-sized characters in handwritten text (such as table_hw), thereby improving inference speed.

[0050] The R-ELAN module enhances robust perception of handwritten cursive features (such as long sequence context of isolated_hw) and characters under table constraints (such as dense typesetting of table_hw).

[0051] Optionally, the improved YOLOv12-L model employs Flash Attention, which reduces GPU memory usage and accelerates attention computation through block-based computation and memory compression techniques. The memory access efficiency of Flash Attention is quantified as follows:

[0052]

[0053] Where T Flash This is the total computation time after optimization using Flash Attention; α<1 represents the time compression factor, T baseline The basic computation time β for the original attention mechanism is a memory optimization factor, and M is the base time. baseline M represents the memory usage of the original attention mechanism. flash This represents the video memory usage after optimization using Flash Attention.

[0054] By adopting the above technical solutions, the Flash Attention optimization enables the model to maintain real-time inference capabilities when processing high-resolution mixed text (such as the overlapping area of ​​printed and handwritten text inline_hw).

[0055] Optionally, for the differentiated detection requirements of three categories—integrated handwritten text, independent handwritten paragraphs, and table-based handwritten text—the improved YOLOv12-L model adopts a multi-task loss function. The expression for the multi-task loss function is as follows:

[0056] L total =λ1L det +λ2L cls +λ3L reg ;

[0057] Where L det To mitigate region detection loss, the focus is on locating small-sized characters in the handwritten table text (table_hw).

[0058] L clsTo classify the loss, three types of handwriting were distinguished;

[0059] L reg To improve the regression loss, the segmentation boundary of the contiguous stroke interval is optimized for the isolated handwritten paragraphs (isolated_hw).

[0060] λ1, λ2, and λ3 are task weights, which are dynamically adjusted to balance the optimization objectives for different scenarios.

[0061] By adopting the above technical solution, the region detection loss is more sensitive to small-sized characters in `table_hw`. By minimizing the overlap error between the predicted bounding box and the ground truth bounding box, it effectively solves the problems of missed detection and location offset of handwritten characters in tables. It avoids overshadowing small targets: traditional single-task loss may ignore small targets due to the high weight of large-sized text. The multi-task design treats the location of `table_hw` as an independent optimization objective, ensuring its detection priority.

[0062] Enhanced handwriting type classification capabilities and fine-grained scene differentiation: Classification loss (such as cross-entropy) forces the model to learn the feature differences of inline_hw (overlapping layout), isolated_hw (free paragraph), and table_hw (table constraints), reducing type misclassification. Semantic information enhancement: Through classification supervision, the model can capture writing features in different scenarios, improving its ability to understand complex mixed scenarios.

[0063] Optimize the segmentation of cursive handwritten text (isolated_hw scenario) with precise boundary regression: The regression loss (such as SmoothL1) is designed for the cursive characteristics of isolated_hw. By optimizing the segmentation boundaries between characters or words, it reduces the misjudgment of character adhesion caused by cursive writing and improves the recognition integrity of long sequence handwritten text.

[0064] Precise segmentation of the cursive writing area can reduce contextual interference (such as the similar cursive forms of "4" and "9"), making it especially suitable for freehand writing of notes, drafts, and other scenarios.

[0065] An optical character recognition system that combines handwritten and printed characters is provided to implement an optical character recognition method that combines handwritten and printed characters. The system includes a memory, a processor, and a display. The memory stores a document image to be processed and a recognition program designed using the optical character recognition method that combines handwritten and printed characters. The processor is communicatively connected to the memory, inputs the document image to be processed into the recognition program, and outputs the recognition result. The display is communicatively connected to the processor and is used to display the recognition result.

[0066] In summary, the present invention has at least one of the following beneficial technical effects:

[0067] This invention provides a method and system for optical character recognition that combines handwritten and printed text. Through a dual-channel model architecture and fine-grained classification strategy, it addresses the problems of low detection accuracy and poor model generalization, achieving efficient and lightweight mixed text parsing. The improved YOLOv12-L model, based on spatial layout and handwriting feature classification, can handle diverse document scenarios and reduce recognition failures caused by scene differences. Through classification optimization, it specifically addresses core issues such as false detections in mixed scenarios, semantic breaks in complex layouts, and table interference, achieving end-to-end scene adaptation for handwriting recognition. Attached Figure Description

[0068] Figure 1 This is a flowchart illustrating an optical character recognition method that combines handwritten and printed text according to the present invention.

[0069] Figure 2 This is a schematic diagram of evidence materials including printed and handwritten text in a specific embodiment of the present invention;

[0070] Figure 3 This is a schematic diagram of an embodiment of the present invention where the handwritten area is masked, and only the printed area is preserved.

[0071] Figure 4 This is a schematic diagram illustrating the output of plain text data in a specific embodiment of the present invention. Detailed Implementation

[0072] The present invention will be further described in detail below with reference to the accompanying drawings.

[0073] This invention discloses an optical character recognition method and system that combines handwritten and printed characters.

[0074] Reference Figures 1-4 Example 1: An optical character recognition method that combines handwritten and printed characters, comprising the following steps:

[0075] Step 1: Use the handwritten OCR model to detect handwritten regions in the input document image and classify them into inline handwritten text in printed text, independent handwritten paragraphs, and table handwritten text.

[0076] Step 2: Mask the area detected by the handwritten OCR model with the background color of the image, and then transmit it to the printed OCR model;

[0077] Step 3: The printed text OCR model outputs the printed text recognition results;

[0078] Step 4: Combine the printed text recognition results with the handwritten text recognition results, and concatenate them according to coordinate positions to generate the final output text.

[0079] By employing a dual-channel model architecture and a fine-grained classification strategy, the problems of low detection accuracy and poor model generalization are addressed, achieving efficient and lightweight mixed text parsing.

[0080] Example 2: The handwritten OCR model is an improved YOLOv12-L model, which classifies handwritten text in input document images based on the differences in spatial layout of handwritten content and scene, writing features, and model optimization objectives.

[0081] Example 3, inline_hw in printed text refers to the overlap between handwritten text and printed text;

[0082] Isolated_hw refers to a long sequence of handwritten text that is freely formatted.

[0083] The table_hw handwriting style refers to small-sized handwritten characters constrained by a table.

[0084] Handwritten and printed characters are processed using independent OCR models to avoid a single model simultaneously adapting to two very different character types (such as the randomness of handwritten characters and the standardization of printed characters), thereby improving their respective recognition accuracy.

[0085] Eliminating cross-interference: By masking the handwritten area with a background color before inputting it into the printed model, interference from handwritten text on printed text recognition can be avoided (e.g., overlapping inline_hw may cause printed characters to break). Similarly, the handwritten text model focuses on three subdivided regions to reduce interference from the printed text background.

[0086] Fine-grained classification optimization: Handwritten characters are divided into three categories: inline_hw, isolated_hw, and table_hw. Model parameters are optimized for different scenarios (such as overlapping layouts, long sequences of free writing, and small characters under table constraints) to improve the localization accuracy and classification accuracy of handwritten character detection (such as avoiding misclassifying handwritten characters in tables as independent paragraphs).

[0087] The improved YOLOv12-L model is based on spatial layout and handwriting feature classification, which can handle diverse document scenarios (such as handwritten annotations in academic papers, numbers filled in tables, and freely handwritten notes), reducing recognition failures caused by scenario differences.

[0088] We have optimized the special attributes of three types of handwritten characters to solve the problem of poor recognition performance of traditional models for non-standard layout text.

[0089] Because inline_hw has strong overlap with printed text and irregular handwriting, the model needs to prioritize learning the detection of the covered area and the ability to resist interference.

[0090] Due to the free-form layout and ligatures in isolated_hw, the robustness of long sequence context modeling and character segmentation needs to be strengthened.

[0091] The character size in table_hw is limited by table constraints, so the positioning accuracy of small characters needs to be improved.

[0092] Through classification optimization, core issues such as false detections in mixed scenarios, semantic breaks in complex layouts, and table interference can be addressed in a targeted manner, enabling end-to-end handwriting recognition to adapt to different scenarios.

[0093] In Example 4, in step 2, the detected handwritten area is replaced with the background color using the handwritten mask mechanism to generate a masked image, which is then input into the printed OCR model.

[0094] Example 5: The handwritten mask mechanism eliminates printed text interference by filling in background color. Its mathematical expression is as follows:

[0095]

[0096] Where I bg I(x, y) represents the average background color, and I(x, y) represents the original pixel value.

[0097] The handwritten digit model based on YOLOv12-L has efficient detection capabilities. Combined with mask operations to reduce invalid regions, it further improves inference speed and meets the requirements for real-time or near-real-time parsing.

[0098] In Example 6, in step 3, the printed text OCR model calls the detection and recognition module of PP-OCRv4 to output the printed text recognition result.

[0099] PP-OCRv4 is an end-to-end OCR system comprising three core modules: text detection, orientation classification, and text recognition. Its modular design allows for independent use of each component in mixed-scene environments.

[0100] Detection model: The improved DB algorithm (based on PPLCNetV3 backbone network) is adopted. The detection accuracy is optimized by PFHead multi-branch fusion structure and dynamic ShrinkRatio strategy. It is particularly good at handling complex layouts and small text sizes.

[0101] Orientation classifier: used to correct the orientation of tilted text boxes (0° / 180° / 90°) to ensure that the input to the recognition model is horizontal text.

[0102] Recognition Model: Based on the SVTR_LCNetV3 architecture, combined with the Lite-Neck lightweight neck network and the GTC-NRTR attention guidance branch, it supports multi-scale training and data mining (DF) strategies.

[0103] The printed text OCR model only calls the detection and recognition modules of PP-OCRv4, skipping the orientation classifier (because printed text has strong regularity and does not require frequent rotation). This choice retains the high-precision detection capability of PP-OCRv4 while simplifying the process and reducing computational overhead.

[0104] Example 7: The improved YOLOv12 introduces a region attention mechanism, which reduces the secondary complexity of self-attention by dividing spatial regions, while preserving a large receptive field. The complexity θ of the region attention mechanism is reduced. AreaAttn The formula is:

[0105]

[0106] Where n is the feature map size, k is the number of region divisions, and d is the feature dimension. k·d represents the local attention calculation of pixels within the region. 2 This represents the computation of global attention across regions. When k < n, the complexity of the regional attention mechanism is θ. AreaAttn From the traditional θ(n) 2 d) dropped to

[0107] The improved YOLOv12-L model employs the R-ELAN module, which optimizes gradient flow and enhances multi-scale feature representation through residual shortcuts and hierarchical feature aggregation strategies. The feature fusion process of the R-ELAN module is modeled as follows:

[0108] F out =(W1·F high +W2·F mid +W3·F low );

[0109] Where F high For high-level semantic features, F mid Mid-layer detail features, F low W1 represents the low-level edge features; W2 and W3 are the learnable weights of the high-level semantic features, mid-level detail features, and low-level edge features, respectively.

[0110] Region attention mechanisms significantly reduce computational overhead, and are especially suitable for detecting small characters in handwritten text (such as table_hw), thereby improving inference speed.

[0111] The R-ELAN module enhances robust perception of handwritten cursive features (such as long sequence context of isolated_hw) and characters under table constraints (such as dense typesetting of table_hw).

[0112] Example 8: The improved YOLOv12-L model uses Flash Attention to reduce GPU memory usage and accelerate attention operations through block computation and memory compression techniques. The memory access efficiency of Flash Attention is quantified as follows:

[0113]

[0114] Where T Flash This is the total computation time after optimization using Flash Attention; α<1 represents the time compression factor, T baseline The basic computation time β for the original attention mechanism is a memory optimization factor, and M is the base time. baseline M represents the memory usage of the original attention mechanism. flash This represents the memory usage after optimization using FlashAttention.

[0115] By adopting the above technical solutions, FlashAttention optimization enables the model to maintain real-time inference capabilities when processing high-resolution mixed text (such as overlapping areas of printed and handwritten text inline_hw).

[0116] Example 9: Addressing the differentiated detection requirements for three categories of handwritten text—integrated handwritten text, independent handwritten paragraphs, and table-based handwritten text—the improved YOLOv12-L model employs a multi-task loss function. The expression for the multi-task loss function is as follows:

[0117] L total =λ1L det +λ2L cls +λ3L reg ;

[0118] Where L det To mitigate region detection loss, the focus is on locating small-sized characters in the handwritten table text (table_hw).

[0119] L cls To classify the loss, three types of handwriting were distinguished;

[0120] L reg To improve the regression loss, the segmentation boundary of the contiguous stroke interval is optimized for the isolated handwritten paragraphs (isolated_hw).

[0121] λ1, λ2, and λ3 are task weights, which are dynamically adjusted to balance the optimization objectives for different scenarios.

[0122] Region detection loss is more sensitive to small characters in table_hw. By minimizing the overlap error between the predicted bounding box and the ground truth bounding box, it effectively solves the problems of missed detection and location offset of handwritten characters in tables. It avoids overshadowing small targets: traditional single-task loss may ignore small targets due to the high weight of large text. The multi-task design treats the location of table_hw as an independent optimization objective, ensuring its detection priority.

[0123] Enhanced handwriting type classification capabilities and fine-grained scene differentiation: Classification loss (such as cross-entropy) forces the model to learn the feature differences of inline_hw (overlapping layout), isolated_hw (free paragraph), and table_hw (table constraints), reducing type misclassification. Semantic information enhancement: Through classification supervision, the model can capture writing features in different scenarios, improving its ability to understand complex mixed scenarios.

[0124] Optimize the segmentation of cursive handwritten text (isolated_hw scenario) with precise boundary regression: The regression loss (such as SmoothL1) is designed for the cursive characteristics of isolated_hw. By optimizing the segmentation boundaries between characters or words, it reduces the misjudgment of character adhesion caused by cursive writing and improves the recognition integrity of long sequence handwritten text.

[0125] Precise segmentation of the cursive writing area can reduce contextual interference (such as the similar cursive forms of "4" and "9"), making it especially suitable for freehand writing of notes, drafts, and other scenarios.

[0126] Example 10: A hybrid handwritten and printed optical character recognition system for implementing a hybrid handwritten and printed optical character recognition method. The system includes a memory, a processor, and a display. The memory stores a document image to be processed and a recognition program designed using a hybrid handwritten and printed optical character recognition method. The processor is communicatively connected to the memory, inputs the document image to be processed into the recognition program, and outputs the recognition result. The display is communicatively connected to the processor and is used to display the recognition result.

[0127] The following specific embodiments illustrate the implementation principle of the present invention:

[0128] 1. Handwritten digit detection data preparation and model training:

[0129] Dataset Construction:

[0130] Collect over 1000 images of documents containing a mix of printed and handwritten text, labeled as printed, inline_hw (inline handwritten text within printed text), isolated_hw (isolated handwritten paragraphs), and table_hw (handwritten text within tables).

[0131] YOLOv12-L Training:

[0132] Training parameters: Initial learning rate 0.001, large input size (960×960), full-process Mosaic enhancement and cosine annealing learning rate scheduling, high localization weight (box=7.5) and DFL loss to optimize detection accuracy, and early stopping mechanism (patience=70) to prevent overfitting, giving full play to the advantages of pre-trained model.

[0133] Data Augmentation: Data augmentation employs full-process Mosaic stitching (with the last 10 rounds off) combined with Mixup (10% probability), RandAugment automatic enhancement, and HSV color perturbation, along with 50% horizontal flipping and ±10% translation scaling, to comprehensively improve multi-scale target detection capabilities and robustness in complex scenes.

[0134] Training results:

[0135] Performance metrics: mAP@50 (mean accuracy) reaches 0.90332.

[0136] Handwritten Script Recognition Data Preparation and Model Training

[0137] Dataset Construction:

[0138] Based on the handwriting detection model described above, more than 40,000 handwriting region images were detected and labeled with the corresponding handwriting content.

[0139] Training with ch_PP-OCRv4_rec_server_train:

[0140] Based on the SVTR_HGNet multi-task architecture, this paper adopts the PPHGNet_small backbone network and a hybrid loss of CTCLoss + NTRLoss, and is optimized for complex handwritten character recognition tasks. The core strategies include:

[0141] 1. Training schedule: 200 epochs of training, using Adam optimizer and cosine annealing learning rate (initial 0.0001), with 5 epochs of warm-up and L2 regularization (coefficient 3e-5) to suppress overfitting;

[0142] 2. Multi-scale adaptation: The input scale is dynamically adjusted by MultiScaleSampler (width 320 combined with height 32 / 48 / 64), and RecConAug (50% probability context enhancement) and RecAug geometric transformation are combined to improve the robustness of the model to deformed text;

[0143] 3. Efficient training: The validation set uses norm_edit_dis (normalized edit distance) as the core metric to achieve a balanced optimization of accuracy and speed.

[0144] The training results are shown in Table 1:

[0145] Table 1

[0146] Evaluation indicators Before training After training Increase relative improvement rate Recognition accuracy 27％ 57％ +30％ 111.1％ Normalized edit distance 54％ 89％ +35％ 64.8％

[0147] Arbitration material analysis case based on the model trained above:

[0148] Input: Evidence materials including printed and handwritten text. For example... Figure 2 As shown;

[0149] Processing flow:

[0150] YOLOv12-L detected handwritten areas such as "signature" and "evidence name".

[0151] The handwritten region is processed by calling the handwritten model ch_PP-OCRv4_rec_server_train to obtain the text data of the handwritten region.

[0152] Masking the handwritten area, only preserving the printed area, the image is fed into the printed text detection and recognition model to obtain the text data for the printed area. For example... Figure 3 As shown;

[0153] Output: The text data from the two regions is concatenated according to coordinate order to obtain plain text data, which is then output, such as... Figure 4 As shown.

[0154] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for optical character recognition that combines handwritten and printed characters, characterized in that, Includes the following steps: Step 1: Use the handwritten OCR model to detect handwritten regions in the input document image and classify them into inline handwritten text in printed text, independent handwritten paragraphs, and table handwritten text. Step 2: Mask the area detected by the handwritten OCR model with the background color of the image, and then transmit it to the printed OCR model; Step 3: The printed text OCR model outputs the printed text recognition results; Step 4: Combine the printed text recognition results with the handwritten text recognition results, and concatenate them according to coordinate positions to generate the final output text; In step 2, the detected handwritten area is replaced with the background color through the handwritten mask mechanism to generate a masked image, which is then input into the printed OCR model. The handwritten text mask mechanism eliminates printed text interference by filling in background color; its mathematical expression is as follows: ； in The average background color. These are the original pixel values; In step 3, the printed text OCR model calls the detection and recognition module of PP-OCRv4 to output the printed text recognition result; The improved YOLOv12 introduces a region attention mechanism, which reduces the secondary complexity of self-attention by dividing the space into regions, while preserving a large receptive field. This reduces the complexity of the region attention mechanism. The formula is: ； Where n is the feature map size, k is the number of region divisions, and d is the feature dimension. This represents the local attention calculation for pixels within a region. This represents the computation of global attention across regions. When k < n, the complexity of the regional attention mechanism is... From traditional Down to ; The handwritten OCR model is an improved YOLOv12-L model that classifies handwritten text in input document images based on the differences in spatial layout of handwritten content and scene, writing features, and model optimization objectives. To address the differentiated detection requirements for three categories of handwritten text—integrated handwritten text, independent handwritten paragraphs, and table-based handwritten text—the improved YOLOv12-L model employs a multi-task loss function. The expression for the multi-task loss function is as follows: ； in To mitigate region detection loss, the focus is on locating small-sized characters in the handwritten table text (table_hw). To classify the loss, three types of handwriting were distinguished; To improve the regression loss, the segmentation boundary of the connected stroke interval is optimized for the isolated handwritten paragraphs (isolated_hw). , and These are task weights, which are dynamically adjusted to balance the optimization goals for different scenarios.

2. The optical character recognition method combining handwritten and printed text according to claim 1, characterized in that, Inline_hw in printed text refers to the overlap between handwritten and printed text. Isolated_hw refers to a long sequence of handwritten text that is freely formatted. The table_hw style refers to small-sized handwritten characters constrained by a table.

3. The optical character recognition method combining handwritten and printed text according to claim 1, characterized in that, The improved YOLOv12-L model employs the R-ELAN module, which optimizes gradient flow and enhances multi-scale feature representation through residual shortcuts and hierarchical feature aggregation strategies. The feature fusion process of the R-ELAN module is modeled as follows: ； in For high-level semantic features, Mid-layer details These are low-level edge features; , and These are the learnable weights for high-level semantic features, mid-level detail features, and low-level edge features, respectively. The improved YOLOv12-L model uses Flash Attention to reduce GPU memory usage and accelerate attention operations through block-based computation and memory compression techniques. The memory access efficiency of Flash Attention is quantified as follows: ； in This is the total computation time after optimization using Flash Attention; Indicates the time compression factor. The basic computation time for the original attention mechanism For memory optimization factor, This represents the memory usage of the original attention mechanism. This represents the video memory usage after optimization using Flash Attention.

4. An optical character recognition system that combines handwritten and printed characters, characterized in that: The system for implementing the optical character recognition method for hybrid handwriting and printed text as described in any one of claims 1-3 includes a memory, a processor, and a display. The memory stores a document image to be processed and a recognition program designed using the optical character recognition method for hybrid handwriting and printed text as described in any one of claims 1-3. The processor is communicatively connected to the memory, inputs the document image to be processed into the recognition program, and outputs the recognition result. The display is communicatively connected to the processor and is used to display the recognition result.

Images