A method and system for structured segmentation recognition of small handwritten documents

Abstract

The application discloses a kind of small language handwritten certificate of structure segmentation identification method and system, including certificate image preprocessing, field region detection cutting, the field obtained by cutting is divided into ordinary text field and strong format field with fixed internal structure;For strong format field image, locate the fixed separator therein as anchor point, according to anchor point, strong format field image is cut into multiple semantic independent subfield image;Respectively using special recognition model matched with each subfield, the corresponding subfield image is identified, and the recognition candidate and visual confidence of each subfield are output;Load business dictionary and field format rule, constrain decoding and candidate rearrangement to each subfield recognition candidate, generate structured recognition candidate;Integrate the main recognition result output by main OCR model and the structured recognition candidate, combine confidence, format legality, dictionary legality to carry out fusion decision, output final recognition result and confidence mark.

Description

Technical Field

[0001] This invention relates to the field of computer vision and OCR document recognition technology, specifically to a structured segmentation recognition method and system for handwritten documents in less commonly spoken languages. Background Technology

[0002] Existing OCR recognition for identification documents typically employs end-to-end OCR models, such as CRNN / CTC, Paddle OCR, SVTR, Tr OCR, or other Transformer OCR models, directly outputting complete strings from entire lines of text or field images. In identification document field recognition, a single master OCR model is often used to uniformly train multiple fields such as name, address, date of birth, identification number, height, blood type, and identification validity period.

[0003] This method, which improves the recognition capabilities of the OCR model to solve the recognition of various fields in document images, struggles to adapt to document recognition in less common languages ​​or even those containing handwritten text. The following example uses a Burmese ID card: For Myanmar identity card NRC numbers, the standard format can be represented as "state code / township code ( A 6-digit code. "State" refers to the state code from 1 to 14, "township" is the administrative division code, and "marker" is usually a fixed Burmese character. The serial number is a 6-digit Burmese number. Existing end-to-end OCR methods generally recognize the complete NRC number as a line of ordinary text, without making full use of structural information such as " / ", "(", ")" separators, fixed markers, valid state-township combinations, and the fixed number of digits in the serial number. This results in recognition results that are not only incorrect in words and letters, but also do not conform to the standards.

[0004] The following are some commonly used technologies for document recognition: 1. Document recognition solution based on text localization, template classification, and field recognition models: For example, existing patent CN109492643B discloses a method that first preprocesses the document image and locates the text, then classifies the text regions according to document type templates, and calls the corresponding target recognition model to output the document text information. This type of solution embodies the idea of ​​"field region cropping + field model," that is, accurate text box location is achieved first, and then the corresponding model accurately recognizes the text. The prerequisite for the application of this type of method is that the field region is stable and easy to locate, and the field character recognition is relatively accurate; otherwise, it will result in missed or incorrect recognition.

[0005] 2. OCR solutions based on page layout, key-value pairs, and structured output: For example, existing patent CN113591657B discloses a method for detecting text lines and using relative position matching information of text lines to identify attribute fields and parameter fields, thereby outputting structured recognition information. This method has high requirements for layout format, such as stable positional relationships between fields and relatively fixed field lengths and spacing. This type of method also requires stable field regions; for handwritten text, where field lengths vary greatly and field spacing is unstable, it can still lead to missed or incorrect recognition.

[0006] 3. OCR solution based on end-to-end deep learning and general enhancement: For example, existing patent CN111723807B utilizes a general machine-printed / handwritten multi-task network or layout key-value pair extraction technology to achieve unified recognition of multiple fields. While this type of solution emphasizes the general coverage of multiple fields, it cannot achieve pixel-level perception and multi-model joint constraint decoding for the fine internal structure of a single, strongly formatted, multilingual, or specially constrained document number (such as a fixed 6-digit sequence).

[0007] 4. OCR recognition scheme based on small sample image data: For example, the existing patent CN120954001A proposes a general "small sample image-cropped OCR" framework: first, the image to be recognized is acquired and cropped into images to construct the original dataset; then, multiple small sample task datasets are divided; semantic associations and sequence information between characters are extracted; the "overall instability coefficient" is calculated; based on the degree of instability, maintain, adjust, and warn signals are generated, and the recognition process is dynamically intervened. The "image-cropped data + small sample task" approach will be suitable for the recognition of document field levels. "Sequential coherence" is meaningful for strongly ordered fields, but it is still a general OCR framework and still relies on the recognition accuracy of the OCR model itself.

[0008] In summary, existing technologies cover areas such as document image acquisition, text detection, layout structuring, end-to-end OCR, handwritten character recognition, multi-model document extraction, and template / format priors. However, due to the scarcity of sample data for less commonly spoken languages, general OCR models have low accuracy in recognizing these languages. Furthermore, recognizing handwritten documents in less commonly spoken languages ​​presents even greater challenges due to the significant variations in field length and spacing.

[0009] The aforementioned existing solutions have the following significant drawbacks: 1. Sparse handwritten samples in low-resource minority languages ​​lead to poor model generalization: Handwritten data in minority languages ​​(such as Burmese) are typical low-resource datasets. Handwriting variations are extreme, and administrative division combinations are sparse. Without a large-scale sample size of tens of thousands, end-to-end general OCR models struggle to cover sufficient handwriting variations and administrative division combinations, resulting in a precipitous drop in recognition rates for obscure town codes.

[0010] 2. Long sequence recognition suffers from structural collapse, easily leading to character misalignment and length drift: Handwritten ID numbers are complex long strings composed of multiple semantic elements (numbers, special characters, and text from minority languages). End-to-end models (such as attention-based or CTC-based networks) are prone to positional confusion when decoding long handwritten sequences. For example, due to the lack of field boundary constraints, the strokes of the handwritten town / district code are often confused with the following serial number, resulting in serious format errors such as character misalignment, field crosstalk, and non-standard serial number length (length drift).

[0011] 3. Diminishing marginal effect of single-field fine-tuning (no significant improvement): Experiments show that when a single field (NRC number image only) is specifically fine-tuned on a general end-to-end OCR model (such as Paddle OCR), the model is too tightly coupled internally, making it difficult to adaptively learn the hard rule constraints of each segment, and the accuracy often reaches a bottleneck.

[0012] 4. Black-box models lack error interpretability and cannot be used for targeted iteration: When the end-to-end model outputs incorrect numbers, developers cannot clearly distinguish from the logs whether it is a state classification error, a township code spelling error, or a missing character in the serial number. Because the source of the error cannot be located, isolated data augmentation or model fine-tuning of specific modules is impossible, resulting in a vague direction for system optimization.

[0013] 5. Lack of Data Feedback Loop and Inadequate Quality Gating: During the data feedback process in the production environment, newly added training samples often contain dirty data such as missing fields (skip), data under manual annotation review (needs_review), bounding box overflow (bbox anomalies), and invalid labels. Existing solutions typically lack a field-level quality gating interception mechanism; directly feeding these samples into the model for retraining would severely pollute the training set. Furthermore, when new administrative division codes not included in the dictionary appear in reality, existing systems cannot dynamically expand the model's output categories and achieve automated weakly supervised pre-annotation. Summary of the Invention

[0014] To address the shortcomings of existing technologies, this invention provides a structured segmentation recognition method and system for handwritten certificates in less commonly spoken languages.

[0015] This invention discloses a structured segmentation recognition method for handwritten certificates in less commonly spoken languages, comprising the following steps: Step 1: Preprocess the image of the document to be recognized, detect the field areas on the front of the document and crop them, and divide the fields into ordinary text fields and strongly formatted fields; Step 2: For ordinary text fields, call the main OCR model to recognize them and output the recognized text and confidence score of the corresponding fields; Step 3: For strongly formatted fields, perform two recognition processes simultaneously: one process calls the main OCR model to output the recognition result of the complete field; the other process starts the structured segmentation recognition process, locates the fixed delimiter inside the strongly formatted field as the anchor point, and cuts the strongly formatted field into multiple semantically independent subfields based on the anchor point. The field-level recognition model matching each subfield is called for recognition, and the recognition candidates and corresponding confidence scores of each subfield are output. Step 4: Combining the pre-built business dictionary and field format rules, perform constraint decoding and candidate rearrangement on the recognition candidates of each subfield to generate complete structured recognition candidates; Step 5: Integrate the strong format field recognition results output by the main OCR model with the structured recognition candidates, and output the final recognition result by combining the confidence level, format validity and dictionary validity. Manually review samples with low confidence or conflicting results.

[0016] As a further improvement of the present invention, the strongly formatted field is a document number field containing a fixed separator, a fixed identifier character, and a fixed-length number sequence; the fixed separator includes a slash, a left parenthesis, and a right parenthesis, and the subfields obtained by splitting according to the anchor point include a state code subfield, a town code subfield, a fixed identifier subfield, and a fixed-length sequence number subfield.

[0017] As a further improvement of the present invention, the field-level recognition model is configured according to the semantic and format characteristics of the sub-fields: the state code sub-field adopts a finite category classification model, the town code sub-field adopts a dictionary-constrained character recognition model, the fixed identifier sub-field adopts a fixed field verification module, and the fixed-length sequence number sub-field adopts a fixed-digit multi-head classification model. Each digit is predicted and output independently, and the output length is structurally forced to be consistent with the preset number of digits.

[0018] As a further improvement of the present invention, in step 4, the hard constraint rules of the constraint decoding include: the state code value belongs to a preset legal numerical range, the town code belongs to a set of legal town codes, the combination of the state code and the town code belongs to a set of legal administrative division pairings, the fixed identifier matches a preset standard string, and the number of characters in the fixed-length sequence number is consistent with the preset length; the candidate rearrangement scoring comprehensive subfields visual recognition confidence, legal administrative division pairing bonus, format violation deduction and edit distance penalty item prioritize retaining candidates that satisfy all hard constraints.

[0019] As a further improvement of the present invention, the fusion decision logic in step 5 specifically includes: If the strong format field result output by the main OCR model satisfies all format rules and dictionary constraints, and the confidence level is higher than the preset threshold, and is consistent with the content of the structured recognition candidate, then the main OCR recognition result is directly output and marked as high confidence. If the main OCR model output has problems such as illegal format, incorrect length, or illegal administrative division pairing, and the structured recognition candidate meets the hard constraints, then the main OCR result is corrected and output after the structured recognition candidate is used. If there is a significant conflict between the two recognition results, and the confidence levels of both are lower than the preset threshold, then a low-confidence result will be output and marked as pending manual review.

[0020] As a further improvement of the present invention, the training data construction step of the field-level recognition model includes: Collect sample images of documents and complete field-level manual verification to generate a list of annotations; For strongly formatted field samples, initial annotations for delimiter anchors are automatically generated using template rules, historical position statistics, or pre-trained models. The position and number of initial anchor points are manually corrected through the annotation page, and the usability of the samples is marked. Training sample sets for each subfield are generated based on the corrected anchor points.

[0021] As a further improvement of the present invention, when constructing subfield training samples, a clipping jitter enhancement is introduced on the basis of manually corrected anchor point clipping boundaries to generate a slightly offset training clipping map. At the same time, a slight affine transformation, brightness change and noise enhancement are combined to simulate the anchor point positioning error during actual inference.

[0022] As a further improvement of the present invention, a field-level quality gating mechanism is implemented during the training data construction process: samples that are skipped at the document level, skipped at the field level, have pending annotations, have empty labels, have missing images, have abnormal bounding boxes, or have illegal field content formats are removed. Abnormal samples are included in the abnormal list for unified management and can only be returned to the training dataset after manual review and correction.

[0023] As a further improvement to the present invention, a continuous learning iteration step is also included: Low-confidence samples, abnormal samples, and two-way conflict samples marked as requiring manual review during the collection and recognition process are added to the training dataset after manual correction. The main OCR model and the recognition models at each field level are iteratively optimized, and the business dictionary and fusion decision rules are updated simultaneously to form a continuous learning closed loop of data feedback.

[0024] This invention also discloses a structured segmentation recognition system for handwritten certificates in less commonly spoken languages, used to implement the above-mentioned method, comprising: The image preprocessing and field detection module is used to preprocess the image of the document to be recognized, detect and crop the field areas on the front of the document, and distinguish them into ordinary text fields and strongly formatted fields. The main OCR recognition module is used to perform general text recognition on all field areas and output the recognized text and corresponding confidence scores. The structured segmentation recognition module is used to locate the fixed delimiter anchor point inside the strongly formatted field, split it into multiple semantically independent subfields, and call the corresponding field-level recognition model to output the recognition candidates for each subfield. The constraint decoding module is used to combine a pre-built business dictionary and field format rules to perform constraint rearrangement on subfield recognition candidates and generate complete structured recognition candidates. The fusion decision module is used to fuse the main OCR recognition result with the structured recognition candidate, output the final recognition result and mark the review status; The continuous iteration module is used to manage manually corrected samples, perform quality gating, and iteratively update the model, dictionary, and fusion rules.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Significantly improves the recognition accuracy of strongly formatted fields. By splitting long serial numbers into multiple subfields for separate recognition and combining dictionary and format constraints, problems such as character misalignment and length drift are effectively solved; under small sample conditions, the field-level model's recognition performance is significantly better than a single end-to-end OCR model.

[0026] 2. Improve error interpretability and optimization efficiency. Outputting recognition results and error statistics by subfield allows for precise identification of error sources, supports targeted data augmentation and model fine-tuning, and avoids the problem of unclear optimization direction in black-box models.

[0027] 3. Reduce annotation costs and form a sustainable iterative closed loop. By automatically pre-annotating anchor points, the annotation cost of field-level samples is reduced. Combined with quality gating and data feedback mechanisms, the model performance can be continuously iterated and optimized as business data accumulates.

[0028] 4. It possesses good versatility and scalability. The solution is not limited to Myanmar ID card recognition, but can be extended to other scenarios involving handwritten document fields in less commonly spoken languages ​​that include fixed separators, fixed-length sequences, and business dictionaries.

[0029] Based on actual sample verification, under the condition of limited samples of handwritten certificates in less commonly spoken languages, the method of this invention can significantly improve the recognition accuracy and format legality of key fields, and has high engineering applicability. Attached Figure Description

[0030] Figure 1 The flowchart shows the structured segmentation recognition method for handwritten certificates in minority languages ​​disclosed in this invention. Figure 2 for Figure 1 A flowchart of training data construction and model training in China; Figure 3 for Figure 1 Flowchart of document recognition application in China; Figure 4 for Figure 1 Flowchart of manual review and continuous iteration in China; Figure 5 This is a framework diagram of the structured segmentation recognition system for handwritten certificates in minority languages ​​disclosed in this invention; Figure 6 This is a desensitized diagram of the annotation example of the core fields on the front of the ID card disclosed in this invention; Figure 7 This is a schematic diagram illustrating the desensitization of NRC number strongly formatted field segmentation and anchor annotation examples disclosed in this invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] The present invention will now be described in further detail with reference to the accompanying drawings: This invention aims to solve the following technical problems: 1. Address the issue of insufficient OCR training samples for handwritten ID documents in less commonly spoken languages.

[0033] Documents in less commonly spoken languages ​​often lack large-scale publicly labeled data, making it difficult for end-to-end models to learn stable field structures and writing variations with only a small number of samples. This invention reduces labeling costs by implementing a closed loop of field splitting, field-level sample construction, automatic pre-labeling, and manual correction, enabling continuous expansion and iteration of training data.

[0034] 2. Resolve the issue of unstable recognition of complex number fields in end-to-end OCR.

[0035] For document numbers containing numbers, minority language characters, fixed identifiers, and special separators, end-to-end OCR is prone to issues such as field misalignment, missing character recognition, duplicate recognition, length drift, and format errors. This invention breaks down the complete number into multiple structural fields for separate recognition, reducing the difficulty for a single model to directly recognize long sequences.

[0036] 3. Resolve the issue of missing structural constraints in the document number field.

[0037] Taking the Myanmar National Recognition Certificate (NRC Number) as an example, it can be broken down into `state`, `township`, `marker`, and `serial`. `state` and `township` have an administrative division relationship, `marker` is a fixed field, and `serial` is a fixed-length numeric string. This invention utilizes an administrative division dictionary, field format rules, and a candidate rearrangement mechanism to structurally constrain the recognition results, avoiding output numbers that conform to the format but have unreasonable content.

[0038] 4. Solve the problem of difficulty in coordinating general OCR and field-level recognition models.

[0039] General OCR models are suitable for covering multiple fields on the front of an ID card, such as name, date of birth, date of issuance, ethnicity / religion, height, and blood type. However, for strongly structured fields like NRC Numbers, relying solely on the main OCR is insufficient to fully utilize the internal rules of the field. This invention proposes a collaborative recognition mechanism between the main OCR and field-level small models: the main OCR outputs a general recognition result, the field-level model outputs structured candidates, and then the confidence level, format validity, dictionary validity, and candidate score are combined for fusion, error correction, or verification marking.

[0040] 5. Solves the problems of high cost of field boundary annotation and difficulty in ensuring the quality of field clipping.

[0041] For strongly structured fields, this invention introduces an automatic anchor bootstrap method. This involves first automatically pre-annotating the positions of key delimiters using templates, image rules, or existing models. For example, it automatically locates anchors such as ` / `, `(`, `)` in NRC Numbers. Then, these anchors are quickly corrected manually via a web annotation page. The corrected anchors are used to generate high-quality field crops, which are then used to train field-level models such as state, township, marker, and serial, forming a closed loop of "automatic pre-annotation - manual correction - field sample generation - model training - re-pre-annotation".

[0042] 6. Solve the problems of OCR results being difficult to interpret and difficult to optimize in a targeted manner.

[0043] Traditional end-to-end OCR can only provide complete field recognition results, making it difficult to determine which field, character, or constraint the error originates from. This invention outputs recognition results, confidence levels, and error types by field, and can separately calculate metrics such as state accuracy, township accuracy, state-township pair validity rate, serial accuracy, and format validity rate, thereby supporting targeted optimization of models and data.

[0044] Therefore, the purpose of this invention is to improve the recognition accuracy and engineering usability of key fields of documents, especially the strongly formatted number field, under the condition of limited training data for handwritten documents in minority languages, through structured segmentation recognition, field-level dedicated models, rule and dictionary constraints, collaboration between the main model and the small model, automatic annotation closed loop, and quality gating mechanism.

[0045] To achieve the above objectives, this invention employs an overall workflow of "document image preprocessing + field detection and cropping + general recognition by main OCR + specific recognition of strongly formatted fields + rule / dictionary constraint fusion + quality feedback closed loop." The system first inputs a front image of an ID card, performing orientation correction, edge cropping, scale normalization, brightness / contrast regularization, and blur quality detection. Subsequently, it detects various field regions on the document, such as document number, issuance date, name, father's name, date of birth, ethnicity / religion, height, and blood type, and crops these field regions into standardized crop images. For ordinary text fields, the system prioritizes the use of the main OCR model for unified recognition. For fields with strict format requirements, the system additionally calls field-level small models or structured recognition pipelines in addition to the main OCR output, and then fuses them through field rules, dictionary constraints, confidence scores, and candidate rearrangement strategies to output the final recognition result and verification marks. The main OCR model can employ an end-to-end text recognition model, such as Paddle OCR, SVTR, CRNN-CTC, or Transformer OCR. Field-level sub-models, depending on the field structure, employ classification models, fixed-digit multi-head classification models, field-level character recognition models, and fixed-identifier verification modules, respectively. For NRC Numbers, the state uses a finite-class classification model, the township uses a dictionary-constrained field recognition model, the marker uses a fixed-field verification module, and the serial uses a six-digit multi-head classification model. In the technical implementation, the main OCR model uses Paddle OCR v5.

[0046] This invention uses Myanmar ID card recognition as a specific embodiment, but its technical solution is not limited to Myanmar ID cards. This invention is applicable to other minority languages, low-resource languages, handwritten or semi-structured document fields, as long as the field has a certain internal structure, format rules, or a maintainable dictionary.

[0047] like Figures 1-4 As shown, the specific identification method of the present invention includes: S1. Training Data Construction and Model Training Phase S11. Document Sample Collection and Preprocessing The system first collects front images of documents in less commonly spoken languages, and then performs preprocessing operations such as orientation correction, edge cropping, size normalization, brightness and contrast processing, and blur detection on the images to bring document images from different shooting sources, with different resolutions, and under different lighting conditions into a unified data processing scale.

[0048] Taking the Myanmar ID card as an example, the system uniformly processes the front image of the document into a standard size and retains metadata such as image source, document_id (unique ID number), image path, and preprocessing status, which serve as the basis for subsequent field detection, manual verification, and model training.

[0049] S12. Detection, cropping, and manual verification of fields on the front of the document. The system locates fields on the front of the document, obtains the bounding boxes (bbox) of each field, and crops out partial images of each field. Fields include plain text fields and strongly formatted fields. Plain text fields include full_name, father_name, and race_religion; short or semi-structured fields include date_of_birth, issue_date, height, and blood_type; strongly formatted fields include nrc_number (NRC ID number), such as... Figure 6 As shown.

[0050] The system writes information such as field crop, field name, bounding box coordinates, initial recognition draft, manual proofreading results, and field status into a corrected manifest. During manual proofreading, fields can be marked as approved, pending review, or invalid (approved, needs_review, or skip). Samples with document-level or field-level skips are not included in the training set. The training, validation, and test sets are divided according to document_id to prevent different fields of the same ID card from being leaked into different splits.

[0051] S13. Construction of training data for the main OCR model The main OCR model is used to recognize multiple core fields on the front of the document. The system filters the field samples that have passed the verification from the corrected manifest, and removes samples with document-level skips, field-level skips, needs_reviews, empty labels, missing images, abnormal bounding boxes, and obviously illegal field content.

[0052] The main OCR training data includes nrc_number, issue_date, full_name, father_name, date_of_birth, race_religion, height, and blood_type. Fields with insufficient training samples or inadequate proofreading quality may be excluded from the main training set.

[0053] The main OCR model is trained using local field images as input and manually proofread text as labels. After training, the system evaluates metrics such as exact match (EM), CER, and NED for each field separately to avoid the overall metrics masking recognition issues in certain fields.

[0054] S14. Strongly formatted field definition and field-level training data construction For fields with fixed structure, fixed length, fixed identifier, or dictionary constraints, the system defines them as strongly formatted fields. Taking the Myanmar National Recognition Certificate (NRC Number) as an example, its structure is state / township (marker) serial, where state is the state code, township is the township code, and marker is a fixed identifier, for example... Serial is a fixed-length string of numbers.

[0055] The system no longer trains NRC Numbers as a single line of text, but instead constructs field-level training samples.

[0056] S15. Strongly formatted field anchor pre-annotation and manual correction For fields like NRC Number that contain fixed delimiters, the system locates the internal anchors of the field. For example, in the state / township(marker) serial, " / ", "(", and ")" serve as key anchors, respectively.

[0057] The system can first generate an anchor bootstrap using template rules, historical sample statistics, lightweight detection models, or existing OCR results, that is, automatically generate an initial annotation for the positions of separators such as " / ", "(", and ")". Subsequently, the anchors are manually corrected through the web annotation page, including dragging their positions, adjusting their sizes, deleting incorrect anchors, and saving incomplete sample states.

[0058] For incomplete, ambiguous, or unlabeled NRC samples, missing anchors can be removed, and the sample can be marked as unsuitable for full field-level training.

[0059] S16. Strongly formatted field segmentation and data augmentation Based on the manually corrected anchors, the system physically segments the complete NRC Number crop into multiple subfields such as state, township, marker, and serial. During cropping, each subfield retains a certain amount of padding to avoid characters being missing due to overly tight boundaries.

[0060] like Figure 7 As shown, the system further annotates the internal anchors, including slash, left parenthesis, and right parenthesis. Based on the anchors, the system splits the complete serial number into four sub-fields: state, township, marker, and serial, and generates field-level training samples for each. In the figure, state, township, marker, and special separators retain their true visual form to reflect the segmentation technique; the serial region uses example Burmese numerals. "Replace the real 6-digit serial number. Among them, state enters the limited category classification model, township enters the dictionary constraint recognition model, marker enters the fixed field verification module, and serial enters the fixed-digit number model."

[0061] When training a small field-level model, the system can introduce enhancements to the field crop, including crop jitter, slight affine transformation, brightness variation, noise, blur, and slight crop offset. Crop jitter refers to generating a slightly offset training crop based on the manually corrected crop box to simulate the situation where the anchor positioning is not completely accurate during actual inference, thereby improving the robustness of the field-level model to boundary errors.

[0062] S17, Field-level Small Model Training The system trains different small models based on the characteristics of each subfield. For the state field, a finite-class classification model is used, such as 1 to 14 classes; for the serial field, a fixed-length digit model is used, such as a 6-headdigit classifier, where each digit independently predicts the Burmese digit. to (0 to 9) Forces a 6-digit output from the model structure to avoid extra characters, missing characters, or length drift in ordinary OCR models.

[0063] For the "township" field, field-level OCR, character combination classification models, or metric learning models are used, combined with a set of valid "township" candidates for training and decoding. For the "marker" field, a fixed output plus visual verification approach can be used. Since the marker is usually a fixed string, for example... The system no longer allows end-to-end OCR to repeatedly learn the fixed field, but instead uses it as part of the structural rules for validation.

[0064] S18, Dictionary Construction and Candidate Constraint Configuration The system maintains a business dictionary and a local override table for strongly formatted fields. Taking NRC Number as an example, the system loads valid state, township, and state-township pairs. For state-township pairs that are temporarily missing from the external dictionary but have been manually confirmed to exist, they can be added to the local override table.

[0065] After training, the field-level model not only outputs visual recognition results, but can also incorporate a dictionary for candidate constraints during the decoding phase. For example, it can limit the candidate range for "township" based on the top-k candidates of the state, thereby reducing invalid combinations in the output.

[0066] S2, Document Recognition Application Stage S21. Inputting ID document image and locating fields During actual recognition, the system receives the image of the document to be recognized and performs the same preprocessing as in the training phase, including orientation correction, edge cropping, size normalization, and image quality detection. Subsequently, the system detects the field regions on the front of the document and crops out each field. For ordinary fields, they are directly fed into the main OCR model; for strongly formatted fields, they are simultaneously processed through both the main OCR recognition process and the field-level structured recognition process.

[0067] S22. The main OCR performs general recognition on the front-facing fields. The main OCR model performs unified recognition on the crop field on the front of the document, outputting the text result and confidence score for each field. For common or semi-structured fields such as full_name, father_name, race_religion, date_of_birth, issue_date, height, and blood_type, the system primarily uses the main OCR output as candidate results and performs validation according to field rules.

[0068] For example, the system can verify whether the date field conforms to the date format, whether the height field conforms to the height format, whether the blood type field belongs to the valid blood type set, and whether the name field is empty or suspected of being garbled. If the main OCR output is valid and has a high confidence level, it can be directly used as a high-confidence result; if the output is empty, has low confidence, or has an abnormal format, it is marked as pending review.

[0069] S23, Structured Recognition of Strongly Formatted Fields For strongly formatted fields such as NRC Numbers, the system not only uses the main OCR to recognize the complete field but also initiates a field-level structured recognition process. The system first locates anchors within the field, such as " / ", "(", and ")", and then segments the field into state, township, marker, and serial based on these anchors. It then calls the state classification model, township field model, marker verification module, and serial six-digit model respectively. The field-level model outputs multiple structured candidates and their confidence scores.

[0070] S24, Dictionary-Constrained Decoding and Candidate Reordering The system combines the field-level model output with the business dictionary to perform constraint decoding and candidate rearrangement. Taking NRCNumber as an example, the system enforces the following constraints: state must be between 1 and 14, township must be in a valid township set, state-township pair must be valid, marker must conform to a fixed identifier, and serial must be a 6-digit number.

[0071] Candidate scores can be calculated by considering factors such as state_visual_score, township_visual_score, serial_visual_score, marker_score, valid_pair_bonus, edit_distance_penalty, and format_violation_penalty. The system prioritizes candidates that meet the hard constraints. If the candidate score difference is too small, the dictionary correction is too large, or the field-level model has low confidence, a high-confidence result will not be output directly, but will be marked as pending review.

[0072] S25, Decision on Fusion of Main OCR and Field-Level Small Model The system simultaneously obtains the main OCR candidate and field-level structured candidates. For ordinary fields, the main OCR candidate is used by default. For strongly formatted fields, the system performs format validation and dictionary validation on the main OCR output. If the main OCR output meets the requirements for format, length, fixed identifiers, dictionary validity, and confidence, and is consistent with or highly similar to the field-level structured candidates, the main OCR result is directly output and marked as high confidence.

[0073] If the main OCR output contains errors such as invalid format, incorrect length, invalid dictionary pair, or low confidence, the field-level model is invoked for recognition to obtain candidate results, which are then incorporated into the fusion decision process. The fusion decision integrates information such as the main OCR confidence, field-level model confidence, format validity, dictionary pair validity, serial number (whether it is a fixed-digit number), marker reliability, edit distance between two candidate outputs, candidate score difference, and historical error types.

[0074] When the field-level model provides a more credible candidate under structural constraints, the system uses the field-level result to correct the main OCR; when the two models conflict and cannot be determined, the system outputs a low-confidence result and marks it for manual review, instead of forcibly correcting the error automatically.

[0075] S26. Output of recognition results The system ultimately outputs the recognition results and quality information for each field, including field text, field type, field bounding box, main OCR confidence, field-level model confidence, format verification results, dictionary verification results, fusion source, reason for low confidence, and whether manual review is required.

[0076] For strongly formatted fields such as NRC Number, the system also outputs structured subfields such as state, township, marker, and serial, as well as the confidence level, candidate source, and error type of each subfield, which helps the business system to explain the source of the error.

[0077] S3, Manual Review and Continuous Iteration Phase S31. Verification of Low-Confidence Samples and Abnormal Samples The system pushes low-confidence samples from the main OCR, low-confidence samples from the field-level model, samples with illegal formatting, dictionary conflict samples, conflict samples between the main OCR and the field-level model, abnormal bounding box samples, and anchor positioning failure samples to manual review. Manual corrections can be made to the field text, bounding box, anchor, field status, and document status. The correction results are written to the correctedmanifest as a source of subsequent training data.

[0078] S32. Quality Gating and Exception List Management Before samples are fed back into the training set, the system performs quality gating. Samples with document-level skips, field-level skips, needs_reviews, empty labels, missing images, abnormal bounding boxes, invalid field formats, abnormal serial bit lengths, or missing anchors that cannot be segmented are not directly included in the training; instead, they are added to an anomaly list. These anomalies can be re-enabled based on manual review results.

[0079] S33, Error Analysis and Model Iteration The system generates assessment reports and error analysis results by field, such as overall exact, overall CER, NRC state accuracy, NRC township accuracy, state-township pair accuracy, serialaccuracy, format valid rate, mean NED, etc.

[0080] When the system detects a persistent occurrence of a certain type of error, such as frequent misidentification of certain towns, incorrect serial number digits, crop offset in certain document formats, or missing state-township pairs, the system can perform targeted optimizations. These optimizations include supplementing manual annotations, correcting bounding boxes, updating anchor annotations, expanding the business dictionary, adding local overrides, updating township classes, retraining field-level small models, fine-tuning the main OCR model, and adjusting fusion decision rules.

[0081] S34, Continuous Learning Loop Through the above process, the system forms a continuous learning closed loop: real business document images → image preprocessing → field localization and cropping → initial OCR recognition → anchor bootstrap → manual correction → corrected manifest → main OCR training → field-level small model training → recognition application → error analysis → data and model iteration.

[0082] This closed loop enables the system to continuously improve the accuracy of key field recognition, the validity of format, and the interpretability of errors, even with a limited initial sample of handwritten ID documents in less common languages, as real business data gradually accumulates.

[0083] like Figure 5 As shown, this invention provides a structured segmentation recognition system for handwritten certificates in less commonly spoken languages, used to implement the above-mentioned method, including: The image preprocessing and field detection module is used to preprocess the image of the document to be recognized, detect and crop the field areas on the front of the document, and distinguish them into ordinary text fields and strongly formatted fields. The main OCR recognition module is used to perform general text recognition on all field areas and output the recognized text and corresponding confidence scores. The structured segmentation recognition module is used to locate the fixed delimiter anchor point inside the strongly formatted field, split it into multiple semantically independent subfields, and call the corresponding field-level recognition model to output the recognition candidates for each subfield. The constraint decoding module is used to combine a pre-built business dictionary and field format rules to perform constraint rearrangement on subfield recognition candidates and generate complete structured recognition candidates. The fusion decision module is used to fuse the main OCR recognition result with the structured recognition candidate, output the final recognition result and mark the review status; The continuous iteration module is used to manage manually corrected samples, perform quality gating, and iteratively update the model, dictionary, and fusion rules.

[0084] The key point of this invention is not simply replacing an OCR network model, but rather combining the overall field recognition of the document, the structured segmentation of strongly formatted fields, the training of field-level small models, business dictionary constraints, the fusion of the main OCR and the small models, and the closed-loop data correction for handwritten ID document recognition in minority languages ​​into a complete technical process. The specific key technical points are as follows: 1. The system combines overall field recognition of the front of the ID card with specialized recognition of strongly formatted fields. The system does not only recognize NRC numbers, nor does it only train a single end-to-end OCR. Instead, it first performs field detection, cropping, and general recognition of multiple core fields on the front of the ID card, and then initiates a field-level specialized recognition process for fields with fixed formats, fixed lengths, or business dictionary constraints, such as NRC numbers, dates, and serial numbers.

[0085] 2. Transform strongly formatted fields from "plain text lines" to "structured field sequences." Taking NRC numbers as an example, the system defines the complete number as four subfields: state, township, marker, and serial, instead of treating the entire string of text as an uninterpretable OCR output. This structured definition allows each type of error to be located in a specific subfield, facilitating training, evaluation, and subsequent error correction.

[0086] 3. Physical segmentation and cropping using internal field anchors. The system uses fixed delimiters such as " / ", "(", and ")" as anchors to locate the internal boundaries of the NRC number crop, thereby cropping the state, township, marker, and serial subfields. This approach differs from simply performing field detection outside the complete image, and can further address issues such as string interference, length drift, and unstable CTC alignment within fields.

[0087] 4. Establish an automatic anchor pre-annotation and manual correction mechanism. New samples are first automatically generated as anchor bootstraps based on historical location statistics, rules, or the model. Then, manual correction is performed by dragging, scaling, and deleting anchor boxes via a web page. For incomplete or ambiguous samples, anchors can be deleted and saved as non-training states. This mechanism reduces the cost of field-level sample annotation and prevents erroneous crops from contaminating the training of small models.

[0088] 5. Different field-level small models are used for different subfields, instead of forcibly recognizing all content with a single model. The state uses a finite-class classification model, the township uses a field-level character recognition / candidate classification model, the marker uses a fixed-field validation or fixed-output module, and the serial uses a six-digit multi-head classification model. The serial model independently predicts each digit from 0 to 9, forcing a 6-digit output from the model structure, avoiding common OCR errors such as extra characters, missing characters, and length errors.

[0089] 6. Introduce state-conditioned administrative division dictionary constraints in township identification. The system limits the number of legal township candidates based on the state top-k results and constructs legal state-township pairs using an external NRC dictionary and a manually verified local override table. When a true pair is found to be outside the candidate space, the system expands the township classes and retrains them, instead of simply performing post-processing string replacement.

[0090] 7. Remove the fixed marker from the end-to-end identification burden. For fixed markers appearing in the NRC number... Instead of relying on the main OCR to repeatedly learn the fixed string, the system uses fixed output, visual verification, or format verification to reduce the drag on the complete exact match caused by marker encoding differences and Burmese character recognition errors.

[0091] 8. The main OCR and field-level small models adopt a fusion decision-making approach, rather than a simple choice between the two. The system simultaneously retains main OCR candidates and field-level structured candidates, and comprehensively considers the confidence of the main OCR, the confidence of the field-level models, the format validity, the validity of the dictionary pair, the serial length, marker validation, candidate edit distance, and candidate score difference to decide whether to output directly, perform structured error correction, or mark for manual review.

[0092] 9. Perform structured risk assessment on the main OCR results that are "formatted correctly but contain incorrect content". Experiments show that erroneous results from the main OCR may still be formatted correctly and have a high confidence level. Therefore, this invention does not rely solely on low confidence thresholds to trigger rollback, but combines state-township dictionaries, field-level visual scores, serial location information, and historical error types for multi-evidence assessment.

[0093] 10. Establish field-level quality gating and anomaly lists. During the training data construction phase, filter samples such as document-level skips, field-level skips, needs_reviews, empty labels, missing images, abnormal bounding boxes, illegal field formats, and abnormal serial number lengths; add abnormal samples to the list instead of discarding them directly, which facilitates subsequent review, correction, and incremental reflow.

[0094] 11. Supports a data loop for continuous learning. Starting with real ID card images, the system follows a continuous iterative process including preprocessing, field detection, OCR pre-recognition, manual verification, corrected manifest, field crop generation, model training, error QA, dictionary expansion, and retraining. When new ID card samples are added to the system, the same process can be reused to continuously improve the recognition capabilities for handwritten fields in less commonly spoken languages.

[0095] 12. It has universality for application to handwritten ID documents in other less commonly spoken languages. As long as the target ID document field contains a region code, fixed separator, fixed identifier, date, fixed-length serial number, or business dictionary, it can be defined as a strongly formatted field, and the anchor, subfield model, dictionary constraints, and fusion decision can be established according to the method of this invention, and it is not limited to Myanmar ID card recognition.

[0096] Example This project used two batches of Myanmar ID card front samples and NRC number verification data for validation. The following experimental data illustrates the effectiveness of the main OCR model, structured field mini-model, dictionary constraints, and data closure in this invention.

[0097] The basic information of the two batches of data samples is shown in Table 1.

[0098] Batch 1 consists of existing real ID card front verification data, totaling 382 document_ids. The original corrected manifest contains 3554 field samples, of which 2894 belong to the core front fields of this round of Core8. Batch 2 consists of newly added ID card front images, totaling 749 images. After preprocessing, all images were successfully generated. After removing duplicates and unusable samples, the corrected manifest contains 747 document_ids and 7470 field samples, of which 5976 belong to the core front fields. When finally constructing the unified front_ocr_core8 dataset, after removing back fields, distinguishing_marks, address_info, skip, needs_review, and abnormal samples, 1117 document_ids and 8019 field samples were retained.

[0099] Table 1 The unified dataset was re-split into train / validation / test in an 8:1:1 ratio. The test split contains 111 document_ids and 811 field samples. By source, batch 1 test samples consist of 34 document_ids, 259 field samples, and 34 NRC numbers, while batch 2 test samples consist of 77 document_ids, 552 field samples, and 77 NRC numbers.

[0100] The results of training the main OCR model on the first and second batches of data are shown in Table 2.

[0101] Table 2 After the second batch of proofreading data was added, the overall accuracy and character error rate of the PaddleOCR main model improved, proving that the new proofreading samples were effective in training the OCR of the front of the document. The improvement was mainly reflected in structured or semi-structured fields such as NRC number, date, ethnicity / religion, and height; while the improvement in open-ended Burmese handwritten fields such as name and paternal name was limited.

[0102] Among the structured fields mentioned above, the NRC number field showed the lowest improvement in recognition accuracy compared to other fields, both in terms of growth rate and absolute value. However, fields like the NRC number, which have strict format specifications, have higher accuracy requirements in practical applications. Given the scarcity of handwritten ID document samples in less common languages, it is difficult to improve recognition accuracy by relying on the capabilities of the main OCR model.

[0103] The results of training the first and second batches of data for the SegOCR field-level mini-model are shown in Table 3.

[0104] Table 3 On the same first batch of test sets, the SegOCR mini-model trained with the second batch of data improved state encoding from 75% to 83.33%, township encoding from 38.89% to 55.6%, and 6-digit Burmese digit encoding from 50% to 58.33% compared to the first batch of models. This shows that the addition of data and field-level training significantly improved the generalization ability of the structured model.

[0105] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A structured segmentation recognition method for handwritten certificates in less commonly spoken languages, characterized in that, Includes the following steps: Step 1: Preprocess the image of the document to be recognized, detect the field areas on the front of the document and crop them, and divide the fields into ordinary text fields and strongly formatted fields; Step 2: For ordinary text fields, call the main OCR model to recognize them and output the recognized text and confidence score of the corresponding fields; Step 3: For strongly formatted fields, perform two recognition processes simultaneously: one process calls the main OCR model to output the recognition result of the complete field; the other process starts the structured segmentation recognition process, locates the fixed delimiter inside the strongly formatted field as the anchor point, and cuts the strongly formatted field into multiple semantically independent subfields based on the anchor point. The field-level recognition model matching each subfield is called for recognition, and the recognition candidates and corresponding confidence scores of each subfield are output. Step 4: Combining the pre-built business dictionary and field format rules, perform constraint decoding and candidate rearrangement on the recognition candidates of each subfield to generate complete structured recognition candidates; Step 5: Integrate the strong format field recognition results output by the main OCR model with the structured recognition candidates, and output the final recognition result by combining the confidence level, format validity and dictionary validity. Manually review samples with low confidence or conflicting results.

2. The method according to claim 1, characterized in that, The strongly formatted field is a document number field containing a fixed delimiter, a fixed identifier character, and a fixed-length number sequence; the fixed delimiter includes a forward slash, a left parenthesis, and a right parenthesis, and the subfields obtained by splitting according to the anchor point include a state code subfield, a town code subfield, a fixed identifier subfield, and a fixed-length sequence number subfield.

3. The method according to claim 2, characterized in that, The field-level recognition model is configured according to the semantic and format characteristics of the sub-fields: the state code sub-field adopts a limited category classification model, the town code sub-field adopts a dictionary-constrained character recognition model, the fixed identifier sub-field adopts a fixed field verification module, and the fixed-length sequence number sub-field adopts a fixed-digit multi-head classification model. Each digit is predicted and output independently, and the output length is forced to be consistent with the preset number of digits from a structural perspective.

4. The method according to claim 2, characterized in that, In step 4, the hard constraint rules for constraint decoding include: the state code value belongs to a preset legal value range, the town code belongs to a set of legal town codes, the combination of the state code and the town code belongs to a set of legal administrative division pairings, the fixed identifier matches a preset standard string, and the number of characters in the fixed-length sequence number is consistent with the preset length; the candidate reordering score is based on the comprehensive subfields of visual recognition confidence, legal administrative division pairing bonus, format violation deduction, and edit distance penalty, and candidates that satisfy all hard constraints are given priority.

5. The method according to claim 1, characterized in that, The fusion decision logic in step 5 specifically includes: If the strong format field result output by the main OCR model satisfies all format rules and dictionary constraints, and the confidence level is higher than the preset threshold, and is consistent with the content of the structured recognition candidate, then the main OCR recognition result is directly output and marked as high confidence. If the main OCR model output has problems such as illegal format, incorrect length, or illegal administrative division pairing, and the structured recognition candidate meets the hard constraints, then the main OCR result is corrected and output after the structured recognition candidate is used. If there is a significant conflict between the two recognition results, and the confidence levels of both are lower than the preset threshold, then a low-confidence result will be output and marked as pending manual review.

6. The method according to claim 1, characterized in that, The training data construction steps for the field-level recognition model include: Collect sample images of documents and complete field-level manual verification to generate a list of annotations; For strongly formatted field samples, initial annotations for delimiter anchors are automatically generated using template rules, historical position statistics, or pre-trained models. The position and number of initial anchor points are manually corrected through the annotation page, and the usability of the samples is marked. Training sample sets for each subfield are generated based on the corrected anchor points.

7. The method according to claim 6, characterized in that, When constructing training samples for subfields, a clipping jitter enhancement is introduced on the basis of manually corrected anchor point clipping boundaries to generate slightly offset training clipping maps. At the same time, slight affine transformation, brightness variation and noise enhancement are combined to simulate the anchor point positioning error during actual inference.

8. The method according to claim 6, characterized in that, During the training data construction process, a field-level quality gating mechanism is implemented: samples that are skipped at the document level, skipped at the field level, have pending annotations, have empty labels, have missing images, have abnormal bounding boxes, or have illegal field content formats are removed. Abnormal samples are included in the abnormal list for unified management and can only be returned to the training dataset after manual review and correction.

9. The method according to claim 1, characterized in that, It also includes continuous learning and iteration steps: Low-confidence samples, abnormal samples, and two-way conflict samples marked as requiring manual review during the collection and recognition process are added to the training dataset after manual correction. The main OCR model and the recognition models at each field level are iteratively optimized, and the business dictionary and fusion decision rules are updated simultaneously to form a continuous learning closed loop of data feedback.

10. A structured segmentation recognition system for handwritten certificates in less commonly spoken languages, used to implement the method as described in any one of claims 1 to 9, characterized in that, include: The image preprocessing and field detection module is used to preprocess the image of the document to be recognized, detect and crop the field areas on the front of the document, and distinguish them into ordinary text fields and strongly formatted fields. The main OCR recognition module is used to perform general text recognition on all field areas and output the recognized text and corresponding confidence scores. The structured segmentation recognition module is used to locate the fixed delimiter anchor point inside the strongly formatted field, split it into multiple semantically independent subfields, and call the corresponding field-level recognition model to output the recognition candidates for each subfield. The constraint decoding module is used to combine a pre-built business dictionary and field format rules to perform constraint rearrangement on subfield recognition candidates and generate complete structured recognition candidates. The fusion decision module is used to fuse the main OCR recognition result with the structured recognition candidate, output the final recognition result and mark the review status; The continuous iteration module is used to manage manually corrected samples, perform quality gating, and iteratively update the model, dictionary, and fusion rules.

Images