A method for training an identification model of a certificate image, an electronic device, and a medium

Abstract

The application discloses a kind of certificate image recognition model training method, electronic equipment and medium, related to certificate image recognition technical field, method includes: using closed source big model to identify multiple certificate images, obtain the identification result corresponding to each certificate image, and the identification result of each certificate image includes multiple key information;The identification result is used to mark the corresponding certificate image, and obtain the certificate image after marking;The certificate image after marking carries multiple labels associated with the key information;Using the certificate image after marking as a sample set to train open source optical character recognition initial model, obtain the final certificate image recognition model.The application has the functions of supporting multilingual, multi-certificate type image recognition and attribute analysis, with the advantages of fast recognition speed, high accuracy, good stability and model lightweight.

Description

Technical Field

[0001] This invention relates to the field of document image recognition technology, and in particular to a training method for a document image recognition model, an electronic device, and a computer-readable medium. Background Technology

[0002] Document image recognition is a technology that automatically extracts structured data from documents. Closed-source large models and lightweight open-source OCR (Optical Character Recognition) small models are the two mainstream solutions for implementing this technology.

[0003] However, there are numerous types of identification documents (such as ID cards, household registration books, passports, driver's licenses, social security cards, and business licenses), and the same type of document from different countries (such as Thailand, Vietnam, Indonesia, and India) varies in language, document format, information items, and structure. For these complex documents, while closed-source large-scale models generally outperform smaller models in terms of recognition accuracy, their results are easily affected by the choice of closed-source large-scale model, leading to instability. Furthermore, closed-source large-scale models suffer from slow recognition speed (typically over 10 seconds). Open-source small-scale OCR models, while offering a significant advantage in recognition speed, suffer from low accuracy (only about 70%). For example, with Vietnamese ID cards, the recognition results often exhibit issues such as truncated residential addresses, text illusions, and omissions or incorrect segmentation of phonetic characters. Therefore, existing solutions fail to meet practical application requirements in terms of accuracy, speed, and stability. Summary of the Invention

[0004] This invention aims to address, to a certain extent, one of the technical problems in related technologies. To this end, this invention provides a training method for a document image recognition model, an electronic device for executing the training method, and a computer-readable medium. This method supports image recognition and attribute parsing for multiple languages ​​and document types, and offers advantages such as fast recognition speed, high accuracy, good stability, and lightweight model.

[0005] To achieve the above objectives, as a first aspect of the present invention, a training method for a document image recognition model is provided, wherein the training method includes:

[0006] Multiple document images are identified using a closed-source large model, and the recognition results for each document image are obtained. The recognition results for each document image include multiple key pieces of information.

[0007] The identification results are used to tag the corresponding document images to obtain tagged document images; the tagged document images carry multiple tags associated with the key information;

[0008] The initial open-source optical character recognition model was trained using the tagged document images as a sample set to obtain the final document image recognition model.

[0009] Optionally, the key information includes fields representing global attributes of the document image and their values;

[0010] The recognition result is used to tag the corresponding document image to obtain a tagged image; the tagged document image carries a tag associated with the key information; including:

[0011] By using fields representing global attributes of the document image and their values, the corresponding document image is tagged to obtain a tagged document image carrying the fields and values ​​of the associated global attributes of the document image.

[0012] Optionally, the key information may also include fields representing local attributes of the document image and their values;

[0013] The recognition result is used to tag the corresponding document image to obtain a tagged image; the tagged document image carries a tag associated with the associated information; including:

[0014] By using fields representing local attributes of the document image and their values, the corresponding document image is labeled to obtain a labeled document image carrying the fields and values ​​of the associated local attributes of the document image.

[0015] Optionally, the step of using the tagged document images as a sample set to train the initial open-source optical character recognition model to obtain the final document image recognition model includes:

[0016] Labeled document images with tags that carry global attributes of associated document images and their values ​​are used as a sample set. An initial model is trained using a reinforcement learning algorithm so that the trained model parameters can learn the overall features of the document images.

[0017] If the accuracy of global attribute recognition of document image is greater than or equal to the preset global attribute recognition accuracy, the trained model parameters are saved to obtain the global attribute recognition model of document image.

[0018] Labeled document images containing fields and values ​​of related local attributes of the document image are used as a sample set. A global attribute recognition model for document images is trained using a reinforcement learning algorithm so that the trained model parameters can learn the local features of the document image.

[0019] If the accuracy rate of local attribute recognition of the document image is greater than or equal to the preset local attribute recognition accuracy rate, the trained model parameters are saved to obtain the final document image recognition model.

[0020] Optionally, the reinforcement learning algorithm includes an outcome reward value and a format reward value;

[0021] The step of training an initial model using a reinforcement learning algorithm or training a global attribute recognition model for document images using a reinforcement learning algorithm includes:

[0022] The total reward value for each training sample is determined based on the result reward value and the format reward value.

[0023] The advantage value of each training session is determined based on the total reward value of the samples and the total number of samples in each training session.

[0024] The loss function is determined based on the determined dominance value of the samples and the group relative strategy optimization, and the model parameters of the initial model or the global attribute recognition model of the document image are trained based on the loss function.

[0025] Optionally, the training method further includes:

[0026] Update the format reward value weight and the result reward value weight according to the fields and values ​​predicted by the model;

[0027] If the model predicts fields that are not predefined or that are missing predefined fields, increase the weight of the format reward value.

[0028] The weight of the result reward value is reduced based on the weight of the increased format reward value.

[0029] Optionally, the training method further includes:

[0030] Data augmentation operations are performed on document images using a closed-source large model; wherein the data augmentation operations include at least one of image cropping, image rotation, image flipping, image illumination preprocessing, and modification of text information in the image.

[0031] Optionally, the training method further includes:

[0032] Multiple closed-source large models are used to recognize the document image, resulting in multiple recognition results. If a newline character is present in the recognition result, the newline character is changed to a display newline character. If a space character is detected to be adjacent to a newline character in the recognition result, the space character is deleted.

[0033] The multiple recognition results are voted on to obtain the voting recognition result corresponding to the document image;

[0034] The identification image is tagged using the recognition results after the voting, resulting in a tagged identification image.

[0035] The initial open-source optical character recognition model was trained using the tagged document images as a sample set to obtain the final document image recognition model.

[0036] As a second aspect of the present invention, an electronic device is provided, comprising:

[0037] One or more processors;

[0038] A memory having stored one or more computer programs that, when executed by one or more processors, cause the one or more processors to implement the training method provided according to the first aspect of the invention.

[0039] Furthermore, as a third aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the training method provided in the first aspect of the present invention.

[0040] The training method for the document image recognition model provided by this invention leverages the high accuracy of closed-source large-scale models in document image recognition. It uses the recognition results of closed-source large-scale models as a sample set for an open-source optical character recognition model with small parameters and optimizability. Through task-based recognition, the recognition results are used as "ground values" to back-annotate the document images. Task-based training and reinforcement learning mechanisms enhance the model's learning ability across different tasks. Simultaneously, during training, a well-designed reward function guides the model to focus more on image attributes corresponding to the task (such as global and local image attributes), enabling the model to specifically learn global and local image features. Ultimately, this results in a lightweight document image recognition model with high accuracy in recognizing both global and local image attributes. This post-training method solves the problems of high cost and low efficiency of relying on manual annotation of document images, difficulty in deploying large models, and slow recognition speed. By distilling the "knowledge" of large models into lightweight open-source models, the open-source models can actively reinforce learning the performance of difficult document samples, thereby achieving extremely high recognition accuracy while ensuring recognition speed and stability. This effectively solves the performance balance problem of lightweight models in practical applications.

[0041] These features and advantages of the present invention will be disclosed in detail in the following specific embodiments and accompanying drawings. The preferred embodiments or means of the present invention will be shown in detail in conjunction with the accompanying drawings, but are not intended to limit the technical solutions of the present invention. In addition, each of these features, elements and components appearing in the following text and drawings is a plurality of, and different symbols or numbers are used for convenience of representation, but all represent parts with the same or similar construction or function. Attached Figure Description

[0042] The present invention will be further described below with reference to the accompanying drawings:

[0043] Figure 1 A flowchart of a training method provided by the present invention;

[0044] Figure 2 A flowchart illustrating one embodiment of step S120 of the training method provided by the present invention;

[0045] Figure 3 A flowchart illustrating another embodiment of step S120 of the training method provided by the present invention;

[0046] Figure 4 A flowchart illustrating one embodiment of step S130 of the training method provided by the present invention;

[0047] Figure 5 A flowchart illustrating one embodiment of step S134 of the training method provided by the present invention;

[0048] Figure 6 A flowchart of one embodiment of step S131a of the training method provided by the present invention;

[0049] Figure 7 A flowchart illustrating another implementation of the training method provided by the present invention;

[0050] Figure 8 A flowchart of step S120 of another embodiment of the training method provided by the present invention;

[0051] Figure 9 A flowchart illustrating the labeling process of the training method provided by this invention;

[0052] Figure 10 The image shows a comparison of the recognition effects before and after optimization for several minority language problem samples provided by this invention.

[0053] Figure 11 A module diagram of an electronic device provided by the present invention;

[0054] Figure 12 This invention provides a schematic diagram of a computer-readable medium.

[0055] Explanation of reference numerals in the attached figures

[0056] Among them, 101 is the processor; 102 is the memory; 103 is the I / O interface; and 104 is the bus. Detailed Implementation

[0057] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described are intended to explain the present invention and should not be construed as limiting the invention.

[0058] The terms "an embodiment," "example," or "trademark" used in this specification refer to a particular feature, structure, or characteristic described in connection with the embodiment itself that may be included in at least one embodiment disclosed in this invention. The phrase "in an embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment.

[0059] Traditional and relatively mature OCR recognition can recognize images of documents, but it requires a large amount of manual sample annotation work in the early stages of model training. Although the model can be trained using annotated samples to obtain recognition results, the output is a continuous text stream that mixes all information together, and subsequent information segmentation and extraction are needed to obtain data with clear relationships. Currently deployable open-source OCR recognition models have fast inference speeds, but they only perform well for general image samples (such as Chinese ID cards) and cannot generalize to different countries, languages, and various complex document types. In other words, open-source OCR recognition models cannot simultaneously meet the requirements of high accuracy and high speed. In addition, although closed-source large models have good generalization ability, the inference results of large models developed by different providers are inconsistent for samples, resulting in poor stability. Furthermore, due to their closed-source nature and slow inference speed, they cannot be directly used for product development.

[0060] In view of this, in order to achieve high-accuracy recognition of multilingual and multi-document type images, and to effectively balance the relationship between recognition accuracy, speed, and stability, as a first aspect of the present invention, a training method for a document image recognition model is provided, such as... Figure 1 As shown, the training method includes:

[0061] In step S110, multiple document images are identified using a closed-source large model to obtain the identification results corresponding to each document image. The identification results of each document image include multiple key pieces of information.

[0062] In step S120, the corresponding document image is tagged using the recognition result to obtain a tagged document image; the tagged document image carries multiple tags associated with the key information;

[0063] In step S130, the tagged document images are used as a sample set to train the open-source optical character recognition initial model, resulting in the final document image recognition model.

[0064] Regarding the recognition results, the recognition result obtained in step S110 is in a structured data format, such as JSON structured data format. In JSON data format, key information (such as "document type", "language", "name", "document expiration") is stored in the form of key-value pairs. Taking the recognition result in JSON format for labeling the document image as an example, the labeling in step S120 essentially establishes an "association relationship" between the document image and the JSON data format. Specifically, the key-value pairs in the JSON data are reverse-mapped and labeled onto the document image, and the final labeled document image carries the label associated with the key-value pairs. Further explanation of the labeling process is that the specific key information to be recognized (i.e., the content of the "key") and the labels to be applied to the image using the closed-source large model are flexibly configured according to task requirements through multi-turn dialogue with the closed-source large model, which greatly enhances the adaptability of this method.

[0065] The training method for the document image recognition model provided by this invention is, strictly speaking, a post-training method. It leverages the high recognition rate of closed-source large-scale models for document images by distilling the "knowledge" of the large model into a lightweight open-source OCR recognition model. Based on the recognition results of the closed-source large-scale model, the document image is labeled, and the open-source OCR recognition model is specifically trained and fine-tuned to obtain a lightweight model whose recognition speed, accuracy, and stability meet the requirements of product applications. It is worth emphasizing that, unlike traditional pre-training methods, the post-training method of this invention does not require the additional generation and storage of a separate file for storing sample labels. The labeled image includes not only the image itself but also the association with key information, making the post-training method of this invention more efficient in its implementation process than traditional pre-training methods.

[0066] Considering the wide variety of document types (such as ID cards, household registration books, passports, driver's licenses, social security cards, and business licenses), and the differences in language, document format, information items, and styles among similar documents from different countries (such as Thailand, Vietnam, Indonesia, and India), the tagging logic provided by this invention does not simply tag all key information onto the document image. Instead, it tags the key information in stages, distinguishing between global and local attributes of the document image. As an optional implementation of step S120, the key information includes fields representing global attributes of the document image and their values; such as... Figure 2 As shown, the recognition result is used to tag the corresponding document image to obtain a tagged image; the tagged document image carries a tag associated with the key information; including:

[0067] In step S121, the corresponding document image is tagged using the fields and values ​​representing the global attributes of the document image to obtain a tagged document image carrying the fields and values ​​of the associated global attributes of the document image.

[0068] Fields representing the global attributes of a document image include, but are not limited to, country, language, document type, document form, and physical side of the document. Values ​​for the country field include, but are not limited to, South Korea, Thailand, Vietnam, Indonesia, India, and Malaysia. Values ​​for the language field include, but are not limited to, Korean, Thai, Vietnamese, Indonesian, and Malay. Values ​​for the document type field include, but are not limited to, ID card, household registration book, passport, driver's license, social security card, and business license. Values ​​for the document form field include, but are not limited to, card and paper. For example, Indian ID cards come in two types: card-type and paper-type. Values ​​for the physical side of the document include, but are not limited to, the front and back of the document. For example, the front of an Indian ID card contains more information, while the back only stores the address; whereas the back of a Thai ID card contains more information.

[0069] After the first stage of tagging establishes a relationship between the tagged document image and global attributes, the second stage of tagging begins. As another optional implementation of step S120, the key information further includes fields representing local attributes of the document image and their values; such as... Figure 3 As shown, the recognition result is used to tag the corresponding document image to obtain a tagged image; the tagged document image carries a tag associated with the associated information; including:

[0070] In step S121, the corresponding document image is tagged using the fields and values ​​representing local attributes of the document image to obtain a tagged document image carrying the fields and values ​​of the associated local attributes of the document image.

[0071] Fields representing local attributes of the document image include, but are not limited to, name, residential address, document number, language phonetic transcription, language syllables, and document validity period. The name field corresponds to the name of the document holder, the residential address field corresponds to the residential address of the document holder (for Indian ID cards, the residential address is usually represented by multiple lines of continuous text), the document number field corresponds to the document number of the document holder (the number of digits and format of document numbers vary from country to country), the language phonetic transcription field corresponds to the specific phonetic symbol, the language syllable field corresponds to the tone of the syllable, and the document validity period field corresponds to the start date and end date of the document.

[0072] After the second stage of labeling, the labeled document image facilitates the establishment of relationships between local image attributes. The purpose of strictly dividing the labeling process into first and second stages is to serve the phased, task-based training strategy provided by this invention. The phased, task-based training strategy generally involves first training the model to learn global image attributes, and then training the model to learn local image attributes. As an optional implementation of step S130, such as... Figure 4As shown, the step of using tagged document images as a sample set to train the initial open-source optical character recognition model to obtain the final document image recognition model includes:

[0073] In step S131, the labeled document images carrying the fields and values ​​of the global attributes of the associated document images are used as a sample set, and the initial model is trained using a reinforcement learning algorithm so that the trained model parameters can learn the overall features of the document images.

[0074] In step S132, if the accuracy of the recognition of global attributes of the document image is greater than or equal to the preset global attribute recognition accuracy, the trained model parameters are saved to obtain the document image global attribute recognition model.

[0075] In step S133, the labeled document images carrying fields and values ​​of related local attributes of the document images are used as a sample set. A global attribute recognition model for document images is trained using a reinforcement learning algorithm so that the trained model parameters can learn the local features of the document images.

[0076] In step S134, if the accuracy of the recognition of local attributes of the document image is greater than or equal to the preset local attribute recognition accuracy, the trained model parameters are saved to obtain the final document image recognition model.

[0077] Steps S131-S134 above detail the training strategy of this invention. First, the open-source optical character recognition initial model is trained using the global attributes (such as nationality, language, document type, and document format) carried by the tagged image, enabling the model to recognize the global attributes of document images. Then, based on the weights of this trained model, further training is conducted using the local attributes (such as name, residential address, language phonetic symbols, language characters, and document expiration date) carried by the tagged image, allowing the model to acquire more nuanced information about the document image. This global-to-local training strategy ensures that the model can grasp both the overall structure of the document image and accurately recognize details, thereby comprehensively optimizing the recognition effect. This phased, progressive training involves continuous fine-tuning of a single model (the initial model) to ultimately obtain a single document image recognition model, rather than training two independent models.

[0078] In the phased and progressive training process, reinforcement learning algorithms can be used to guide the model being trained to focus on specific attributes (global attributes or local attributes), thereby motivating the model to learn and strengthen feature representations related to specific tasks (learning global attribute tasks or learning local attribute tasks), thereby improving the model's performance, recognition ability and accuracy in terms of global and local attributes.

[0079] In reinforcement learning, both the first and second stages of model training employ reward functions that match the current training objective. These reward functions guide the model's optimization decision-making process. The reinforcement learning algorithm includes a result reward value and a format reward value; as an optional implementation of step S131 or step S134, such as... Figure 5 As shown, training the initial model using a reinforcement learning algorithm or training the global attribute recognition model for document images using a reinforcement learning algorithm includes:

[0080] In step S131a, the total reward value of each training sample is determined based on the result reward value and the format reward value;

[0081] In step S131b, the advantage value of each training sample is determined based on the total reward value of the sample and the total number of samples in each training session.

[0082] In step S131c, the loss function is determined based on the determined advantage value of the sample and the group relative strategy optimization, and the model parameters of the initial model or the global attribute recognition model of the document image are trained based on the loss function.

[0083] The reward value of the result satisfies the following formula (1):

[0084] (1)

[0085] in, The reward value for the result of the i-th document image; This is an indicator function. When the condition is true, the indicator function takes the value 1; when the condition is false, the indicator function takes the value 0. This refers to the value corresponding to field k in the predicted i-th document image; Let K be the value of field k in the key information carried by the i-th document image after tagged; K is the set of all fields k in the key information.

[0086] The reward value in the specified format satisfies the following formula (2):

[0087]

[0088] in, The format reward value for the i-th document image; The data format consists of all fields and values ​​of the predicted i-th document image; For the predicted i-th document image, it is the set of all fields.

[0089] The total reward value satisfies the following formula (3):

[0090] (3)

[0091] in, As the weight of the result reward value, The format reward value weight, and Update the format reward value weight and the result reward value weight based on the predicted fields and values;

[0092] The advantage value satisfies the following formula (4):

[0093] (4)

[0094] Where i and j are the sample numbers for each training iteration; A i Let be the advantage value of the i-th sample;

[0095] This represents the total reward value for the i-th sample. is the total reward value for the j-th sample; n is the total number of samples corresponding to each training iteration.

[0096] The loss function satisfies the following formula (5):

[0097]

[0098] Where θ represents the model parameters; This is the loss function for the parameters;

[0099] n is the total number of samples for each training iteration; i is the i-th training sample; a i The predicted fields and values ​​constitute structured data, which is in JSON format; i Let i be the input state of the i-th sample; This is the current strategy; Given model parameters θ, the input state s for the i-th sample i Predicted structured data a i The probability of; This is the old strategy; In the model with parameters In the case of the input state s of the i-th sample i Predicted structured data a i The probability of.

[0100] To further distinguish this invention from traditional supervised learning, the differences are emphasized again here. Traditional supervised learning requires not only manual annotation of sample ground truth information but also the storage of a separate ground truth file. During model training, the model is directly told that "the correct answer for the i-th document image is the i-th data point in the ground truth file," allowing the model to learn and imitate. In contrast, this invention employs reinforcement learning with a reward function. There is no separate ground truth file, and there is no need to extract ground truth from it. During model training, the model attempts to "generate" structured data (such as JSON), and then scores the generated JSON based on the reward function (result reward and format reward) (i.e., calculate the total reward). Reinforcement learning guides the model's learning decisions through the reward function, teaching it how to generate JSON that yields higher rewards. This differs from traditional strategies that use a differentiable loss function similar to cross-entropy to optimize the model.

[0101] The format reward value focuses on whether the format of the JSON generated by the model is correct. For example, whether there are missing or missing fields in the JSON data predicted by the currently trained model parameters for a certain type of document (e.g., when predicting Thai ID cards, the model may be confused about whether the "religious belief" field should be output, and thus choose not to output the "religious belief" field and its corresponding value, resulting in the final generated JSON data lacking the "religious belief" field), text illusion (the generated JSON data fills in fields that are not originally there, such as filling in "height" or "expiration date" fields that do not exist in some documents), and other formatting issues.

[0102] The result reward value focuses on whether the value corresponding to each key (field) in the model's predicted JSON data is correct. Specifically, the edit distance is used as the result reward function to compare the predicted value of a certain field with the true value; the smaller the edit distance, the closer it is to the true value. For example, in documents from India, Indonesia, Vietnam, and Thailand, where there are many letters with insufficient continuity or multiple characters for a single sound, the result reward value can be used to correct the prediction results. Typically, after the format reward value indicates that the predicted JSON data format is correct, the result reward value is further used to check whether the model's predicted content is correct.

[0103] To guide model learning more intelligently, the direction and focus of model learning are dynamically adjusted. During training, the format reward value and the result reward value are updated based on the prediction results. As an optional implementation of step S131a, such as... Figure 6 As shown, the training method further includes:

[0104] In step S131a1, the format reward value weight and the result reward value weight are updated according to the fields and values ​​predicted by the model.

[0105] In step S131a2, if there are non-predefined fields or predefined fields missing in the fields predicted by the model, the weight of the format reward value is increased.

[0106] In step S131a3, the weight of the result reward value is reduced according to the weight of the increased format reward value.

[0107] In the early stages of training, the model may not even understand basic formatting. Therefore, the weight of the formatting reward value can be increased to ensure the model can generate "valid" JSON where all keys (fields / key names) are present. If illusion problems (i.e., the presence of non-predefined fields) or missing fields (i.e., missing predefined fields) are found in the generated JSON, the weight of the formatting reward value can be increased further and adjusted accordingly. The relationship between the model and the reward function is to reduce the weight of the result reward value. In the later stages of training, after the format generated by the model stabilizes, the weight of the result reward value is appropriately increased, so that the model can focus its attention on improving the recognition accuracy. The reward function of reinforcement learning integrates multiple optimization objectives (format standardization and content correctness) together, which jointly guide the model to optimize training in the direction of the optimal comprehensive result. This effectively solves the common problems of "content illusion" and "field omission" when generating structured JSON, and improves the reliability and accuracy of the model output.

[0108] Due to the difficulty in obtaining image samples of documents in less commonly spoken languages ​​and the wide variety of document types, the training method further includes the following steps to expand the diversity of document samples, reduce reliance on manually collected image samples, and improve the generalization ability and accuracy of the document image recognition model of this invention:

[0109] Data augmentation operations are performed on document images using a closed-source large model; wherein the data augmentation operations include, but are not limited to, image cropping, image rotation, image flipping, image illumination preprocessing, and modification of text information in the image.

[0110] For example, currently only 30 samples have been collected from Malaysia. Data augmentation can triple the number of samples from Malaysia. Furthermore, text information in document images can be modified using image editing models (such as GPT-4o and QwenEdit) and then the model can be re-identified to improve its recognition performance.

[0111] Image cropping can randomly extract and scale local areas of an image, forcing the model to focus on local features of key regions rather than relying on a fixed image layout; image rotation can rotate randomly without direction within a limited angle (such as -45° to +45°), enhancing the model's ability to extract orientation-insensitive features; illumination preprocessing includes stacking to adjust illumination intensity and contrast, randomly adding grayscale blocks to the image to simulate occlusion, and synthesizing bright spots at random positions on the image to simulate specular reflections to deal with light spot interference from the plastic seal surface of documents. It can also randomly add noise to the image and simulate image quality degradation.

[0112] Considering that a single closed-source large-scale model may have recognition bias, and that different closed-source large-scale models each have their advantages in document recognition, their output results may also differ. As a preferred embodiment of this invention, an open-source optical character recognition model is trained using the recognition results of multiple closed-source large-scale models to obtain the final document image recognition model. As an optional implementation, such as... Figure 7 As shown, the training method further includes:

[0113] In step S110, multiple closed-source large models are used to recognize the document image, resulting in multiple recognition results for the document image; if a newline character is present in the recognition result, the newline character is changed to a display newline character; if a space character and a newline character are detected to appear adjacent to each other in the recognition result, the space character is deleted.

[0114] In step S120, multiple recognition results are voted on to obtain a voted recognition result corresponding to the document image;

[0115] In step S130, the identification image is tagged using the recognition results after the voting to obtain the tagged identification image;

[0116] In step S140, the tagged document images are used as a sample set to train the open-source optical character recognition initial model, resulting in the final document image recognition model.

[0117] Considering that address information in documents written in less common languages ​​is often in the form of continuous multi-line text (open-source models often only recognize the top line of this type of information, missing address information in other lines, leading to address truncation in the recognition results), although large models can completely recognize multi-line address information, if the recognition result of a closed-source large model is specified to be in JSON format, after the large model performs JSON format processing, it will automatically add space characters instead of line break characters, causing confusion between line break characters and space characters. Therefore, when line break characters are present in the recognition result, the line break characters are changed to visible line break characters, such as converting line break characters to visible markers "<|newline|>". When space characters and line break characters are detected to appear adjacently, the space characters are deleted. Through the above processing (step S110), the model can more easily learn the original text line break features and distinguish between text line break features and space features.

[0118] Taking the closed-source large models Gemini, GPT, and Qwen as examples, after processing the recognition results of each large model using JSON, the output recognition results (translated into Chinese for ease of understanding) are as follows:

[0119] Gemini_json

[0120] {

[0121] Language: Thai

[0122] Nationality: Thailand

[0123] "Document Type": "ID Card"

[0124] "Physical side of document": "Back"

[0125] Name: Zhang Er

[0126] "Religion": "Buddhism"

[0127] ID Number: 1-2345-67890-12-3

[0128] "Document Validity Period": "June 22, 2022 - June 21, 2032"

[0129] }

[0130] GPT_json

[0131] {

[0132] Language: Thai

[0133] Nationality: Thailand

[0134] "Document Type": "ID Card"

[0135] "Physical side of document": "Back"

[0136] Name: Zhang San

[0137] "Religion": "Buddhism"

[0138] ID Number: 1-2345-67890-12-4

[0139] "Document Validity Period": "June 22, 2022 - June 21, 2032"

[0140] }

[0141] Qwen_json

[0142] {

[0143] Language: Thai

[0144] Nationality: Thailand

[0145] "Document Type": "ID Card"

[0146] "Physical side of document": "Back"

[0147] Name: Zhang San

[0148] "Religion": "Buddhism"

[0149] ID Number: 1-2345-67890-12-5

[0150] "Document Validity Period": "June 22, 2022 - June 21, 2032"

[0151] }

[0152] The advantage of the JSON format lies in associating fields and values ​​in the recognition result. Compared to a continuous text stream output by traditional OCR character recognition (language: Thai, nationality: Thai, document type: ID card, physical side, back, name: Zhang San, religion: Buddhism, document number: 1-2345-67890-12-3, document validity: June 22, 2022 - June 21, 2032), where the association between "Zhang San" and "name" is uncertain, the JSON format always follows the specific content "Zhang San" under the "name" field, eliminating ambiguity. Upon closer inspection, it was found that the JSON recognition results of the three closed-source large models were not entirely consistent. A voting process could be performed on the results to obtain a relatively accurate recognition result, which could then be used to label the image. As an optional implementation of step S120, such as... Figure 8 As shown, the voting process for multiple recognition results to obtain a voted recognition result corresponding to the document image includes:

[0153] In step S121, key-value pairs are extracted from multiple recognition results to obtain multiple sets of key-value pairs;

[0154] In step S122, the key-value pairs are grouped by key name, and the key values ​​corresponding to each key name are counted and voted. The key value with the highest number of votes is taken as the final key value of that key name.

[0155] In step S123, each key name and its corresponding final key value are combined into a key-value pair to obtain the identification result after voting.

[0156] Specifically, after grouping and counting the votes for the recognition results of Gemini_json, GPT_json, and Qwen_json, we get: "Language": "Thai" 3 votes, "Nationality": "Thailand" 3 votes, "Document Type": "ID Card" 3 votes, "Document Physical Side": "Back" 3 votes, "Religion": "Buddhism" 3 votes, "Document Expiration": "June 22, 2022 - June 21, 2032" 3 votes. The number of votes for the above key names is consistent with the number of closed-source large models used, so the recognition results for the above key names do not need to be processed; however, the recognition results for "Name" and "Document Number" show inconsistencies, such as "Name": "Zhang Er" 1 vote. For the "Name": "Zhang Er" received 2 votes, the final identification result is "Name": "Zhang Er". For the "ID Number": "1-2345-67890-12-3" received 1 vote, "ID Number": "1-2345-67890-12-4" received 1 vote, and "ID Number": "1-2345-67890-12-5" received 1 vote. In the case of a tie, the output of the model with the best historical performance is selected. By analyzing the model with the best historical performance, Qwen is identified. Therefore, the final value for "ID Number" is "1-2345-67890-12-5". Finally, each key name and its corresponding key value after voting are combined into a key-value pair to obtain the identification result after voting.

[0157] It is important to emphasize that the document image recognition model of this invention can recognize multiple documents from multiple countries, in multiple languages, and in multiple languages, unlike models trained separately for a single country or language. When a document image is input into the document image recognition model, the model will initially determine the nationality based on the language. For example, if it sees only Chinese, it will prioritize the nationality as Chinese; if it sees Thai, it will prioritize the nationality as Thai; and if it sees only Vietnamese, it will prioritize the nationality as Vietnamese (because the writing systems of these three languages ​​are very different, and there is no possibility of confusion; for example, Thai has unique circled letters, and Vietnamese has…). / By analyzing special characters such as "", and based on the "country name identifier" or unique fields on the document (such as "ethnicity" for China, "height scale" for Thailand), and the "number format" (such as 18 digits for China, 12 digits for Vietnam), the nationality information on the document can be accurately inferred, avoiding misjudgments caused by different countries using the same language. Since the model has comprehensive image recognition capabilities after training (recognizing global or local attributes), it can accurately extract specific attribute information from the recognition results according to task requirements.

[0158] During the inference phase, since the mechanism of the large model is kvcache, the matrices from the previous N rounds can be cached. In the N+1 round, there is no need to start the entire dialogue from scratch; relevant information can be retrieved directly from the cache (working memory).

[0159] Finally, a brief description of the configuration of the document image recognition model of this invention is provided. Model foundation: Qwen2 is used as the initial open-source optical character recognition model, integrating the capabilities of closed-source large models such as Gemini, GPT, or Qwen; Training data: The average sample size for each minority language country is approximately 500 images, with the original sample size for Malaysia being 30 images, which was expanded to 90 images after data augmentation; Performance indicators: The recognition accuracy has increased from the original 70% to over 90%, the model parameter size is controlled within 3 bytes, and the single recognition time is within 3 seconds, far superior to the inference speed of the original closed-source large models (over 10 seconds). The sample labeling process is as follows... Figure 9 As shown, during the initial interaction, the country code (TH - Thailand) and rotation angle (0 degrees) were first confirmed, and the requirements for the output data fields were standardized (e.g., names must include the original language and English, the format of date of birth, place, gender, and ethnicity, spaces must be removed from the ID number, nationality must be displayed in two letters, addresses must include the original language address and the English translation, and the format of issuance date and expiration date), as well as special processing rules (e.g., date conversion, name cleanup, address formatting, and fixed fields). Then, the output format was strictly standardized to JSON format, all fields must be strings, and the output must have a single-layer structure. Finally, the actual processing results were given. To visually demonstrate the effectiveness of this method, an appendix is ​​attached. Figure 10 Comparison images of the recognition performance of several minority language problem samples before and after optimization are given. Before optimization, an open-source optical character recognition initial model was used, and after optimization, the document image recognition model of this invention was used.

[0160] The challenge of the training method for the document image recognition model provided in this invention lies not only in how to distill the knowledge of a large model into a small model, combining the high accuracy of the large model with the fast recognition speed of the small model, but also in how to enable a single model to handle complex scenarios involving multiple countries, multiple languages, and multiple document types, while ensuring that its recognition performance for both global attributes (such as layout and country category) and local attributes (such as specific text fields) of document images is practically applicable. In particular, it is crucial to prevent a seesaw effect during fine-tuning where the recognition performance of a single country increases while the recognition performance of other countries decreases (this problem arises because features are not mapped to countries). The training method of this invention employs a phased training and multi-task learning strategy, introducing reinforcement learning and reward functions for gradual fine-tuning. This guides the model to focus on key recognition areas. For example, the first step identifies global attributes such as country / form / type, emphasizing the learning of global features; the second step identifies text, emphasizing local features. Finally, the resulting document image recognition model supports image recognition and attribute parsing for multiple languages ​​and document types, and boasts advantages such as fast recognition speed, high accuracy, good stability, and lightweight design.

[0161] As a second aspect of the present invention, an electronic device is provided, such as... Figure 11 As shown, it includes:

[0162] One or more processors 101;

[0163] The memory 102 stores one or more computer programs that, when executed by the one or more processors 101, cause the one or more processors 101 to implement the training method provided according to the first aspect of the invention.

[0164] The tool may also include one or more I / O interfaces 103 connected between the processor 101 and the memory 102, configured to enable information interaction between the processor 101 and the memory 102.

[0165] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit 101 (CPU); the first memory 102 is a device with data storage capabilities, including but not limited to random access memory 102 (RAM, more specifically SDRAM, DDR, etc.), read-only memory 102 (ROM), electrically erasable programmable read-only memory 102 (EEPROM), and flash memory (FLASH); the I / O interface 103 (read-write interface) is connected between the processor 101 and the memory 102, enabling information interaction between the processor 101 and the memory 102, including but not limited to a data bus 104 (Bus).

[0166] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0167] Furthermore, as a third aspect of the present invention, a computer-readable medium is also provided having a computer program stored thereon, such as Figure 12 As shown, when the computer program is executed by the processor, it implements the training method provided by the first aspect of the present invention.

[0168] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. Accordingly, the computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can implement the methods of any of the above embodiments. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0169] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art should understand that the present invention includes, but is not limited to, the contents described in the accompanying drawings and the specific embodiments above. Any modifications that do not depart from the functional and structural principles of the present invention will be included within the scope of the claims.

Claims

1. A training method for a document image recognition model, characterized in that, The training method includes: Multiple document images are identified using a closed-source large model, and the recognition results for each document image are obtained. The recognition results for each document image include multiple key pieces of information. The identification results are used to tag the corresponding document images to obtain tagged document images; the tagged document images carry multiple tags associated with the key information; The initial open-source optical character recognition model was trained using tagged document images as a sample set to obtain the final document image recognition model, including: Labeled document images with tags that carry global attributes of associated document images and their values ​​are used as a sample set. An initial model is trained using a reinforcement learning algorithm so that the trained model parameters can learn the overall features of the document images. If the accuracy of global attribute recognition of document image is greater than or equal to the preset global attribute recognition accuracy, the trained model parameters are saved to obtain the global attribute recognition model of document image. Labeled document images containing fields and values ​​of related local attributes of the document image are used as a sample set. A global attribute recognition model for document images is trained using a reinforcement learning algorithm so that the trained model parameters can learn the local features of the document image. If the accuracy rate of local attribute recognition of the document image is greater than or equal to the preset local attribute recognition accuracy rate, the trained model parameters are saved to obtain the final document image recognition model.

2. The training method according to claim 1, characterized in that, The key information includes fields representing global attributes of the document image and their values; The recognition results are used to mark the corresponding document images to obtain marked images; The tagged document image carries tags associated with the key information; including: By using fields representing global attributes of the document image and their values, the corresponding document image is tagged to obtain a tagged document image carrying the fields and values ​​of the associated global attributes of the document image.

3. The training method according to claim 2, characterized in that, The key information also includes fields representing local attributes of the document image and their values; The recognition results are used to mark the corresponding document images to obtain marked images; The tagged document image carries a tag associated with the key information; include: By using fields representing local attributes of the document image and their values, the corresponding document image is labeled to obtain a labeled document image carrying the fields and values ​​of the associated local attributes of the document image.

4. The training method according to claim 1, characterized in that, The reinforcement learning algorithm includes a result reward value and a format reward value; The step of training an initial model using a reinforcement learning algorithm or training a global attribute recognition model for document images using a reinforcement learning algorithm includes: The total reward value for each training sample is determined based on the result reward value and the format reward value. The advantage value of each training session is determined based on the total reward value of the samples and the total number of samples in each training session. The loss function is determined based on the determined dominance value of the samples and the group relative strategy optimization, and the model parameters of the initial model or the global attribute recognition model of the document image are trained based on the loss function.

5. The training method according to claim 4, characterized in that, The training method also includes: Update the format reward value weight and the result reward value weight according to the fields and values ​​predicted by the model; If the model predicts fields that are not predefined or that are missing predefined fields, increase the weight of the format reward value. The weight of the result reward value is reduced based on the weight of the increased format reward value.

6. The training method according to any one of claims 1 to 3, characterized in that, The training method also includes: Data augmentation operations are performed on document images using a closed-source large model; wherein the data augmentation operations include at least one of image cropping, image rotation, image flipping, image illumination preprocessing, and modification of text information in the image.

7. The training method according to claim 1, characterized in that, The training method also includes: Multiple closed-source large models are used to recognize the document image, resulting in multiple recognition results. If a newline character is present in the recognition result, the newline character is changed to a display newline character. If a space character is detected to be adjacent to a newline character in the recognition result, the space character is deleted. The multiple recognition results are voted on to obtain the voting recognition result corresponding to the document image; The identification image is tagged using the recognition results after the voting, resulting in a tagged identification image. The initial open-source optical character recognition model was trained using the tagged document images as a sample set to obtain the final document image recognition model.

8. An electronic device, characterized in that, include: One or more processors; A memory having stored one or more computer programs that, when executed by one or more processors, cause the one or more processors to implement the training method according to any one of claims 1 to 7.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method according to any one of claims 1 to 7.

Images