Method for training optical character recognition model and system therefor

Abstract

A method and system for training an OCR (Optical Character Recognition) model are provided. A method for training an OCR model according to some embodiments may include the steps of obtaining a recognition result for a first text block of a document image and a confidence level for said recognition result using an OCR model; if said confidence level is below a threshold, providing at least a portion of the document image to a VLM (Vision-Language Model) to verify the recognition result; and determining a label for the first text block based on the result of the verification. According to this method, the human and time costs required to build a training set can be significantly reduced.

Description

Technology Field

[0001] The present disclosure relates to a technology that efficiently supports the training of an Optical Character Recognition (OCR) model. Background Technology

[0002] Optical Character Recognition (OCR) technology, which recognizes text from document images, is a foundational technology essential in various fields such as electronic document processing, administrative automation, and data analysis. Recently, there have been active attempts to improve recognition accuracy by applying deep learning models, and the performance of such models depends heavily on the quality of the datasets used for training. In other words, for OCR models to respond to various document types in real-world environments, it is necessary to construct large-scale training sets that are accurately labeled.

[0003] However, constructing such high-quality training sets requires manual labeling by humans, which entails enormous time and cost. For this reason, building large-scale, high-quality training sets for OCR tasks is practically very difficult, and there is a need for technical solutions that can generate highly accurate training sets while minimizing human intervention. Prior art literature

[0004] Korean Patent Publication No. 10-2025-0104686 (Published July 8, 2025) The problem to be solved

[0005] Various technical problems to be solved through some embodiments of the present disclosure relate to a method and system capable of efficiently supporting the training of an Optical Character Recognition (OCR) model.

[0006] Specifically, the technical problem to be solved through some embodiments of the present disclosure is to provide a method and a system capable of constructing a high-quality training set for an OCR model.

[0007] In addition, another technical problem to be solved through some embodiments of the present disclosure is to provide a method and system that can reduce the human and time costs required to build a training set for an OCR model.

[0008] In addition, another technical problem to be solved through some embodiments of the present disclosure is to provide a method and a system capable of effectively constructing a high-performance OCR model.

[0009] The technical problems of the present disclosure are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by a person skilled in the art of the present disclosure from the description below. means of solving the problem

[0010] A method for training an OCR model according to some embodiments of the present disclosure for solving the technical problem described above may include, in a method performed by at least one processor, the step of obtaining a recognition result for a first text block of a document image and a confidence level for said recognition result using an OCR (Optical Character Recognition) model; the step of verifying said recognition result by providing at least a portion of said document image to a VLM (Vision-Language Model) when said confidence level is less than a threshold value; and the step of determining a label for said first text block based on the result of said verification.

[0011] In some embodiments, the step of verifying the recognition result may include verifying the recognition result based on the document image and the question-answering result of the VLM based on the recognition result.

[0012] In some embodiments, the VLM may be trained based on a question-and-answer task for a plurality of document images.

[0013] In some embodiments, the step of verifying the recognition result may include: deriving a verification target answer based on the recognition result; generating a verification question corresponding to the verification target answer using the VLM or another VLM; inputting at least a portion of the document image and a prompt including the verification question into the VLM to generate a reference answer corresponding to the verification question; and comparing the verification target answer with the reference answer.

[0014] In some embodiments, the step of generating the verification question comprises a prompt including the answer to be verified and a set of instructions, and the step of generating the verification question by inputting at least a portion of the document image into the VLM or the other VLM, wherein the set of instructions may include instructions requesting the generation of a question in which the answer to be verified is the correct answer by referring to at least a portion of the document image.

[0015] In some embodiments, the step of generating the reference answer includes a prompt comprising the verification question and a set of instructions, and the step of generating the reference answer by inputting at least a portion of the document image into the VLM, wherein the set of instructions includes instructions requesting the generation of an answer corresponding to the verification question by referencing at least a portion of the document image, and the prompt may further include characteristic information of the answer to be verified.

[0016] In some embodiments, the answer to be verified may include noun-centered text or named entities derived from the recognition result.

[0017] In some embodiments, the step of verifying the recognition result may include requesting verification of the recognition result to the verifier's terminal when the test similarity between the answer to be verified and the reference answer is less than the threshold.

[0018] In some embodiments, the document image further includes a second text block in addition to the first text block, and the method may further include the step of combining the label of the first text block and the label of the second text block; and the step of building a training set for an OCR task by matching and storing the result of the combination with the document image.

[0019] In some embodiments, the method may further include the steps of: building a training set reflecting the label; updating the OCR model using the training set; and performing labeling for other document images using the updated OCR model.

[0020] In some embodiments, the document image is an image belonging to a target domain, and the method may further include the step of constructing a training set of the target domain by reflecting the label; and the step of fine-tuning the OCR model using the training set.

[0021] A system for training an OCR model according to some embodiments of the present disclosure for solving the technical problem described above comprises: one or more processors; and a memory for storing a computer program executed by the one or more processors, wherein the computer program may include instructions for: obtaining a recognition result for a first text block of a document image and a confidence level for said recognition result using an OCR (Optical Character Recognition) model; providing at least a portion of said document image to a VLM (Vision-Language Model) to verify said recognition result when said confidence level is below a threshold; and determining a label for said first text block based on the result of said verification.

[0022] A computer program according to some embodiments of the present disclosure for solving the technical problem described above may be stored in a non-transitory computer-readable recording medium to execute the steps of: obtaining a recognition result for a first text block of a document image and a confidence level for said recognition result using an Optical Character Recognition (OCR) model, and if said confidence level is below a threshold, providing at least a portion of said document image to a Vision-Language Model (VLM) to verify said recognition result; and determining a label for said first text block based on the result of said verification. Effects of the invention

[0023] According to some embodiments of the present disclosure, recognition results for text blocks of document images and reliability information regarding them are obtained using an Optical Character Recognition (OCR) model, and if the reliability is below a threshold, the recognition results can be verified by performing a Vision-Language Model (VLM)-based question-answering. The VLM can accurately verify the recognition results of the OCR model by utilizing OCR capabilities and document understanding capabilities, and as a result, the intervention of a reviewer can be minimized during the training set construction process. Accordingly, a high-quality training set for the OCR model (or OCR task) can be constructed, and at the same time, the cost required for constructing the training set can be significantly reduced.

[0024] In addition, by using a VLM specialized for document images and question-and-answer tasks, the accuracy of VLM-based verification can be further improved.

[0025] In addition, noun-centered text and named entities may be derived as answers subject to verification from the recognition results of the OCR model. In this case, since the difficulty of the question-answering performed through VLM is lowered, the accuracy of VLM-based verification can be further improved.

[0026] In addition, by repeatedly performing a series of processes leading to labeling using an OCR model, and updating the training set axes and the OCR model based on the labeling results, the quality of the training set and the performance of the OCR model can be effectively improved.

[0027] In addition, by constructing a training set of the target domain using a pretrained OCR model and fine-tuning the OCR model using it, an OCR model that can be utilized in the target domain can be effectively implemented.

[0028] The effects according to the technical concept of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below. Brief explanation of the drawing

[0029] FIG. 1 is an exemplary drawing for explaining the operation of a system for training an Optical Character Recognition (OCR) model according to some embodiments of the present disclosure at the system level. FIG. 2 is an exemplary drawing for further explaining the operation of a system for OCR model training according to some embodiments of the present disclosure. FIG. 3 is an exemplary flowchart illustrating a method for training an OCR model according to some embodiments of the present disclosure. FIG. 4 is an exemplary diagram for further explaining the recognition result and reliability information acquisition steps illustrated in FIG. 3. FIG. 5 is an exemplary flowchart illustrating a method for verifying recognition results based on a Vision-Language Model (VLM) according to some embodiments of the present disclosure. FIG. 6 is an exemplary drawing for explaining a method for deriving a verification target answer according to some embodiments of the present disclosure. FIG. 7 is an exemplary diagram to further explain the verification question generation step illustrated in FIG. 5. FIG. 8 is an exemplary diagram to further explain the reference answer generation step illustrated in FIG. 5. FIG. 9 is an exemplary drawing illustrating a case in which inspector-based verification is performed according to some embodiments of the present disclosure. FIG. 10 is an exemplary drawing illustrating an OCR model training process according to some embodiments of the present disclosure. FIG. 11 illustrates an exemplary computing device capable of implementing a system for OCR model training according to some embodiments of the present disclosure. Specific details for implementing the invention

[0030] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings. The advantages and features of the present disclosure and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the attached drawings. However, the technical concept of the present disclosure is not limited to the following embodiments but can be implemented in various different forms. The following embodiments are provided merely to complete the technical concept of the present disclosure and to fully inform those skilled in the art of the scope of the present disclosure, and the technical concept of the present disclosure is defined only by the scope of the claims.

[0031] In describing the various embodiments of the present disclosure, if it is determined that a detailed description of related known configurations or functions could obscure the essence of the present disclosure, such detailed description is omitted.

[0032] Unless otherwise defined, terms used in the following embodiments (including technical and scientific terms) may be used in a meaning commonly understood by those skilled in the art to which this disclosure pertains, but this may vary depending on the intent of those skilled in the art, case law, the emergence of new technology, etc. The terms used in this disclosure are for describing the embodiments and are not intended to limit the scope of this disclosure.

[0033] In the following embodiments, singular expressions include plural concepts unless the context clearly specifies them as singular. Additionally, plural expressions include singular concepts unless the context clearly specifies them as plural.

[0034] In addition, terms such as first, second, A, B, (a), (b), etc. used in the following embodiments are used merely to distinguish one component from another, and the essence, order, or sequence of the said component is not limited by such terms.

[0035] In the following embodiments, components described using terms such as ~part or unit, module, block, ~or, ~er, etc., and functional blocks illustrated in the drawings may be implemented in the form of software, hardware, or a combination thereof. Software may include, for example, machine code, firmware, embedded code (or software), application software, or a combination thereof. Additionally, hardware may include, for example, electrical circuits, electronic circuits, processors, computers, integrated circuits, integrated circuit cores, passive components, or a combination thereof. As more specific examples, such as ~part, module, etc., may include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0036] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0037] FIG. 1 is an exemplary drawing for explaining the operation of a system (10) for training an OCR (Optical Character Recognition) model according to some embodiments of the present disclosure at the system level. In FIG. 1, the system (10) is depicted as a 'training system,' and for clarity of the present disclosure, the system (10) will also be referred to as a 'training system' below.

[0038] Referring to FIG. 1, the training system (10) is a computing system capable of supporting efficient training of an OCR model (12). For example, the training system (10) can build a training set for an OCR model (12) (or OCR task) by performing labeling on multiple document images (11) while minimizing the intervention of a reviewer, and can implement a high-performance OCR model (12) through a training (or update) process using the same.

[0039] Document images may encompass, without limitation, various forms of images in which text is visually arranged. For example, the scope of document images may include not only images obtained from paper or electronic documents, but also images obtained from various materials containing text, such as receipts, identification cards, and web pages. Furthermore, document images may be generated in various ways, such as scanning paper documents, taking photographs, digital conversion (e.g., converting digital documents in text format into images), or screen capturing, but the scope of the present disclosure is not limited thereto.

[0040] These document images may, depending on the case, be named as 'text images', 'text-containing images', etc.

[0041] Hereinafter, the operation of the training system (10) will be explained in more detail with reference to FIG. 2.

[0042] FIG. 2 is an exemplary drawing for further explaining the operation of a training system (10) according to some embodiments of the present disclosure.

[0043] As illustrated in FIG. 2, the training system (10) can acquire a document image (22) from a document image DB (21) and output a recognition result (23) for each text block (e.g., recognition unit such as a continuously arranged text section, part and / or area, etc.) included in the document image through an OCR model (12). At this time, the training system (10) can also acquire information on the reliability of the OCR model (12) for the recognition result (23) (e.g., recognition reliability per text block).

[0044] The OCR model (12) may be a pre-trained neural network-based model. The OCR model (12) may be in a state where only basic training has been performed, or it may be in a state where additional training (e.g., subsequent training, fine-tuning, etc.) has been performed. Additionally, the OCR model (12) may be a model trained using, for example, document images of various domains, but the scope of the present disclosure is not limited thereto.

[0045] The structure, input, and output of the OCR model (12) can be designed and configured in various ways. For example, the OCR model (12) may be configured to output bounding box information, recognition results, and / or recognition reliability by detecting each text block in an input document image and recognizing the text within each block. However, the scope of the present disclosure is not limited thereto.

[0046] Next, the training system (10) determines whether verification is necessary based on the reliability of the recognition result (23) (see 24), and if verification is not necessary, it can match the document image (22) with the recognition result (23) and store them in the training DB (25) (i.e., the recognition result (23) is set as the label of the document image (22)). Here, the training DB (25) can be understood as a storage where a training set for an OCR task is stored.

[0047] On the other hand, if verification is required, the training system (10) first performs automatic verification using a Vision-Language Model (VLM) (see 26) and can support manual verification by an inspector as needed (see 27, 28). In this case, the human and time costs required to build the training set can be effectively reduced by minimizing the intervention of the inspector through the VLM-based automatic verification process. The training system (10) can accurately verify the recognition result (23) by performing question answering based on the document image (22) through the VLM, which will be explained in detail later with reference to the drawings from Fig. 3 onwards.

[0048] Next, the training system (10) can determine the label of the document image (22) by reflecting the verification result, and match the determined label to the document image (22) and store it in the training DB (25).

[0049] VLM refers to a model (e.g., a pretrained model) equipped with the ability to understand natural language and images, and can be designed and implemented in various ways. For example, a VLM may be a model specialized for document images, trained based on question-answering tasks regarding various document images, but the scope of this disclosure is not limited thereto.

[0050] In some cases, VLM may be a model that undergoes additional fine-tuning (e.g., tuning based on question-answering tasks) using document images specialized for the target domain in addition to pre-training.

[0051] Meanwhile, the training system (10) can update the OCR model (12) (e.g., additional training) using the dataset (i.e., training set) accumulated in the training DB (25), and can repeatedly perform the process described above on other document images using the updated OCR model (12). By doing so, the quality of the training set and the performance of the OCR model (12) can be gradually improved.

[0052] The above-described training system (10) may be implemented with at least one computing device. For example, all functions of the training system (10) may be implemented in a single computing device, or a first function of the training system (10) may be implemented in a first computing device and a second function may be implemented in a second computing device. Alternatively, a specific function of the training system (10) may be implemented in multiple computing devices.

[0053] A computing device may include any device equipped with computing functions, and for an example of such a device, refer to FIG. 11. Since a computing device is a collection of various components (e.g., memory, processor, etc.) that interact, it may be referred to as a 'computing system' depending on the case. Of course, the term computing system may also encompass the concept of a collection of multiple computing devices that interact.

[0054] Referring again to Fig. 1, the explanation will be provided.

[0055] The OCR system (10) can recognize text in a document image (14) using a trained OCR model (12) (see 15). Here, the OCR system (13) may be the same system as the training system (10) or may be a separate system distinct from each other.

[0056] The OCR model (12) and / or OCR system (13) can be utilized in various application fields.

[0057] For example, the OCR model (12) can be used to recognize specific text (e.g., personal information such as name, date of birth, number, etc.) from an identification card image (e.g., national ID card, passport, driver's license, etc.). The recognized text can be used as base information to automate, for example, user identity verification procedures, account opening procedures, and / or real-name verification procedures. In such examples, the OCR system (13) can be linked with or integrated with a system responsible for identity verification procedures to realize the automation of said procedures.

[0058] As another example, the OCR model (12) can be used to recognize specific text (e.g., information on key items such as amount, date, item name, etc.) in images of structured or semi-structured documents such as invoices, receipts, and tax invoices. The recognized text can be used to support document processing automation related to accounting processing, expense processing, etc.

[0059] As another example, the OCR model (12) can be used to recognize specific text (e.g., clauses, condition-related phrases, etc.) in images of natural language-based documents such as contracts, insurance documents, etc. The recognized text can be used to support document content analysis automation, document classification, and information retrieval efficiency.

[0060] Up to now, the operation of a training system (10) according to some embodiments of the present disclosure has been schematically described with reference to FIGS. 1 and 2. Hereinafter, various methods for training an OCR model will be described with reference to FIGS. 3 and subsequent drawings.

[0061] The methods described below may be performed by at least one processor (hereinafter abbreviated as 'processor') or implemented in said processor. For example, at least some of the steps / operations of the methods described below may be performed by a processor equipped with the training system (10) described above.

[0062] For the sake of ease of understanding, the following description assumes that all steps / operations of the methods described below are performed by a processor equipped with a specific computing system (or device). Therefore, if the subject of a specific step / operation is omitted, it can be understood that the corresponding step / operation is performed by the processor of that computing system. However, in actual environments, some steps / operations of the methods described below may be performed on a different computing system.

[0063] FIG. 3 is an exemplary flowchart illustrating a method for training an OCR model according to some embodiments of the present disclosure. However, this is merely an exemplary embodiment for achieving the purpose of the present disclosure, and it is understood that some steps may be added or deleted as necessary.

[0064] As illustrated in FIG. 3, the method according to the embodiments may begin at step S31 of acquiring a document image. For example, the training system (10) may acquire a document image from a pre-prepared document image DB (21). In some cases, the training system (10) may acquire a document image by performing image conversion on a text-format document.

[0065] In some embodiments, the document images may belong to a target domain (i.e., a domain to which the OCR model is primarily applied). For example, the training system (10) may collect document images belonging to the target domain (e.g., identification card images, receipt images, documents in a specific field, etc.) to build a training set (or OCR model (12)) of the target domain. Any method may be used to collect these document images.

[0066] In step S32, recognition results for text blocks in a document image and reliability information for said recognition results are obtained using an OCR model. If multiple text blocks exist in the document image, recognition results and reliability information can be obtained for each text block.

[0067] For example, as illustrated in FIG. 4, let us assume that a plurality of text blocks (e.g., 42, 43) exist in a document image (41). In this case, the training system (10) can input the document image (41) into an OCR model (12) to obtain a recognition result for each text block (e.g., 42). As described above, the OCR model (12) may be configured to detect each of the text blocks (e.g., 42, 43) in the form of a bounding box and to recognize the text contained within the bounding box (e.g., 44, 45), but the scope of the present disclosure is not limited thereto. The training system (10) can obtain reliability information regarding the recognition result along with the recognition result of each text block (e.g., 42).

[0068] In step S33, it is determined whether the confidence level is below a threshold. If it is below the threshold, the recognition result is designated as a verification target and step S34 is performed, and if it is above the threshold, step S37 may be performed. If multiple text blocks exist in the document image, each recognition result of the text blocks with a confidence level below the threshold may be designated as a verification target.

[0069] The threshold value may be a pre-set fixed value or a value that varies depending on the situation. For example, the threshold value may be adjusted based on the performance of the OCR model (12). For instance, if the performance of the OCR model (12) is low (e.g., early in training), the threshold value may be raised as it is necessary to judge reliability more conservatively. Conversely, if the performance of the OCR model (12) is high (e.g., mid-to-late training), the threshold value may be lowered as the performance of the OCR model (12) improves.

[0070] In step S34, the recognition result of the corresponding text block is automatically verified by providing the document image to the VLM. This can be understood as a process of automatically verifying the recognition result by utilizing the VLM's OCR capabilities and document understanding capabilities. However, the specific verification method may vary depending on the embodiment.

[0071] In some embodiments, the recognition result can be verified by performing a question-and-answer based on the document image through the VLM. This will be explained in detail shortly with reference to FIGS. 5 to 9, etc.

[0072] In some other embodiments, the recognition result of the OCR model can be verified through the recognition result of the VLM for the document image. For example, the training system (10) may construct a prompt based on instructions requesting recognition of a corresponding text block by referring to a document image, and may input the prompt and at least a part of the document image (e.g., an image containing the corresponding text block) into the VLM (e.g., 70 in FIG. 7) to obtain the recognition result of the corresponding text block. Then, the training system (10) may perform automatic verification by comparing the obtained recognition result with the recognition result of the OCR model (12). In some cases, the training system (10) may construct a prompt based further on the recognition result of the OCR model (12) and instructions requesting verification by comparing it with the recognition result of the VLM.

[0073] In some other embodiments, the recognition result may be verified based on various combinations of the embodiments described above. For example, the training system (10) may perform verification in the manner according to the first embodiment (e.g., a question-and-answer-based embodiment) when the reliability is less than a first threshold, and may perform verification by using the method according to the first embodiment (e.g., a question-and-answer-related embodiment) and the method according to the second embodiment (i.e., an embodiment different from the first embodiment) together when the reliability is less than a second threshold (wherein the second threshold is a value smaller than the first threshold).

[0074] Hereinafter, a method for verifying VLM-based recognition results according to several embodiments of the present disclosure will be described in more detail with reference to FIGS. 5 to 9.

[0075] FIG. 5 is an exemplary flowchart illustrating a VLM-based recognition result verification method according to some embodiments of the present disclosure. However, this is merely an exemplary embodiment for achieving the purpose of the present disclosure, and it is understood that some steps may be added or deleted as necessary.

[0076] As illustrated in FIG. 5, first, a verification target answer is derived based on the recognition result of the OCR model (S51). Here, the verification target answer refers to a text (or phrase) used as an answer in the process of verifying the recognition result through question and answer. The number of verification target answers may be one or multiple, and the specific method of derivation may be designed in various ways.

[0077] For example, as illustrated in FIG. 6, noun-centered text (64, e.g., single noun, compound noun, noun phrase, etc.) may be designated as the answer to be verified in the recognition result (63) for the text block (62) of the document image (61). Such noun-centered text can improve the accuracy of VLM-based verification by lowering the difficulty of the question and answer and ensuring clarity. Noun-centered text (64) may be derived by applying natural language processing techniques, such as part-of-speech analysis, to the recognition result (63), but the scope of the present disclosure is not limited thereto. FIG. 6 illustrates an example in which a 'noun phrase' is designated as the answer to be verified.

[0078] As another example, verb-centered text (e.g., single verbs, verb phrases, etc.) in the recognition result can be designated as the answer to be verified.

[0079] As another example, a named entity in the recognition result may be designated as the answer to be verified. For instance, the training system (10) may use a named entity recognition model to detect one or more named entities and designate an answer to be verified among them.

[0080] As another example, an appropriate answer to be verified can be derived from the recognition result through a Large Language Model (LM). For instance, a training system (10) can construct a prompt based on the recognition result and instructions requesting the selection of text (e.g., main keywords, main phrases, etc.) that can be used as an answer in the question-and-answer process from the recognition result, and input the prompt into the LLM to derive an answer to be verified. In this case, text suitable for use as an answer can be accurately selected from the recognition result by an LLM equipped with natural language understanding capabilities.

[0081] As another example, text containing the relevant recognition result (e.g., noun-centered text, etc.) can be derived as the answer to be verified.

[0082] As another example, the answer to be verified may be derived based on various combinations of the examples described above.

[0083] In some embodiments, the number of answers to be verified derived may be determined based on the reliability of the recognition result. For example, the training system (10) may increase the number of answers to be verified derived as the reliability of the OCR model (12) for the recognition result decreases. In this case, the verification intensity may be increased as the reliability decreases, and as a result, the accuracy of the verification process may be improved.

[0084] In step S52, a verification question corresponding to the answer to be verified is generated based on the document image using the VLM. For example, as illustrated in FIG. 7, the training system (10) may generate a verification question (77) by configuring a prompt (72) containing the answer to be verified (74) and a set of instructions (75), and inputting at least a part of the document image (71) (e.g., the entire image or a partial image containing a related text block, etc.) and the corresponding prompt (72) into the VLM (70). That is, the training system (10) may obtain the verification question (77) from the output (76) of the VLM (70).

[0085] The instruction set (75) may include instructions that request the generation of a question with the answer to be verified (74) as the correct answer by referring, for example, to an input document image (71). Such instructions may induce the VLM (70) to understand the content of the input document image (71) and accurately generate a question corresponding to the answer to be verified (74).

[0086] In some cases, the set of instructions (75) may further include instructions requesting an explanation of why the generated verification question (77) corresponds to the answer to be verified (74). In this case, the output (76) of the VLM (70) may further include an explanation of the reason for the correspondence.

[0087] Meanwhile, the VLM (70) may generate one verification question (77) from one verification target answer (74), or generate multiple verification questions. Alternatively, the VLM (70) may generate multiple verification questions from multiple verification target answers (e.g., one-to-one correspondence, one-to-many correspondence, etc.). This generation method can be controlled by the aforementioned set of instructions (75).

[0088] As described above, VLM (70) may be a model that has been pre-trained (or trained) through a question-and-answer task on various document images. In this case, the question-and-answer-based verification process for the recognition result can be performed more accurately. Additionally, VLM (70) may be a model that has additionally undergone fine-tuning using document images of the target domain (e.g., question-and-answer task-based fine-tuning) in addition to pre-training, but the scope of the present disclosure is not limited thereto.

[0089] In step S53, a reference answer for a verification question is generated based on a document image using a VLM. For example, as illustrated in FIG. 8, a training system (10) may generate a reference answer (84) for a verification question (77) by configuring a prompt (81) containing a verification question (77) and a set of instructions (82), and inputting at least a portion of the document image (71) and the corresponding prompt (81) into a VLM (70). That is, the training system (10) may obtain the reference answer (84) from the output (83) of the VLM (70). FIG. 8 illustrates a case where a reference answer (84) for a verification question (77) is generated following FIG. 7.

[0090] The instruction set (82) may include instructions that request the generation of an answer corresponding to a verification question (77) by referring, for example, to an input document image (71). These instructions may induce the VLM (70) to understand the content of the input document image (71) and accurately generate an answer to the verification question (77).

[0091] In some cases, the set of instructions (82) may further include instructions requesting an explanation of why the generated reference answer (84) is appropriate as the answer (or correct answer) to the verification question (77). In this case, the output (83) of the VLM (70) may further include an explanation of the reason.

[0092] Additionally, the prompt (81, e.g., set of instructions (82), etc.) may further include characteristic information of the answer to be verified (74). Here, the characteristic information may include, for example, the part of speech (e.g., noun, etc.), form (e.g., short answer, noun phrase, verb phrase, etc.), and related description (e.g., "it is a named entity of a specific category," "it is related to specific content of the document image (71)," etc.) of the answer to be verified (74), but is not limited thereto. This characteristic information can further improve the accuracy of VLM-based verification by acting as a hint that lowers the difficulty of the answer generation process.

[0093] For reference, FIG. 8 illustrates an example in which a reference answer (84) is generated using the same VLM (70) as FIG. 7, but the scope of the present disclosure is not limited thereto. The reference answer (84) and the verification question (77) may each be generated by different VLMs.

[0094] In step S54, verification of the recognition result is performed by comparing the answer to be verified with the reference answer. For example, as illustrated in FIG. 9, the training system (10) can calculate the text similarity between the answer to be verified (74) and the reference answer (84) and compare it with a threshold (see 91). The text similarity may be calculated based on an edit distance, such as the Levenshtein distance, for example, but the scope of the present disclosure is not limited thereto. If, as a result of the comparison, the text similarity is less than the threshold, additional verification may be performed through the verifier (90).

[0095] Up to this point, a method for verifying recognition results based on a VLM according to several embodiments of the present disclosure has been described with reference to FIGS. 5 through 9. As described above, by using a VLM (e.g., 70) equipped with OCR capabilities and document understanding capabilities, the recognition results of an OCR model can be accurately verified based on question-and-answer. Furthermore, if a VLM specialized for document images and question-and-answer tasks is used, the verification accuracy can be further improved.

[0096] Referring again to Fig. 3, the explanation will be provided.

[0097] In step S35, it is determined whether manual verification is required. That is, it is determined whether the VLM-based automatic verification is insufficient or has failed. If it is determined that manual verification is required, step S36 is performed, and if the opposite is true, step S37 may be performed.

[0098] For example, referring again to FIG. 9, the training system (10) may determine that verification by a reviewer (90) is required if the text similarity between the answer to be verified (74) and the reference answer (84) is below a threshold.

[0099] The threshold may be a preset fixed value or a value that varies depending on the situation. For example, the threshold may be adjusted based on the performance of the OCR model (12) and / or the reliability of the OCR model (12) regarding the recognition results. For instance, if the performance of the OCR model (12) or the reliability of the recognition is relatively low, the threshold may be raised for more thorough verification. Conversely, the threshold may be lowered.

[0100] Meanwhile, in some embodiments, the verification strength may be adjusted based on the performance and / or recognition reliability of the OCR model. For example, the training system (10) may increase the verification strength as the performance and / or recognition reliability of the OCR model (12) is relatively low. The increase in verification strength may be implemented, for example, by increasing the number of question-answers or increasing the threshold, but the scope of the present disclosure is not limited thereto.

[0101] In step S36, verification of the recognition result is performed through an inspector. For example, the training system (10) may request verification of the recognition result by transmitting data required for verification (e.g., document image, recognition result of the OCR model (12), etc.) to the inspector's (90) terminal. Then, the inspector (90) can check the received data and perform verification through a verification UI (User Interface) displayed on the terminal. For instance, the inspector (90) can check whether the recognition result of the OCR model (12) is correct and, if there are parts that are misrecognized, correct them. The verification result (e.g., confirmation result, correction result, etc.) can be transmitted back to the training system (10) through the terminal.

[0102] In step S37, the label for the corresponding text block is determined. However, the specific method of determination may vary depending on the case.

[0103] For example, if the reliability of the recognition result is above a threshold, the recognition result of the OCR model (12) can be determined as the label of the corresponding text block.

[0104] As another example, if verification based on an inspector (90) is performed, the label of the text block can be determined by reflecting the verification result (e.g., the recognition result modified by the inspector (90) is set as the label).

[0105] As another example, if the answer to be verified and the reference answer match, the recognition result of the OCR model (12) can be determined as the label of the corresponding text block.

[0106] As another example, if the text similarity between the answer to be verified and the reference answer is greater than a threshold but the two answers do not match, the recognition result of the OCR model (12) may be determined as the label of the text block, or the result in which the reference answer is reflected in the recognition result may be determined as the label. For instance, the training system (10) may replace the answer to be verified in the recognition result with the reference answer and determine the result as the label of the text block. Alternatively, the training system (10) may use an LLM to generate text in which the reference answer is naturally reflected in the recognition result and determine this as the label of the text block. Alternatively, the training system (10) may input at least a portion of the document image, the recognition result, and the reference answer into a VLM (e.g., 70) to generate text in which the reference answer is reflected and determine this as the label of the text block.

[0107] As another example, the label of the text block may be determined based on various combinations of the examples described above.

[0108] In step S38, a training set for an OCR task is constructed by reflecting the labels. For example, the training system (10) can match the corresponding labels to at least a portion of the document images and store them in the training DB (25) (i.e., added to the training set). The training system (10) can construct a high-quality training set for an OCR task by repeating this process for other document images (or other text blocks).

[0109] If multiple text blocks exist in a document image, the labels of each text block are aggregated, and the aggregated result is matched to the corresponding document image and added to the training set. For example, let us assume that a first text block and a second text block exist within a document image. In this case, the training system (10) aggregates the label of the first text block and the label of the second text block, and the aggregated result is matched to the corresponding document image and stored in the training DB (25).

[0110] The aforementioned training set can be utilized in various ways.

[0111] For example, as illustrated in FIG. 10, the training system (10) can update (i.e., additional training) the OCR model (12) using the corresponding training set. Then, the training system (10) can perform labeling on other document images (e.g., 101) using the updated OCR model (12) (see steps S31 to S37) and store the labeling results in the training DB (25) (e.g., the labeling results may be added to the existing training set or used as a new training set). During this labeling process, the training system (10) can verify the recognition results of the OCR model (12) based on the VLM (70) and, if necessary, request verification from the terminal of the inspector (90). A series of processes including labeling using the OCR model (12), building a training set based on the labeling results, and updating the OCR model (12) can be performed repeatedly, and through this repetition, the quality of the training set and the performance of the OCR model (12) can be gradually improved.

[0112] As another example, the training system (10) can build (or train) a new OCR model using the training set. That is, the training system (10) can build a new OCR model that is distinct from the existing OCR model (12) using the training set (e.g., training an OCR model in an initialized state, or performing additional training on some trained OCR models).

[0113] As another example, the training system (10) may build a training set of the target domain and use it to fine-tune the OCR model (12). In this example, the OCR model (12) may be a model pre-trained using document images of various domains, but the scope of the disclosure is not limited thereto.

[0114] As another example, a training set may be utilized based on various combinations of the examples described above. For example, referring again to FIG. 10, the training system (10) may use an OCR model (12) to perform labeling on document images (e.g., 101) obtained from a document image DB (21) of a target domain, and may fine-tune the OCR model (12) using the results of such labeling.

[0115] Up to this point, a method for training an OCR model according to several embodiments of the present disclosure has been described with reference to FIGS. 3 through 9. As described above, a recognition result for a text block of a document image and reliability information thereof are obtained using an OCR model, and if the reliability is below a threshold, the recognition result can be verified by performing a VLM-based question-and-answer session. The VLM can accurately verify the recognition result of the OCR model by utilizing OCR capabilities and document comprehension capabilities, and as a result, the intervention of a reviewer can be minimized during the training set construction process. Accordingly, a high-quality training set for the OCR model (or OCR task) can be constructed, and at the same time, the cost required for constructing the training set can be significantly reduced.

[0116] In addition, by using a VLM specialized for document images and question-and-answer tasks, the accuracy of VLM-based verification can be further improved.

[0117] In addition, noun-centered text and named entities may be derived as answers subject to verification from the recognition results of the OCR model. In this case, since the difficulty of the question-answering performed through VLM is lowered, the accuracy of VLM-based verification can be further improved.

[0118] In addition, by repeatedly performing a series of processes leading to labeling using an OCR model, and updating the training set axes and the OCR model based on the labeling results, the quality of the training set and the performance of the OCR model can be effectively improved.

[0119] In addition, by constructing a training set of the target domain using a pre-trained OCR model and fine-tuning the OCR model using it, an OCR model that can be utilized in the target domain can be effectively implemented.

[0120] Hereinafter, with reference to FIG. 11, an exemplary computing device (110) capable of implementing the training system (10) described above will be described.

[0121] FIG. 11 is an exemplary hardware configuration diagram showing a computing device (110).

[0122] As illustrated in FIG. 11, a computing device (110) may include one or more processors (111), a bus (113), a communication interface (114), a memory (112) for loading a computer program (116) executed by the processor (111), and a storage (115) for storing the computer program (116). However, FIG. 11 illustrates only the components related to the embodiments of the present disclosure. Therefore, a person skilled in the art to which the present disclosure belongs will understand that other general-purpose components may be included in addition to the components (111 to 116) illustrated in FIG. 11. That is, the computing device (110) may include various additional components in addition to the components (111 to 116) illustrated in FIG. 11. Furthermore, depending on the case, the computing device (110) may be configured in a form in which some of the components (111 to 116) illustrated in FIG. 11 are omitted. Below, each component of the computing device (110) is described.

[0123] The processor (111) can control the overall operation of each component of the computing device (110). The processor (111) may be configured to include at least one of a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), GPU (Graphic Processing Unit), NPU (Neural Processing Unit), TPU (Tensor Processing Unit), VPU (Vision Processing Unit), APU (Accelerated Processing Unit), or any other type of processor well known in the art of the present disclosure. Additionally, the processor (111) may perform operations for at least one application or program to execute specific operations / steps / methods. The computing device (110) may have one or more processors.

[0124] Next, the memory (112) may store various data, commands and / or information. The memory (112) may load a computer program (116) from storage (115) to execute specific operations / steps / methods. The memory (112) may be implemented as volatile memory such as RAM, but the technical scope of the present disclosure is not limited thereto.

[0125] The bus (113) can provide communication functions between components of the computing device (110). The bus (113) can be implemented as various types of buses, such as an address bus, a data bus, and a control bus.

[0126] The communication interface (114) can support wired and wireless internet communication of the computing device (110). Additionally, the communication interface (114) may support various communication methods other than internet communication. To this end, the communication interface (114) may be configured to include a communication module well known in the art of the present disclosure.

[0127] Storage (115) may store one or more computer programs (116) non-temporarily. Storage (115) may be configured to include non-volatile memory such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, a hard disk, a removable disk, or any form of computer-readable recording medium well known in the art to which this disclosure belongs.

[0128] A computer program (116) may include instructions that cause a processor (111) to perform specific operations / steps / methods when loaded into memory (112). That is, the processor (111) can perform specific operations / steps / methods by executing the loaded instructions.

[0129] For example, a computer program (116) may include instructions to perform operations such as obtaining a recognition result for a first text block of a document image and a confidence level for said recognition result using an OCR model (e.g., 12), providing at least a portion of the document image to a VLM (e.g., 70) to verify the recognition result when the confidence level is below a threshold, and determining a label for the first text block based on the result of the verification.

[0130] As another example, a computer program (116) may include instructions to perform at least some of the operations / steps / methods described with reference to FIGS. 1 to 10.

[0131] As illustrated, a training system (10) according to some embodiments of the present disclosure can be implemented through a computing device (110).

[0132] Meanwhile, in some embodiments, the computing device (110) illustrated in FIG. 11 may refer to a virtual machine implemented based on cloud technology. For example, the computing device (110) may be a virtual machine running on one or more physical servers included in a server farm. In this case, at least some of the processor (111), memory (112), and storage (115) illustrated in FIG. 11 may be virtual hardware, and the communication interface (114) may also be implemented as a virtualized networking element such as a virtual switch.

[0133] Up to now, with reference to FIG. 11, an exemplary computing device (110) capable of implementing a training system (10) according to some embodiments of the present disclosure has been described.

[0134] Various embodiments of the present disclosure and effects according to those embodiments have been described with reference to FIGS. 1 to 11. The effects according to the technical concept of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below.

[0135] Furthermore, just because the above embodiments describe a plurality of components being combined into one or operating in combination, the technical concept of the present disclosure is not necessarily limited to these embodiments. That is, within the scope of the purpose of the technical concept of the present disclosure, all such components may be selectively combined into one or more combinations to operate.

[0136] The technical concept of the present disclosure described above may be implemented as computer-readable code on a computer-readable recording medium. A computer program recorded on a computer-readable recording medium may be transmitted to another computing device via a network such as the Internet and installed on said computing device, thereby being used on said computing device.

[0137] Although operations are depicted in a specific order in the drawings, it should not be understood that the operations must necessarily be executed in the specific order depicted or in a sequential order, or that all depicted operations must be executed to obtain the desired result. In certain situations, multitasking and parallel processing may be advantageous. Although various embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art will understand that the technical concept of the present disclosure may be implemented in other specific forms without altering the technical concept or essential features thereof. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of protection of the present disclosure shall be interpreted by the claims below, and all technical concepts within the equivalent scope shall be interpreted as being included within the scope of rights of the technical concept defined by the present disclosure. Explanation of the symbols

[0138] 10: Training System 12: OCR (Optical Character Recognition) Model 13: OCR System 21: Document Image DB (database) 25: Training DB 70: Vision-Language Model (VLM) 90: Inspector 110: Computing device 111: Processor 112: Memory 113: Bus 114: Communication Interface 115: Storage 116: Computer Program

Claims

Claim 1 A method for training an OCR model, comprising: a step of obtaining a recognition result for a first text block of a document image and a confidence level for said recognition result using an OCR (Optical Character Recognition) model; a step of verifying said recognition result based on at least a portion of said document image and a question-and-answer result of a VLM (Vision-Language Model) based on said recognition result when said confidence level is less than a threshold value; and a step of determining a label for said first text block based on said verification result, wherein the step of verifying said recognition result comprises: a step of deriving a verification target answer based on said recognition result; a step of generating a verification question corresponding to said verification target answer; a step of inputting a prompt including said verification question and at least a portion of said document image into said VLM to generate a reference answer corresponding to said verification question; and a step of comparing said verification target answer with said reference answer. Claim 2 delete Claim 3 A method for training an OCR model according to claim 1, wherein the VLM is trained based on a question-and-answer task for a plurality of document images. Claim 4 A method for training an OCR model, wherein, in claim 1, the step of generating the verification question comprises the step of generating the verification question using the VLM or another VLM. Claim 5 A method for training an OCR model according to claim 1, wherein the step of generating a verification question comprises a prompt including a set of instructions and a verification target answer, and the step of generating the verification question by inputting at least a portion of the document image into the VLM or another VLM, and the set of instructions includes instructions requesting the generation of a question in which the verification target answer is the correct answer by referring to at least a portion of the document image. Claim 6 A method for training an OCR model according to claim 1, wherein the step of generating the reference answer comprises a prompt including the verification question and a set of instructions, and the step of generating the reference answer by inputting at least a portion of the document image into the VLM, wherein the set of instructions includes instructions requesting the generation of an answer corresponding to the verification question by referencing at least a portion of the document image, and the prompt further includes characteristic information of the answer to be verified. Claim 7 A method for training an OCR model according to claim 1, wherein the answer to be verified includes noun-centered text or named entities derived from the recognition result. Claim 8 A method for OCR model training according to claim 1, wherein the step of verifying the recognition result further includes the step of requesting verification of the recognition result to a verifier's terminal when the text similarity between the answer to be verified and the reference answer is less than a threshold. Claim 9 A method for training an OCR model according to claim 1, wherein the document image further comprises a second text block in addition to the first text block, and further comprises the step of combining the label of the first text block and the label of the second text block; and the step of constructing a training set for an OCR task by matching and storing the result of the combination with the document image. Claim 10 A method for training an OCR model according to claim 1, further comprising: a step of constructing a training set reflecting the above labels; a step of updating the OCR model using the above training set; and a step of performing labeling on other document images using the updated OCR model. Claim 11 A method for training an OCR model according to claim 1, further comprising: a step of constructing a training set of the target domain by reflecting the label, wherein the document image is an image belonging to the target domain; and a step of fine-tuning the OCR model using the training set. Claim 12 A system for OCR model training comprising: one or more processors; and a memory for storing a computer program executed by the one or more processors, wherein the computer program comprises: an operation to obtain a recognition result for a first text block of a document image and a confidence level for said recognition result using an OCR (Optical Character Recognition) model; an operation to verify said recognition result based on at least a portion of said document image and a question-and-answer result of a VLM (Vision-Language Model) based on said recognition result when said confidence level is less than a threshold; and instructions for an operation to determine a label for said first text block based on said verification result, wherein the operation to verify said recognition result comprises: an operation to derive a verification target answer based on said recognition result; an operation to generate a verification question corresponding to said verification target answer; an operation to input a prompt including said verification question and at least a portion of said document image into said VLM to generate a reference answer corresponding to said verification question; and an operation to compare said verification target answer with said reference answer. Claim 13 delete Claim 14 A system for OCR model training, wherein, in claim 12, the operation of generating the verification question includes the operation of generating the verification question using the VLM or another VLM. Claim 15 A system for training an OCR model, wherein, in paragraph 12, the computer program further comprises instructions for: constructing a training set reflecting the label; updating the OCR model using the training set; and performing labeling for other document images using the updated OCR model. Claim 16 A computer program that is combined with a computer processor and is stored in a non-transitory computer-readable recording medium to execute the steps of: obtaining a recognition result for a first text block of a document image and a confidence level for said recognition result using an OCR (Optical Character Recognition) model; verifying said recognition result based on at least a portion of said document image and a question-and-answer result of a VLM (Vision-Language Model) based on said recognition result when said confidence level is below a threshold; and determining a label for said first text block based on said verification result, wherein the step of verifying said recognition result comprises: deriving a verification target answer based on said recognition result; generating a verification question corresponding to said verification target answer; inputting a prompt including said verification question and at least a portion of said document image into said VLM to generate a reference answer corresponding to said verification question; and comparing said verification target answer with said reference answer.

Images