Optical character recognition method and related product
By combining a graph-based text model driven by multiple training datasets with a large language model, the problem of insufficient adaptability of optical character recognition models is solved, and high-quality conversion from unformatted files to formatted files is achieved, improving the model's versatility and accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ABC FINANCIAL TECH CO LTD
- Filing Date
- 2025-06-17
- Publication Date
- 2026-08-04
AI Technical Summary
Existing optical character recognition technology models are not universal and require different data to be collected for training in different scenarios, resulting in insufficient model adaptability and generalization ability.
We employ a graph-to-text model training method driven by multiple training datasets, combining a validation model and a large language model. Through multiple iterations of training and validation, we improve the model's versatility and accuracy.
It enables intelligent conversion from unformatted to formatted files, improving the model's adaptability and output quality under diverse inputs, ensuring that the generated files conform to standards in both structure and semantics, and enhancing the practicality and usability of the documents.
Smart Images
Figure CN120708236B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of optical character recognition technology, and in particular to an optical character recognition method and related products. Background Technology
[0002] With the rapid development of information technology, enterprises have accumulated a large amount of information, including a large number of paper documents, PDF files, pictures, tables and other unformatted files. In most cases, if enterprises want to use these files, they first need to convert the unformatted files into editable formatted files, and then carry out subsequent data archiving, data structuring, data analysis and other operations.
[0003] Currently, conventional optical character recognition (OCR) typically involves collecting a large amount of data based on specific scenarios. This data is then used to train models built using deep learning or machine learning algorithms, enabling the models to detect, recognize, and restore text layouts, converting uneditable, unformatted text into editable, formatted text.
[0004] However, conventional OCR currently has several problems, such as the lack of universality of models. Different scenarios require the collection of different data and the training of different models. For example, for PDF files, if the PDF file is entirely text, the model needs to be trained using a text training set of the same type; if the PDF file contains table data, the model needs to be trained using a table training set of the same type. Summary of the Invention
[0005] To address the aforementioned issues, this application provides an optical character recognition method and related products, with the aim of improving the versatility of optical character recognition models.
[0006] The embodiments of this application disclose the following technical solutions:
[0007] The first aspect of this application provides an optical character recognition method, including:
[0008] Obtain unformatted files; the unformatted files include at least PDF files, images, and tables;
[0009] The unformatted file is input into the graph-to-text model to obtain a formatted file corresponding to the unformatted file; the formatted file includes at least a Word file; the graph-to-text model is obtained through two training iterations and one validation; the two training iterations include training the graph-to-text model using a first training dataset and training it using a second training dataset; the one validation includes validating the graph-to-text model using a validation model and the second training dataset; the first training dataset includes formatted training files and unformatted training files corresponding to the formatted training files; the second training dataset includes unformatted training files and formatted training files corresponding to the unformatted training files.
[0010] The formatted file is validated using a large language model to obtain the validated file.
[0011] Optionally, the method further includes:
[0012] Construct the first training dataset, the second training dataset, and the third training dataset; the third training dataset includes formatted training files and validated formatted training files.
[0013] Construct graph-to-text model, large language model, text-to-graph model, and validation model;
[0014] Using the second training dataset, with the goal of minimizing the first loss value, the graph-text model is trained with parameter adjustment to obtain the graph-text model after the first parameter adjustment.
[0015] Using the first training dataset, with the goal of minimizing the second loss value, the graph-text model after the first parameter adjustment is trained to obtain the graph-text model after the second parameter adjustment.
[0016] Using the third training dataset, with the aim of minimizing the third loss value, the large language model is trained with parameter adjustment to obtain the parameter-adjusted large language model.
[0017] Using the first training dataset, with the goal of minimizing the fourth loss value, the text image model is trained with parameter adjustment to obtain the text image model after the first parameter adjustment.
[0018] Using the second training dataset, with the goal of minimizing the fifth loss value, the text image model after the first parameter adjustment is trained to obtain the text image model after the second parameter adjustment.
[0019] The graph-to-text model and the text-to-graph model after the second parameter adjustment are validated and adjusted using the validation model and the second training dataset to obtain the final graph-to-text model and the final text-to-graph model.
[0020] Optionally, the step of using the first training dataset to perform parameter adjustment training on the graph-text model after the first parameter adjustment with the aim of minimizing the second loss value, to obtain the graph-text model after the second parameter adjustment, specifically includes:
[0021] The formatted training file corresponding to the unformatted training file in the first training dataset is used as the input of the graph-to-text model after the first parameter adjustment, and the unformatted training file in the first training dataset is used as the target output of the graph-to-text model after the first parameter adjustment. With the goal of minimizing the second loss value, the graph-to-text model after the first parameter adjustment is trained with parameter adjustment to obtain the graph-to-text model after the second parameter adjustment. The second loss value is calculated based on the target output and the actual output of the graph-to-text model after the first parameter adjustment.
[0022] Optionally, the step of using the validation model and the second training dataset to validate and adjust the second parameter-adjusted graph-to-text model and the second parameter-adjusted text-to-graph model to obtain the final graph-to-text model and the final text-to-graph model specifically includes:
[0023] The unformatted training files in the second training dataset are used as input to the graph-to-text model after the second parameter adjustment, and the formatted training files corresponding to the unformatted training files in the second training dataset are used as input to the graph-to-text model after the second parameter adjustment, so as to obtain the output of the graph-to-text model after the second parameter adjustment and the preset layer of the graph-to-text model after the second parameter adjustment, respectively.
[0024] The output corresponding to the preset layer of the image-text model after the second parameter adjustment is used as the first input of the verification model, and the output corresponding to the preset layer of the text-image model after the second parameter adjustment is used as the second input of the verification model to obtain the output of the verification model;
[0025] If the output of the verification model satisfies the iteration termination condition, the iteration ends, and the final graph-to-text model and the final text-to-graph model are obtained.
[0026] If the output of the verification model does not meet the iteration termination condition, then the parameters of the second parameter-adjusted graph-to-text model and the second parameter-adjusted text-to-graph model are adjusted, and the process returns to the step of using the unformatted training file in the second training dataset as the input of the second parameter-adjusted graph-to-text model, and using the formatted training file corresponding to the unformatted training file in the second training dataset as the input of the second parameter-adjusted text-to-graph model, to obtain the outputs corresponding to the preset layers of the second parameter-adjusted graph-to-text model and the second parameter-adjusted text-to-graph model, respectively.
[0027] Optionally, the graph-to-text model includes convolutional neural networks and recurrent neural networks.
[0028] Optionally, the text-based graph model includes a Transformer model.
[0029] A second aspect of this application provides an optical character recognition device, comprising:
[0030] The acquisition module is used to acquire unformatted files; the unformatted files include at least PDF files, images, and tables.
[0031] A format conversion module is used to input the unformatted file into the graph-to-text model to obtain a formatted file corresponding to the unformatted file; the formatted file includes at least a Word file; the graph-to-text model is obtained through two training iterations and one validation; the two training iterations include training the graph-to-text model using a first training dataset and training the graph-to-text model using a second training dataset; the one validation includes validating the graph-to-text model using a validation model and the second training dataset; the first training dataset includes formatted training files and unformatted training files corresponding to the formatted training files; the second training dataset includes unformatted training files and formatted training files corresponding to the unformatted training files.
[0032] The verification module is used to verify the formatted file using a large language model to obtain the verified file.
[0033] A third aspect of this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the optical character recognition method provided in the first aspect.
[0034] The fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the optical character recognition method provided in the first aspect.
[0035] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the optical character recognition method provided in the first aspect.
[0036] Compared with the prior art, this application has the following beneficial effects:
[0037] This application includes obtaining unformatted files; the unformatted files include at least PDF files, images, and tables; inputting the unformatted files into a graph-to-text model to obtain formatted files corresponding to the unformatted files; the formatted files include at least Word files; the graph-to-text model is obtained through two training iterations and one validation; the two training iterations include training the graph-to-text model using a first training dataset and training the graph-to-text model using a second training dataset; the one validation includes validating the graph-to-text model using a validation model and the second training dataset; the first training dataset includes formatted training files and unformatted training files corresponding to the formatted training files; the second training dataset includes unformatted training files and formatted training files corresponding to the unformatted training files; and validating the formatted files using a large language model to obtain validated files.
[0038] In the model training phase, this application employs multiple training datasets to iteratively train the graph-to-text (Graph-to-Text) model. These training datasets include not only different types of unformatted files (such as PDF documents, images, scanned documents, and table screenshots) but also their corresponding structured formatted files (such as Word documents and Excel spreadsheets). Through this multimodal, multi-format data-driven approach, the graph-to-text model can fully learn the layout structure, semantic expression, and format conversion rules between different document types, thereby significantly improving its adaptability to diverse inputs and enhancing its generalization and versatility. During model optimization, a validation model is introduced into the training process. After the graph-to-text model generates preliminary formatted results, a validation model is used to evaluate its quality and provide feedback optimization, further improving the accuracy and structural integrity of the model's output.
[0039] This application further introduces a large language model to perform semantic-level verification and optimization of the formatted files generated by the graph-to-text model. The large language model analyzes the text content across dimensions such as syntax, logic, and contextual coherence, automatically identifying and correcting potential language errors, inappropriate word choice, or semantic ambiguity. This ensures that the final output formatted file not only conforms to standards in structure but also possesses high readability and professionalism in semantic expression. This effectively improves the practicality and usability of the generated documents. Attached Figure Description
[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0041] Figure 1 A flowchart illustrating an optical character recognition method provided in this application embodiment;
[0042] Figure 2 A first schematic diagram of the training process provided in the embodiments of this application;
[0043] Figure 3 A second schematic diagram of the training process provided in the embodiments of this application;
[0044] Figure 4 This is a structural diagram of an optical character recognition device provided in an embodiment of this application. Detailed Implementation
[0045] As described earlier, conventional OCR currently suffers from several problems, such as the lack of model universality. Different scenarios require the collection of different data and the training of different models. For example, for PDF files, if the PDF file is entirely text, the model needs to be trained using a text training set of the same type; if the PDF file contains table data, the model needs to be trained using a table training set of the same type.
[0046] In view of the above problems, this application provides an optical character recognition method and related products. The method includes: acquiring an unformatted file; the unformatted file includes at least a PDF file, an image, and a table; inputting the unformatted file into a graph-to-text (Graph-to-Text) model to obtain a formatted file corresponding to the unformatted file; the formatted file includes at least a Word file; the graph-to-text model is obtained through two training iterations and one verification; the two training iterations include training the graph-to-text model using a first training dataset and training the graph-to-text model using a second training dataset; the one verification includes verifying the graph-to-text model using a verification model and the second training dataset; the first training dataset includes a formatted training file and an unformatted training file corresponding to the formatted training file; the second training dataset includes an unformatted training file and a formatted training file corresponding to the unformatted training file; and verifying the formatted file using a large language model to obtain a verified file.
[0047] This application comprehensively improves the versatility, accuracy, and output quality of the graph-to-text model when processing unformatted files by combining model training driven by multiple training datasets, model-assisted optimization through validation, and semantic validation using a large language model. It achieves intelligent conversion from raw documents such as images or PDFs to high-quality formatted documents.
[0048] Explanation of relevant terms:
[0049] Human annotation: Human annotation refers to the process of labeling or classifying data by human experts. This is a common method for creating training sets in machine learning, especially in supervised learning. Through human annotation, algorithms can learn to recognize patterns and make predictions. For example, in image recognition tasks, workers might manually label images to explain their content.
[0050] Neural network models: Neural network models are a class of artificial intelligence algorithms inspired by biological nervous systems. They consist of a large number of simple processing units (nodes or "neurons") interconnected to form a hierarchical structure. Each connection has weight parameters, which can be adjusted to change the model's behavior. Neural networks excel at handling nonlinear problems and are widely used in fields such as image recognition, speech recognition, and natural language processing.
[0051] Model training: Model training is a crucial step in machine learning. It involves providing the model with a set of known input and output data (i.e., the training set), enabling the model to learn the relationship between the inputs and outputs. During this process, the model continuously adjusts its internal parameters (such as weights in a neural network) to minimize prediction error. After sufficient training, the model should theoretically be able to make accurate predictions or classifications on unseen new data.
[0052] Fine-tuning refers to the process of further training an already trained model (usually a large pre-trained model) for a specific task. This process typically involves using a smaller learning rate and a smaller dataset, with the aim of better adapting the model to a specific task or domain while retaining general features and knowledge learned from the original large-scale data. Fine-tuning is a form of transfer learning widely used in fields such as natural language processing and computer vision. Through fine-tuning, the performance of a model on a specific task can be improved without training the model from scratch.
[0053] Image-to-Text (Image-to-Text) models are those capable of converting image content into textual descriptions. These models have wide applications in creative design, multimedia content creation, image annotation, and automatic labeling. Through deep learning techniques, particularly the combination of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), Image-to-Text models can understand and analyze objects, scenes, and context within images, transforming them into natural language descriptions.
[0054] Text-to-Image Large-Scale Models: These are deep learning models capable of transforming text descriptions into images. They are typically based on deep learning techniques, particularly the Transformer model, which understands natural language input and generates corresponding images. Text-to-Image Large-Scale Models are not limited to generating real-world images; they can also generate virtual images that do not exist in reality.
[0055] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.
[0056] Figure 1 A flowchart of an optical character recognition method provided in this application embodiment is shown below. Figure 1 As shown, an optical character recognition method includes:
[0057] S101: Get an unformatted file.
[0058] This application does not specify the method for obtaining unformatted files. For example, it may involve automatically obtaining unformatted files from various cloud storage platforms, office collaboration platforms, or customer management systems through API interfaces or SDK integration; or using web crawling technology to extract images or PDF files containing text content from web pages. This application does not specify the specific form of unformatted files. For example, unformatted files may include at least PDF files, images, and tables.
[0059] S102: Input the unformatted file into the graph-text model to obtain the formatted file corresponding to the unformatted file.
[0060] This application does not limit the formatted file; for example, formatted files must at least include Word files. This application does not specifically limit the graph-to-text model; any open-source graph-to-text model can be used, such as neural network models (e.g., convolutional neural networks and recurrent neural networks), or mixed DIT models.
[0061] This application does not limit the training process of the graph-to-text (Graph-to-Text) model, but it must ensure that the Graph-to-Text model has undergone extensive training so that it can fully learn the layout structure, semantic expression, and format conversion rules between different types of documents, thereby significantly improving its adaptability to diverse inputs and enhancing the model's generalization ability and versatility. For example, the Graph-to-Text model is obtained through two training iterations and one validation iteration; the two training iterations include training the Graph-to-Text model using a first training dataset and training it using a second training dataset; the one validation iteration includes validating the Graph-to-Text model using a validation model and the second training dataset; the first training dataset includes formatted training files and unformatted training files corresponding to the formatted training files; the second training dataset includes unformatted training files and formatted training files corresponding to the unformatted training files.
[0062] By training the model multiple times using multiple training datasets, the graph-to-text model can adapt to different types of unformatted documents (such as scanned documents, handwritten notes, and printed documents), thus improving the model's versatility and robustness.
[0063] S103: Validate the formatted file using a large language model to obtain the validated file.
[0064] The large language model used in this application can be any open-source large language model, such as the qwen2.5-72b model. The formatted file is validated using the qwen2.5-72b model to obtain the validated file. This application introduces a large language model to perform semantic validation on the generated formatted file, correcting potential language errors and logical inconsistencies, ensuring that the output file is not only correctly formatted but also semantically clear and accurately expressed. The entire process can automatically convert from unformatted to formatted files without manual intervention, greatly improving document processing efficiency and reducing the workload of manual proofreading.
[0065] This application achieves universal OCR recognition and post-verification across various scenarios and data types, while improving recognition accuracy. It saves labor costs, eliminating the need for separate model training, deployment, and maintenance for each scenario; it is also more convenient to use, as business personnel do not need to understand the relevant technical details and can simply call a single interface.
[0066] The above describes the main technical solution of this application. Further implementations of the main technical solution are now introduced. Details are as follows:
[0067] Regarding the training process of graph-based text models and large language models, this application provides an optional embodiment:
[0068] Figure 2 A first schematic diagram of the training process provided in the embodiments of this application, as shown below. Figure 2 As shown:
[0069] S201: Construct the first training dataset, the second training dataset, and the third training dataset. The third training dataset includes formatted training files and validated formatted training files.
[0070] This application does not limit the construction method of the first training dataset, the second training dataset, and the third training dataset. For example, a large amount of real editable formatted data of the bank, such as Word, Excel, etc., can be collected; a large amount of real non-editable formatted data of the bank that needs to be processed, such as PDF, etc., can be collected; and the first training dataset, the second training dataset, and the third training dataset can be constructed based on the collected data.
[0071] The first training dataset consists of two parts: input and output. The input is a collection of real, editable, formatted files, and the output is unformatted files such as PDFs and images that need to be processed in the business scenario, converted using a format conversion tool. The first training dataset should contain no fewer than 100,000 entries.
[0072] The second training dataset consists of two parts: input and output. The input is real, uneditable, unformatted files that need to be processed, such as PDFs and images. This data is processed using manual annotation to obtain the output, which is an editable, formatted file required in the business scenario. The second training dataset contains no fewer than 100,000 records.
[0073] The output of the third training dataset consists of the input (text portion) of the first training dataset and the output (text portion) of the second training dataset. The input portion is the output portion of each data entry, obtained by randomly adding, deleting, or modifying text, with the three processing ratios ranging from 1% to 25%, performed randomly. After these operations, the third training dataset contains no fewer than 2 million data entries.
[0074] Three training datasets are introduced, namely the first training dataset, the second training dataset, and the third training dataset, which are used for different training tasks at different stages. The first training dataset focuses on the input-output mapping relationship in real-world scenarios. The second training dataset emphasizes the consistency of image-to-text conversion. The third training dataset is used for semantic-level optimization to improve the language quality of the generated results. The multi-source training data effectively improves the model's adaptability and generalization to different types of unformatted files.
[0075] S202: Construct a graph-to-text model, a large language model, a text-to-graph model, and a validation model.
[0076] The image-to-text model converts unformatted files like PDFs into formatted files containing editable text data. The large language model validates the output of the image-to-text model, producing a validated file; the large language model primarily corrects grammatical, textual, and logical errors in the input, as well as rearranging layout issues. The text-to-image model converts formatted files containing editable text data into unformatted files like images and PDFs. The validation model ensures consistency between the input and output of the image-to-text and text-to-image models, achieving higher OCR recognition accuracy.
[0077] The text-based graph model can use any open-source text-based graph model, such as Alibaba's Tongyi Wanxiang or the Transformer model.
[0078] S203: Using the second training dataset, with the goal of minimizing the first loss value, the graph-to-text model is trained with parameter adjustment to obtain the graph-to-text model after the first parameter adjustment.
[0079] S204: Using the first training dataset, with the goal of minimizing the second loss value, perform parameter adjustment training on the graph-to-text model after the first parameter adjustment to obtain the graph-to-text model after the second parameter adjustment.
[0080] In this embodiment, the graph-to-text model was trained twice and then validated once. However, this embodiment only demonstrates a relatively specific training method and does not imply that the graph-to-text model in this application must undergo two training sessions and one validation. The purpose of performing two training sessions and one validation in this application is mainly to improve the accuracy of the graph-to-text model. Therefore, if there are other training methods to improve the accuracy of the graph-to-text model, this application is also applicable. Furthermore, this application does not limit the training process of the first and second training sessions of the graph-to-text model. For example, the unformatted training files in the second training dataset can be used as the input of the graph-to-text model, and the formatted training files corresponding to the unformatted training files in the second training dataset can be used as the target output of the graph-to-text model. With the goal of minimizing the first loss value, the graph-to-text model is trained with parameter adjustment to obtain the graph-to-text model after the first parameter adjustment. The first loss value is calculated based on the target output and the actual output of the graph-to-text model.
[0081] The formatted training file corresponding to the unformatted training file in the first training dataset is used as the input of the graph-to-text model after the first parameter adjustment, and the unformatted training file in the first training dataset is used as the target output of the graph-to-text model after the first parameter adjustment. With the goal of minimizing the second loss value, the graph-to-text model after the first parameter adjustment is trained with parameter adjustment to obtain the graph-to-text model after the second parameter adjustment. The second loss value is calculated based on the target output and the actual output of the graph-to-text model after the first parameter adjustment.
[0082] The graph-to-text model employs a two-stage parameter adjustment training mechanism. It first uses the second training dataset for initial optimization to reduce recognition errors, and then uses the first training dataset for fine-tuning to improve structural fidelity.
[0083] S205: Using the third training dataset, with the aim of minimizing the third loss value, the parameters of the large language model are adjusted and trained to obtain the parameter-adjusted large language model.
[0084] This application does not limit the specific training process of the large language model. For example, the formatted training file in the third training dataset is used as the input of the large language model, and the validated formatted training file in the third training dataset is used as the target output of the large language model. With the goal of minimizing the third loss value, the parameters of the large language model are adjusted and trained to obtain the parameter-adjusted large language model. The third loss value is calculated based on the target output and the actual output of the large language model.
[0085] A large language model is introduced and specifically optimized using a third training dataset. This dataset contains original formatted files and validated high-quality formatted files, which guide the large language model to learn how to correct grammatical errors and optimize expression logic, significantly improving the readability and professionalism of the final output document.
[0086] S206: Using the first training dataset, with the goal of minimizing the fourth loss value, the text image model is trained with parameter adjustment to obtain the text image model after the first parameter adjustment.
[0087] This application does not limit the first training process of the text-based image model. For example, the formatted training file in the first training dataset is used as the input of the text-based image model, and the unformatted training file corresponding to the formatted training file in the first training dataset is used as the target output of the text-based image model. With the goal of minimizing the fourth loss value, the text-based image model is trained with parameter adjustment to obtain the text-based image model after the first parameter adjustment. The fourth loss value is calculated based on the target output and the actual output of the text-based image model.
[0088] S207: Using the second training dataset, with the goal of minimizing the fifth loss value, perform parameter adjustment training on the Wensheng image model after the first parameter adjustment to obtain the Wensheng image model after the second parameter adjustment.
[0089] This application does not limit the second training process of the text-based image model. For example, the formatted training file corresponding to the unformatted training file in the second training dataset can be used as the input of the text-based image model after the first parameter adjustment, and the unformatted training file in the second training dataset can be used as the target output of the text-based image model after the first parameter adjustment. With the goal of minimizing the fifth loss value, the text-based image model after the first parameter adjustment is trained to adjust the parameters to obtain the text-based image model after the second parameter adjustment. The fifth loss value is calculated based on the target output and the actual output of the text-based image model after the first parameter adjustment.
[0090] The Wensheng graph model also employs a two-stage training strategy to ensure its stability and accuracy under different input conditions.
[0091] S208: Using the verification model and the second training dataset, the second parameter-adjusted graph-to-text model and the second parameter-adjusted text-to-graph model are verified and adjusted to obtain the final graph-to-text model and the final text-to-graph model.
[0092] This application does not limit the verification and adjustment process of the graph-to-text model after the second parameter adjustment and the text-to-graph model after the second parameter adjustment. For example, Figure 3A second schematic diagram of the training process provided in the embodiments of this application, as shown below. Figure 3 As shown, S301: The unformatted training file in the second training dataset is used as the input of the graph-to-text model after the second parameter adjustment, and the formatted training file corresponding to the unformatted training file in the second training dataset is used as the input of the graph-to-text model after the second parameter adjustment, so as to obtain the output of the graph-to-text model after the second parameter adjustment and the preset layer of the graph-to-text model after the second parameter adjustment, respectively.
[0093] S302: Use the output corresponding to the preset layer of the image-to-text model after the second parameter adjustment as the first input of the verification model, and use the output corresponding to the preset layer of the text-to-image model after the second parameter adjustment as the second input of the verification model to obtain the output of the verification model.
[0094] This application does not limit the structure of the validation model. For example, the validation model consists of two branches, forming a double-tower structure. The two branches have identical structures and parameters. Each branch includes two parts. The first part has three branches, each consisting of a four-layer convolutional neural network. The kernel sizes of each layer are set to 7*1, 5*1, 3*1, and 1*1. The second part includes four fully connected layers. The number of nodes in the first layer is the same as the output of the convolutional neural network in the first part. The second layer has 1024 nodes, the third layer has 512 nodes, and the fourth layer has 128 nodes. During training, the loss function for this part can use the L1 loss function, which measures the difference between the output vectors of the two branches, with the goal of minimizing the difference.
[0095] S303: Does the output of the verification model satisfy the iteration termination condition?
[0096] S304: If the output of the verification model does not meet the iteration termination condition, then the parameters of the graph-to-text model after the second parameter adjustment and the text-to-graph model after the second parameter adjustment are adjusted, and the process returns to S301.
[0097] S305: If the output of the verification model satisfies the iteration termination condition, the iteration ends, and the final graph-to-text model and the final text-to-graph model are obtained.
[0098] The above embodiments describe the model training process in detail. After the model training is completed, the trained graph-to-text model and large language model can be obtained. The trained graph-to-text model and large language model can be deployed online and used by business personnel as needed.
[0099] The above embodiments train the model with the goal of minimizing the loss value. As an optional embodiment, the model can also be trained based on the number of training iterations. For example, if the graph-text model and the large language model have already undergone overall parameter tuning training, then when faced with a new training set, the trained graph-text model and the large language model can be fine-tuned. Specifically, this includes:
[0100] Construct the first training dataset, the second training dataset, and the third training dataset; the third training dataset includes formatted training files and validated formatted training files.
[0101] Construct a graph-to-text model, a large language model, a text-to-graph model, and a validation model.
[0102] Using the second training dataset, the graph-to-text model was trained with fine-tuning parameters for approximately 10 epochs, resulting in the graph-to-text model after its first parameter adjustment. One epoch refers to all simulated data being processed through the model once.
[0103] Using the first training dataset, with the aim of training for approximately 20 epochs, the graph-to-text model after the first parameter adjustment is fine-tuned to obtain the graph-to-text model after the second parameter adjustment.
[0104] Using the third training dataset, the large language model is trained with fine-tuned parameters for approximately 20 epochs to obtain the parameter-adjusted large language model.
[0105] Using the first training dataset, the text-based image model is trained with fine-tuned parameters for approximately 20 epochs to obtain the text-based image model after the first parameter adjustment.
[0106] Using the second training dataset, with the aim of training for approximately 20 epochs, the textural image model after the first parameter adjustment is fine-tuned to obtain the textural image model after the second parameter adjustment.
[0107] The graph-to-text model and the text-to-graph model after the second parameter adjustment are validated and adjusted using the validation model and the second training dataset to obtain the final graph-to-text model and the final text-to-graph model.
[0108] This application achieves unified modeling and processing for multiple business scenarios by constructing a generalized model structure; it avoids the large amount of repetitive work caused by developing, training, and maintaining models separately for each specific task; it significantly reduces the workload of technical personnel and the management burden of operation and maintenance personnel, thereby greatly reducing human resource input and operating costs. Only one model needs to be trained and deployed to meet the needs of multiple application scenarios; compared with the method of running multiple models in parallel, this application is more lightweight in terms of training resources, computational overhead, and deployment complexity; at the same time, it supports rapid iteration and flexible expansion to adapt to ever-changing business needs.
[0109] Within a unified model framework, data from different business scenarios can be fused and trained; data silos are broken down, enabling the model to learn common features across a wider data distribution; the model's generalization ability is improved, enhancing its adaptability and robustness to new scenarios; and a leap from "dispersed data utilization" to "data collaborative driving" is achieved, fully leveraging the potential value of multi-source heterogeneous data. This application avoids information loss and error propagation in intermediate stages through end-to-end modeling with a single model; reduces the uncertainty caused by model switching or combination, and improves the overall system's predictive stability and result reliability; in practical applications, it demonstrates higher accuracy and lower false positive rate.
[0110] The model interface is well-encapsulated, and the user interface is simple and intuitive. Business users only need to input basic parameters or upload raw data to obtain the desired results. There's no need to understand the model's internal structure, algorithm principles, or parameter tuning techniques, greatly lowering the barrier to entry. This allows non-technical personnel to efficiently apply artificial intelligence technology to their daily work, improving the overall level of business intelligence. It effectively reduces high labor costs; one model solves problems in all scenarios, making training, deployment, and maintenance more convenient. It maximizes the potential of data by combining data from different scenarios. One model fulfills the requirements, resulting in less error accumulation and higher accuracy. It's simple to use; business users don't need to understand any related technical details.
[0111] Figure 4 A structural diagram of an optical character recognition device provided in an embodiment of this application is shown below. Figure 4 As shown, based on the optical character recognition method provided in the preceding embodiments, this application also provides an optical character recognition device including:
[0112] The acquisition module is used to acquire unformatted files; the unformatted files include at least PDF files, images, and tables.
[0113] The format conversion module is used to input the unformatted file into the graph-to-text model to obtain a formatted file corresponding to the unformatted file; the formatted file includes at least a Word file; the graph-to-text model is obtained through two training sessions and one validation; the two training sessions include training the graph-to-text model using a first training dataset and training the graph-to-text model using a second training dataset; the one validation includes validating the graph-to-text model using a validation model and the second training dataset; the first training dataset includes a formatted training file and an unformatted training file corresponding to the formatted training file; the second training dataset includes an unformatted training file and a formatted training file corresponding to the unformatted training file.
[0114] The verification module is used to verify the formatted file using a large language model to obtain the verified file.
[0115] As an optional embodiment, the device further includes:
[0116] A dataset construction module is used to construct the first training dataset, the second training dataset, and the third training dataset. The third training dataset includes formatted training files and validated formatted training files.
[0117] The model building module is used to build graph-to-text models, large language models, text-to-graph models, and validation models.
[0118] The first training module is used to train the graph-to-text model by adjusting the parameters using the second training dataset with the aim of minimizing the first loss value, so as to obtain the graph-to-text model after the first parameter adjustment.
[0119] The second training module is used to train the graph-to-text model after the first parameter adjustment using the first training dataset with the goal of minimizing the second loss value, so as to obtain the graph-to-text model after the second parameter adjustment.
[0120] The third training module is used to train the large language model by adjusting the parameters using the third training dataset with the aim of minimizing the third loss value, so as to obtain the large language model with adjusted parameters.
[0121] The fourth training module is used to train the text image model by adjusting the parameters using the first training dataset with the goal of minimizing the fourth loss value, so as to obtain the text image model after the first parameter adjustment.
[0122] The fifth training module is used to train the text image model after the first parameter adjustment using the second training dataset with the goal of minimizing the fifth loss value, so as to obtain the text image model after the second parameter adjustment.
[0123] The sixth training module is used to verify and adjust the graph-to-text model and the text-to-graph model after the second parameter adjustment using the verification model and the second training dataset, so as to obtain the final graph-to-text model and the final text-to-graph model.
[0124] As an optional embodiment, the second training module is specifically used for:
[0125] The formatted training file corresponding to the unformatted training file in the first training dataset is used as the input to the graph-to-text model after the first parameter adjustment. The unformatted training file in the first training dataset is used as the target output of the graph-to-text model after the first parameter adjustment. With the goal of minimizing the second loss value, the graph-to-text model after the first parameter adjustment is trained with adjusted parameters to obtain the graph-to-text model after the second parameter adjustment. The second loss value is calculated based on the target output and the actual output of the graph-to-text model after the first parameter adjustment.
[0126] As an optional embodiment, the sixth training module specifically includes:
[0127] The first input unit is used to take the unformatted training file in the second training dataset as the input of the graph-to-text model after the second parameter adjustment, and take the formatted training file corresponding to the unformatted training file in the second training dataset as the input of the graph-to-text model after the second parameter adjustment, so as to obtain the output of the graph-to-text model after the second parameter adjustment and the output corresponding to the preset layer of the graph-to-text model after the second parameter adjustment.
[0128] The second input unit is used to take the output corresponding to the preset layer of the graph-to-text model after the second parameter adjustment as the first input of the verification model, and take the output corresponding to the preset layer of the text-to-graph model after the second parameter adjustment as the second input of the verification model, so as to obtain the output of the verification model.
[0129] The first judgment unit is used to end the iteration if the output of the verification model satisfies the iteration termination condition, and obtain the final graph-to-text model and the final text-to-graph model.
[0130] The second judgment unit is used to adjust the parameters of the graph-to-text model and the text-to-graph model after the second parameter adjustment if the output of the verification model does not meet the iteration termination condition, and then return to the first input unit.
[0131] As an optional embodiment, the graph-to-text model includes convolutional neural networks and recurrent neural networks.
[0132] As an optional embodiment, the text-based graph model includes a Transformer model.
[0133] This application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement an optical character recognition method.
[0134] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements an optical character recognition method.
[0135] This application provides a computer program product, including a computer program that, when executed by a processor, implements an optical character recognition method.
[0136] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and apparatus embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0137] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. An optical character recognition method, characterized in that, The optical character recognition method includes: Obtain unformatted files; the unformatted files include at least PDF files, images, and tables; The unformatted file is input into the trained graph-to-text model to obtain a formatted file corresponding to the unformatted file; the formatted file includes at least a Word file; a graph-to-text model, a large language model, a text-to-graph model, and a validation model are constructed, and both the graph-to-text model and the text-to-graph model are obtained through two training iterations and one validation iteration; the graph-to-text model is trained for the first time using a second training dataset, and the graph-to-text model after the first training is trained for the second time using a first training dataset; the text-to-graph model is trained for the first time using the first training dataset, and the text-to-graph model after the first training is trained for the second time using the second training dataset; the validation model and the second training dataset are used to validate and adjust the graph-to-text model and the text-to-graph model after the second training, resulting in a trained graph-to-text model and a trained text-to-graph model; the first training dataset includes a formatted training file and an unformatted training file converted from the formatted training file; the second training dataset includes an unformatted training file and a formatted training file obtained by annotating the unformatted training file. The formatted file is validated using a large language model to obtain the validated file.
2. The optical character recognition method according to claim 1, characterized in that, The method further includes: Construct the first training dataset, the second training dataset, and the third training dataset; the third training dataset includes formatted training files and validated formatted training files. Construct graph-to-text model, large language model, text-to-graph model, and validation model; Using the second training dataset, with the goal of minimizing the first loss value, the graph-to-text model is trained with parameter adjustment to obtain the graph-to-text model after the first training. Using the first training dataset, with the goal of minimizing the second loss value, the graph-text model after the first training is trained with parameter adjustment to obtain the graph-text model after the second training. Using the third training dataset, with the aim of minimizing the third loss value, the large language model is trained with parameter adjustment to obtain the parameter-adjusted large language model. Using the first training dataset, with the goal of minimizing the fourth loss value, the text image model is trained with parameter adjustment to obtain the text image model after the first training. Using the second training dataset, with the goal of minimizing the fifth loss value, the parameters of the text image model after the first training are adjusted and trained to obtain the text image model after the second training. The second-trained graph-to-text model and the second-trained text-to-graph model are validated and adjusted using the validation model and the second training dataset to obtain the trained graph-to-text model and the trained text-to-graph model.
3. The optical character recognition method according to claim 2, characterized in that, The step of using the first training dataset to perform parameter adjustment training on the first-trained graph-to-text model with the aim of minimizing the second loss value, to obtain the second-trained graph-to-text model, specifically includes: The formatted training file corresponding to the unformatted training file in the first training dataset is used as the input of the graph-to-text model after the first training, and the unformatted training file in the first training dataset is used as the target output of the graph-to-text model after the first training. With the goal of minimizing the second loss value, the graph-to-text model after the first training is trained by adjusting the parameters to obtain the graph-to-text model after the second training. The second loss value is calculated based on the target output and the actual output of the graph-to-text model after the first training.
4. The optical character recognition method according to claim 2, characterized in that, The step of using the validation model and the second training dataset to validate and adjust the second-trained graph-to-text model and the second-trained text-to-graph model to obtain the trained graph-to-text model and the trained text-to-graph model specifically includes: The unformatted training files in the second training dataset are used as input to the second trained graph-to-text model, and the formatted training files corresponding to the unformatted training files in the second training dataset are used as input to the second trained text-to-graph model, so as to obtain the output of the second trained graph-to-text model and the preset layer of the second trained text-to-graph model respectively. The output of the preset layer of the second-trained graph-to-text model is used as the first input of the verification model, and the output of the preset layer of the second-trained text-to-graph model is used as the second input of the verification model to obtain the output of the verification model. If the output of the verification model satisfies the iteration termination condition, the iteration ends, and the trained graph-to-text model and the trained text-to-graph model are obtained. If the output of the verification model does not meet the iteration termination condition, then the parameters of the second-trained graph-to-text model and the second-trained text-to-graph model are adjusted, and the process returns to the step of using the unformatted training file in the second training dataset as the input of the second-trained graph-to-text model, and using the formatted training file corresponding to the unformatted training file in the second training dataset as the input of the second-trained text-to-graph model, to obtain the outputs of the second-trained graph-to-text model and the preset layer of the second-trained text-to-graph model, respectively.
5. The optical character recognition method according to claim 1, characterized in that, The trained graph-to-text model includes convolutional neural networks and recurrent neural networks.
6. The optical character recognition method according to claim 2, characterized in that, The text-based graph model includes the Transformer model.
7. An optical character recognition device, characterized in that, The optical character recognition device includes: The acquisition module is used to acquire unformatted files; the unformatted files include at least PDF files, images, and tables. A format conversion module is used to input the unformatted file into the trained graph-to-text model to obtain a formatted file corresponding to the unformatted file; the formatted file includes at least a Word file; a graph-to-text model, a large language model, a text-to-graph model, and a validation model are constructed, wherein both the graph-to-text model and the text-to-graph model are obtained through two training iterations and one validation iteration; the graph-to-text model is trained for the first time using a second training dataset, and the graph-to-text model after the first training is trained for the second time using a first training dataset; the text-to-graph model is trained for the first time using the first training dataset, and the text-to-graph model after the first training is trained for the second time using the second training dataset; the validation model and the second training dataset are used to validate and adjust the graph-to-text model and the text-to-graph model after the second training, resulting in a trained graph-to-text model and a trained text-to-graph model; the first training dataset includes formatted training files and unformatted training files converted from formatted training files; the second training dataset includes unformatted training files and formatted training files annotated from unformatted training files. The verification module is used to verify the formatted file using a large language model to obtain the verified file.
8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the optical character recognition method according to any one of claims 1-6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the optical character recognition method according to any one of claims 1-6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the optical character recognition method according to any one of claims 1-6.