Optical character recognition method and related product

Abstract

The invention discloses an optical character recognition method and a related product. The method comprises the following steps: acquiring an unformatted file; inputting the non-formatted file into a graph-to-text model to obtain a formatted file corresponding to the non-formatted file; and verifying the formatted file by using the large language model to obtain a verified file. In the model training stage, a plurality of training data sets are adopted to carry out iteration training on the graph-to-text model for multiple times. The training data set not only comprises different types of non-formatted files (such as PDF documents, images, scanners, table screenshots and the like), but also covers corresponding structured formatted files (such as Word documents, Excel tables and the like). Through the multi-mode and multi-format data driving mode, the graph-to-text model can fully learn layout structures, semantic expressions and format conversion rules among different types of documents, and the generalization ability and universality of the model are enhanced.

Description

Technical Field

[0001] The present application relates to the technical field of optical character recognition, and in particular to an optical character recognition method and related products. Background Art

[0002] With the rapid development of information technology, enterprises have accumulated a large amount of information, including a large number of paper documents, PDF files, pictures, tables and other unformatted files. In most cases, if enterprises want to use these files, they first need to convert the unformatted files into editable formatted files, and then implement subsequent data archiving, data structuring, data analysis and other operations.

[0003] Currently, conventional optical character recognition (OCR) generally collects large amounts of data based on specific scenarios; this large amount of data is used to train models built with deep learning or machine learning algorithms, enabling the models to detect, recognize, and restore text, converting uneditable non-formatted text into editable formatted text.

[0004] However, conventional OCR currently suffers from various issues, such as the lack of universal models. Different scenarios require the collection of different data and the training of different models. For example, for PDF files, if the PDF file is entirely text, the model must be trained using a training set of text of the same type; if the PDF file contains table data, the model must be trained using a training set of tables of the same type. Summary of the Invention

[0005] Based on the above problems, the present application provides an optical character recognition method and related products, the purpose of which is to improve the versatility of the optical character recognition model.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] A first aspect of the present application provides an optical character recognition method comprising:

[0008] Obtaining an unformatted file; the unformatted file includes at least a PDF file, a picture, and a table;

[0009] Inputting the unformatted file into a graphic text model to obtain a formatted file corresponding to the unformatted file; the formatted file includes at least a Word file; the graphic text model is obtained through two trainings and one verification; the two trainings include training the graphic text model using a first training data set and training the graphic text model using a second training data set; the one verification includes verifying and training the graphic text model using a verification model and a second training data set; the first training data set includes a formatted training file and an unformatted training file corresponding to the formatted training file; the second training data set includes the unformatted training file and a formatted training file corresponding to the unformatted training file;

[0010] The formatted file is verified using a large language model to obtain a verified file.

[0011] Optionally, the method further includes:

[0012] Constructing the first training data set, the second training data set, and the third training data set; the third training data set includes a formatted training file and a verified formatted training file;

[0013] Build graph-to-text models, large language models, text-to-graph models, and verification models;

[0014] Using the second training data set, with the goal of minimizing the first loss value, to perform parameter adjustment training on the graph-to-text model, thereby obtaining a graph-to-text model after the first parameter adjustment;

[0015] Using the first training data set, with the goal of minimizing the second loss value, performing parameter adjustment training on the graph-to-text model after the first parameter adjustment, to obtain a graph-to-text model after the second parameter adjustment;

[0016] Using the third training data set, with the goal of minimizing the third loss value, to perform parameter adjustment training on the large language model, to obtain a large language model after parameter adjustment;

[0017] Using the first training data set, with the goal of minimizing the fourth loss value, performing parameter adjustment training on the Vincent graph model to obtain the Vincent graph model after the first parameter adjustment;

[0018] Using the second training data set, with the goal of minimizing the fifth loss value, performing parameter adjustment training on the Vincent graph model after the first parameter adjustment, to obtain the Vincent graph model after the second parameter adjustment;

[0019] The verification model and the second training data set are used to verify and adjust the image-to-text model after the second parameter adjustment and the text-to-image model after the second parameter adjustment to obtain a final image-to-text model and a final text-to-image model.

[0020] Optionally, the step of using the first training data set to perform parameter adjustment training on the graph-to-text model after the first parameter adjustment with the goal of minimizing the second loss value to obtain the graph-to-text model after the second parameter adjustment specifically includes:

[0021] The formatted training files corresponding to the unformatted training files in the first training data set are used as the input of the graph-based text model after the first parameter adjustment, and the unformatted training files in the first training data set are used as the target output of the graph-based text model after the first parameter adjustment. With the purpose of minimizing the second loss value, parameter adjustment training is performed on the graph-based text model after the first parameter adjustment to obtain a graph-based text model after the second parameter adjustment; the second loss value is calculated based on the target output and the actual output of the graph-based text model after the first parameter adjustment.

[0022] Optionally, using the verification model and the second training data set to verify and adjust the image-to-text model after the second parameter adjustment and the text-to-image model after the second parameter adjustment to obtain a final image-to-text model and a final text-to-image model specifically includes:

[0023] Using the unformatted training files in the second training data set as input to the graph-to-text model after the second parameter adjustment, and using the formatted training files corresponding to the unformatted training files in the second training data set as input to the text-to-graph model after the second parameter adjustment, respectively obtaining the output corresponding to the graph-to-text model after the second parameter adjustment and the preset layer of the text-to-graph model after the second parameter adjustment;

[0024] Using the output corresponding to the preset layer of the graph-text model after the second parameter adjustment as the first input of the verification model, and using the output corresponding to the preset layer of the text-graph model after the second parameter adjustment as the second input of the verification model to obtain the output of the verification model;

[0025] If the output of the verification model meets the iteration end condition, the iteration ends to obtain the final graph-text model and the final text-graph model;

[0026] If the output of the verification model does not meet the iteration end condition, the parameters of the graph-text model after the second parameter adjustment and the culture-graph model after the second parameter adjustment are adjusted, and the step of returning to the step of using the unformatted training file in the second training data set as the input of the graph-text model after the second parameter adjustment, and using the formatted training file corresponding to the unformatted training file in the second training data set as the input of the culture-graph model after the second parameter adjustment, and obtaining the output corresponding to the preset layer of the graph-text model after the second parameter adjustment and the culture-graph model after the second parameter adjustment respectively.

[0027] Optionally, the graph-to-text model includes a convolutional neural network and a recurrent neural network.

[0028] Optionally, the text graph model includes a Transformer model.

[0029] A second aspect of the present application provides an optical character recognition device, comprising:

[0030] An acquisition module is used to acquire unformatted files; the unformatted files include at least PDF files, pictures and tables;

[0031] A format conversion module is configured to input the unformatted file into a graphic text model to obtain a formatted file corresponding to the unformatted file; the formatted file includes at least a Word file; the graphic text model is obtained through two trainings and one verification; the two trainings include training the graphic text model using a first training data set and training the graphic text model using a second training data set; the one verification includes verifying and training the graphic text model using a verification model and a second training data set; the first training data set includes a formatted training file and an unformatted training file corresponding to the formatted training file; the second training data set includes the unformatted training file and a formatted training file corresponding to the unformatted training file;

[0032] The verification module is used to verify the formatted file using a large language model to obtain a verified file.

[0033] A third aspect of the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the optical character recognition method provided in the first aspect.

[0034] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the optical character recognition method provided in the first aspect.

[0035] A fifth aspect of the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the optical character recognition method provided in the first aspect.

[0036] Compared with the existing technology, this application has the following beneficial effects:

[0037] The present application includes obtaining an unformatted file; the unformatted file includes at least a PDF file, a picture, and a table; inputting the unformatted file into a graphic text model to obtain a formatted file corresponding to the unformatted file; the formatted file includes at least a Word file; the graphic text model is obtained through two trainings and one verification; the two trainings include training the graphic text model using a first training data set and training the graphic text model using a second training data set; the one verification includes verifying and training the graphic text model using a verification model and a second training data set; the first training data set includes a formatted training file and an unformatted training file corresponding to the formatted training file; the second training data set includes the unformatted training file and the formatted training file corresponding to the unformatted training file; and verifying the formatted file using a large language model to obtain a verified file.

[0038] During the model training phase, this application uses multiple training data sets to perform multiple iterative training on the graph-based text model. The training data set not only includes different types of unformatted files (such as PDF documents, images, scans, table screenshots, etc.), but also covers corresponding structured formatted files (such as Word documents, Excel tables, etc.). Through this multimodal, multi-format data-driven approach, the graph-based text model can fully learn the layout structure, semantic expression and format conversion rules between different types of documents, thereby significantly improving its adaptability in the face of diversified inputs and enhancing the generalization ability and versatility of the model. During the model optimization process, a verification model is introduced to participate in the training process, that is, after the graph-based text model generates a preliminary formatted result, the verification model is used to perform quality assessment and feedback optimization, thereby further improving the accuracy and structural integrity of the model output results.

[0039] This application further introduces a large language model to perform semantic-level verification and optimization on the formatted files generated by the graph-to-text model. By analyzing the text content's grammar, logic, and contextual coherence, the large language model automatically identifies and corrects potential language errors, inappropriate wording, or semantic ambiguity, ensuring that the final output formatted files not only conform to standards in structure but also possess a high degree of readability and professionalism in semantic expression. This effectively improves the practicality and usability of the generated documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 A flowchart of an optical character recognition method provided in an embodiment of the present application;

[0042] Figure 2 A first schematic diagram of the training process provided in an embodiment of the present application;

[0043] Figure 3 A second schematic diagram of the training process provided in an embodiment of the present application;

[0044] Figure 4 This is a structural diagram of an optical character recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] As previously described, conventional OCR currently suffers from various issues, including model non-universality. Different scenarios require the collection of different data and the training of different models. For example, for PDF files, if the PDF is entirely text, the model must be trained using a training set of text data of the same type; if the PDF contains table data, the model must be trained using a training set of tables of the same type.

[0046] In view of the above problems, the present application provides an optical character recognition method and related products, the method comprising: obtaining an unformatted file; the unformatted file comprises at least a PDF file, a picture and a table; inputting the unformatted file into a graphic text model to obtain a formatted file corresponding to the unformatted file; the formatted file comprises at least a word file; the graphic text model is obtained through two trainings and one verification; the two trainings comprise training the graphic text model using a first training data set and training the graphic text model using a second training data set; the one verification comprises verifying and training the graphic text model using a verification model and a second training data set; the first training data set comprises a formatted training file and an unformatted training file corresponding to the formatted training file; the second training data set comprises an unformatted training file and a formatted training file corresponding to the unformatted training file; and verifying the formatted file using a large language model to obtain a verified file.

[0047] This application comprehensively improves the versatility, accuracy, and output quality of the image-to-text model when processing unformatted files by combining model training driven by multiple training datasets, verification model-assisted optimization, and semantic verification of large language models, thereby achieving intelligent conversion from original documents such as images or PDFs to high-quality formatted documents.

[0048] Explanation of relevant terms:

[0049] Manual annotation: Manual annotation is the process of labeling or categorizing data by human experts. This is a common method for creating training sets in machine learning, particularly in supervised learning. Human annotation allows algorithms to learn to recognize patterns and make predictions. For example, in image recognition tasks, a human might manually label an image to describe its content.

[0050] Neural network models: Neural network models are a type of artificial intelligence algorithm inspired by biological neural systems. They consist of a large number of simple processing units (nodes or "neurons") interconnected in a hierarchical structure. Each connection has a weight parameter, and these weights can be adjusted to modify the model's behavior. Neural networks are particularly adept at handling nonlinear problems and are widely used in fields such as image recognition, speech recognition, and natural language processing.

[0051] Model training: Model training is a key step in machine learning. It involves providing a set of data with known inputs and outputs (the training set) to the model, enabling it to learn the relationship between inputs and outputs. During this process, the model continuously adjusts its internal parameters (such as the weights in a neural network) to minimize prediction error. After sufficient training, the model should theoretically be able to make accurate predictions or classifications on new, unseen data.

[0052] Fine-tuning: Fine-tuning is the process of further training an already trained model (usually a large pre-trained model) for a specific task. This process typically involves using a smaller learning rate and a smaller dataset. The goal is to better adapt the model to the specific task or domain while retaining the general features and knowledge learned from the original large-scale data. Fine-tuning is a form of transfer learning widely used in fields such as natural language processing and computer vision. Through fine-tuning, the performance of a model on a specific task can be improved without training the model from scratch.

[0053] Image-to-Text Models: Image-to-Text Models are models that convert image content into textual descriptions. These models have broad applications in creative design, multimedia content creation, image annotation, and automatic labeling. Using deep learning techniques, particularly the combination of convolutional neural networks (CNNs) and recurrent neural networks (RNNs), Image-to-Text Models can understand and analyze objects, scenes, and context in images and convert them into natural language descriptions.

[0054] Text-to-Image Big Model: A text-to-image big model is a deep learning model that can convert text descriptions into images. These models are typically based on deep learning techniques, particularly the Transformer model, and can understand natural language input and generate corresponding images. Text-to-Image Big Models are not limited to generating existing images; they can also generate virtual images that do not exist in reality.

[0055] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0056] Figure 1 A flowchart of an optical character recognition method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, an optical character recognition method includes:

[0057] S101: Obtain an unformatted file.

[0058] This application does not specifically limit the method for obtaining unformatted files. For example, unformatted files can be automatically obtained from various cloud storage platforms, office collaboration platforms, or customer management systems through API interfaces or SDK integration; or images or PDF files containing text content can be extracted from web pages using crawler technology. This application does not specifically limit the specific form of unformatted files. For example, unformatted files can include at least PDF files, images, and tables.

[0059] S102: Input the unformatted file into a graph-to-text model to obtain a formatted file corresponding to the unformatted file.

[0060] This application does not limit the formatted files, for example, the formatted files at least include Word files. This application does not specifically limit the graph-to-text model, and the graph-to-text model can use any open source graph-to-text model, for example, the graph-to-text model includes a neural network model (such as a convolutional neural network and a recurrent neural network), or the graph-to-text model includes a mixed-element DIT model.

[0061] This application does not limit the training process of the graph-generated text model, but it is necessary to ensure that the graph-generated text model has undergone a large amount of training, so that the graph-generated text model can fully learn the layout structure, semantic expression and format conversion rules between different types of documents, thereby significantly improving its adaptability when facing diverse inputs and enhancing the generalization ability and versatility of the model. For example, the graph-generated text model is obtained through two trainings and one verification; the two trainings include training the graph-generated text model using a first training data set and training the graph-generated text model using a second training data set; the one verification includes verifying and training the graph-generated text model using a verification model and a second training data set; the first training data set includes a formatted training file and a non-formatted training file corresponding to the formatted training file; the second training data set includes an unformatted training file and a formatted training file corresponding to the unformatted training file.

[0062] By using multiple training datasets for multiple training sessions, the image-to-text model can adapt to different types of unformatted files (such as scans, handwritten notes, and printed documents), improving the versatility and robustness of the model.

[0063] S103: Using a large language model to verify the formatted file, to obtain a verified file.

[0064] The large language model in this application can use any open source large language model, for example, the large language model is the qwen2.5-72b model. The formatted file is verified using the qwen2.5-72b model to obtain a verified file. This application introduces a large language model to perform semantic verification on the generated formatted file, correct potential language errors and logical inconsistencies, and ensure that the output file is not only formatted correctly, but also has clear semantics and accurate expression. The entire process can complete the automatic conversion from unformatted files to formatted files without human intervention, which greatly improves the efficiency of document processing and reduces the workload of manual proofreading.

[0065] This application achieves universal OCR recognition and post-verification of various data in various scenarios, while improving recognition accuracy. This reduces labor costs by eliminating the need for separate model training, deployment, and maintenance for each scenario. It is also more convenient to use, as business personnel can directly call it through a single interface without having to understand the relevant technical details.

[0066] The above is the main technical solution of this application. Now we will introduce the expansion and implementation of the main technical solution of this application. The details are as follows:

[0067] Regarding the training process of the graph-to-text model and the large language model, this application provides an optional embodiment:

[0068] Figure 2 The first schematic diagram of the training process provided in the embodiment of the present application is as follows: Figure 2 As shown:

[0069] S201: Constructing the first training data set, the second training data set, and the third training data set. The third training data set includes a formatted training file and a verified formatted training file.

[0070] This application does not limit the method of constructing the first training data set, the second training data set, and the third training data set. For example, a large amount of real editable formatted data of the bank, such as Word, Excel, etc., is collected; a large amount of non-editable formatted data that the bank actually needs to process, such as PDF, etc., is collected; and the first training data set, the second training data set, and the third training data set are constructed based on the collected data.

[0071] The first training dataset consists of two parts: input and output. The input is collected, real, editable, and formatted files. The output is converted into non-formatted files such as PDFs and images, using a format conversion tool, that are required for business scenarios. The first training dataset should have no fewer than 100,000 records.

[0072] The second training dataset consists of two parts: input and output. The input is the actual, non-editable, unformatted files that need to be processed, such as PDFs and images. This data is processed using manual annotation to generate the output, which is the editable, formatted files required in business scenarios. The second training dataset is no less than 100,000 records.

[0073] The output of the third training dataset is the input (text portion) of the first training dataset and the output (text portion) of the second training dataset. The input is the output portion of each data item, which is obtained by randomly adding, deleting, and modifying text. The proportion of these three types of processing varies from 1% to 25%. After these operations, the third training dataset has no less than 2 million characters.

[0074] The first, second and third training datasets are introduced, which are used for training tasks at different stages respectively; the first training dataset focuses on the input-output mapping relationship in real scenarios; the second training dataset emphasizes the consistency of image-to-text conversion; the third training dataset is used for semantic-level optimization to improve the language quality of the generated results; multi-source training data effectively improves the model's adaptability and generalization to different types of unformatted files.

[0075] S202: Construct a graph-to-text model, a large language model, a text-to-graph model, and a verification model.

[0076] The image-to-text model converts unformatted files, such as PDFs, into formatted files with editable text data. The large language model verifies the output of the image-to-text model to produce a verified file. The large language model primarily corrects grammatical, textual, and logical errors in the input, as well as rearranges the layout for issues like layout. The text-to-image model converts formatted files with editable text data into unformatted files, such as images and PDFs. The verification model ensures input and output consistency between the image-to-text model and the text-to-image model, achieving higher OCR recognition accuracy.

[0077] The text-based graph model can use any open-source text-based graph model, such as Alibaba's Tongyi Wanxiang or the Transformer model.

[0078] S203: Using the second training data set, with the goal of minimizing the first loss value, perform parameter adjustment training on the graph-to-text model to obtain the graph-to-text model after the first parameter adjustment.

[0079] S204: Using the first training data set, with the goal of minimizing the second loss value, perform parameter adjustment training on the graph-to-text model after the first parameter adjustment, to obtain the graph-to-text model after the second parameter adjustment.

[0080] In this embodiment, the graph-generated text model is first trained twice and then verified once, but this embodiment only shows a more specific training method and does not mean that the graph-generated text model in this application must be trained twice and verified once. The two training and one verification in this application are mainly to improve the accuracy of the graph-generated text model. Therefore, if there are other training methods to improve the accuracy of the graph-generated text model, this application is also applicable; and this application does not limit the training process of the first training and the second training of the graph-generated text model. For example, the unformatted training file in the second training data set is used as the input of the graph-generated text model, and the formatted training file corresponding to the unformatted training file in the second training data set is used as the target output of the graph-generated text model. With the purpose of minimizing the first loss value, the graph-generated text model is trained with parameter adjustment to obtain the graph-generated text model after the first parameter adjustment; the first loss value is calculated based on the target output and the actual output of the graph-generated text model.

[0081] The formatted training files corresponding to the unformatted training files in the first training data set are used as the input of the graph-based text model after the first parameter adjustment, and the unformatted training files in the first training data set are used as the target output of the graph-based text model after the first parameter adjustment. With the purpose of minimizing the second loss value, parameter adjustment training is performed on the graph-based text model after the first parameter adjustment to obtain a graph-based text model after the second parameter adjustment; the second loss value is calculated based on the target output and the actual output of the graph-based text model after the first parameter adjustment.

[0082] The graph-to-text model adopts a two-stage parameter adjustment training mechanism, using the second training dataset for preliminary optimization to reduce recognition errors; and then using the first training dataset for fine-tuning to improve structural restoration.

[0083] S205: Parameter adjustment training is performed on the large language model using the third training data set with the goal of minimizing the third loss value to obtain a large language model with adjusted parameters.

[0084] This application does not limit the specific training process of the large language model. For example, the formatted training file in the third training data set is used as the input of the large language model, and the verified formatted training file in the third training data set is used as the target output of the large language model. With the purpose of minimizing the third loss value, the large language model is trained by parameter adjustment to obtain the large language model after parameter adjustment; the third loss value is calculated based on the target output and the actual output of the large language model.

[0085] A large language model was introduced and optimized using a third training dataset. This dataset, which includes both the original formatted files and the verified high-quality formatted files, guides the large language model in learning how to correct grammatical errors and optimize expression logic. This significantly improves the readability and professionalism of the final output document.

[0086] S206: Using the first training data set, performing parameter adjustment training on the Wensheng graph model with the goal of minimizing the fourth loss value, to obtain the Wensheng graph model after the first parameter adjustment.

[0087] This application does not limit the first training process of the Vincent graph model. For example, the formatted training files in the first training data set are used as the input of the Vincent graph model, and the unformatted training files corresponding to the formatted training files in the first training data set are used as the target output of the Vincent graph model. With the goal of minimizing the fourth loss value, the Vincent graph model is trained by parameter adjustment to obtain the Vincent graph model after the first parameter adjustment; the fourth loss value is calculated based on the target output and the actual output of the Vincent graph model.

[0088] S207: Using the second training data set, with the goal of minimizing the fifth loss value, perform parameter adjustment training on the Wensheng graph model after the first parameter adjustment to obtain the Wensheng graph model after the second parameter adjustment.

[0089] The present application does not limit the second training process of the Vincent graph model. For example, the formatted training file corresponding to the unformatted training file in the second training data set is used as the input of the Vincent graph model after the first parameter adjustment, and the unformatted training file in the second training data set is used as the target output of the Vincent graph model after the first parameter adjustment. With the goal of minimizing the fifth loss value, the Vincent graph model after the first parameter adjustment is parameter-adjusted and trained to obtain the Vincent graph model after the second parameter adjustment. The fifth loss value is calculated based on the target output of the Vincent graph model after the first parameter adjustment and the actual output of the Vincent graph model after the first parameter adjustment.

[0090] The Wensheng graph model also adopts a two-step training strategy to ensure its stability and accuracy under different input conditions.

[0091] S208: Using the verification model and the second training data set, the image-to-text model after the second parameter adjustment and the text-to-image model after the second parameter adjustment are verified and adjusted to obtain a final image-to-text model and a final text-to-image model.

[0092] This application does not limit the verification and adjustment process of the graph-text model after the second parameter adjustment and the text-graph model after the second parameter adjustment. For example, Figure 3This is a second schematic diagram of the training process provided in the embodiment of the present application, as shown in FIG. Figure 3 As shown, S301: the unformatted training files in the second training data set are used as the input of the graph-to-text model after the second parameter adjustment, and the formatted training files corresponding to the unformatted training files in the second training data set are used as the input of the text-to-graph model after the second parameter adjustment, and the outputs corresponding to the preset layer of the graph-to-text model after the second parameter adjustment and the text-to-graph model after the second parameter adjustment are respectively obtained.

[0093] S302: Using the output corresponding to the preset layer of the graph-text model after the second parameter adjustment as the first input of the verification model, and using the output corresponding to the preset layer of the text-graph model after the second parameter adjustment as the second input of the verification model to obtain the output of the verification model.

[0094] This application does not limit the structure of the verification model. For example, the verification model consists of two branches and is a double-tower structure. The two branches have the same structure and parameters. Each branch consists of two parts. The first part has 3 branches, each of which consists of a 4-layer convolutional neural network. The convolution kernel size of each layer is set to 7*1, 5*1, 3*1 and 1*1. The second part includes 4 fully connected layers. The number of nodes in the first layer is the same as the convolutional neural network output of the first part. The number of nodes in the second layer is 1024, the number of nodes in the third layer is 512, and the number of nodes in the fourth layer is 128. During training, the loss function of this part can use the L1 loss function, which measures the difference in the vectors output by the two branches, and the goal is to minimize the difference.

[0095] S303: Check whether the output of the verification model meets the iteration end condition.

[0096] S304: If the output of the verification model does not meet the iteration end condition, the parameters of the graph-text model after the second parameter adjustment and the text-graph model after the second parameter adjustment are adjusted, and the process returns to S301.

[0097] S305: If the output of the verification model meets the iteration end condition, the iteration ends to obtain the final graph-to-text model and the final text-to-graph model.

[0098] The above examples discuss the model training process in detail. After model training is complete, a trained graph-to-text model and a large language model are obtained. The trained graph-to-text model and the large language model are deployed online and can be called upon by business personnel as needed.

[0099] The above embodiment trains the model with the goal of minimizing the loss value. As an optional embodiment, the model can also be trained based on the number of training times. For example, if the graph-based text model and the large language model have undergone overall parameter adjustment training, when faced with a new training set, the trained graph-based text model and the large language model can be fine-tuned, specifically including:

[0100] The first training data set, the second training data set and the third training data set are constructed; the third training data set includes a formatted training file and a verified formatted training file.

[0101] Build graph-to-text models, large language models, text-to-graph models, and verification models.

[0102] The graph-based model is fine-tuned using the second training dataset for approximately 10 epochs to obtain a graph-based model after the first parameter adjustment. One epoch means that all simulated data has passed through the model once.

[0103] The first training data set is used to train for approximately 20 epochs, and parameter fine-tuning training is performed on the graph-to-text model after the first parameter adjustment to obtain the graph-to-text model after the second parameter adjustment.

[0104] The third training data set is used to train the large language model for approximately 20 epochs to perform parameter fine-tuning training to obtain a large language model with adjusted parameters.

[0105] The first training data set is used to train the Wensheng graph model for approximately 20 epochs, thereby performing parameter fine-tuning training on the Wensheng graph model to obtain the Wensheng graph model after the first parameter adjustment.

[0106] The second training data set is used to train for approximately 20 epochs to perform parameter fine-tuning training on the Wensheng graph model after the first parameter adjustment to obtain the Wensheng graph model after the second parameter adjustment.

[0107] The verification model and the second training data set are used to verify and adjust the image-to-text model after the second parameter adjustment and the text-to-image model after the second parameter adjustment to obtain a final image-to-text model and a final text-to-image model.

[0108] This application achieves unified modeling and processing for multiple business scenarios by building a universal model structure. This avoids the repetitive work of developing, training, and maintaining models for each specific task. This significantly reduces the workload of technical personnel and the management burden of operations and maintenance personnel, thereby significantly reducing manpower investment and operating costs. Only one model needs to be trained and deployed to meet the needs of multiple application scenarios. Compared to running multiple models in parallel, this application is more lightweight in terms of training resources, computing overhead, and deployment complexity. It also supports rapid iteration and flexible expansion to adapt to changing business needs.

[0109] Under the unified model framework, data from different business scenarios can be integrated and trained; data silos can be broken down, allowing the model to learn common features under a wider data distribution; the model's generalization ability can be improved, enhancing its adaptability and robustness to new scenarios; and a transition from "data decentralized utilization" to "data collaborative drive" can be achieved, fully leveraging the potential value of multi-source heterogeneous data. This application avoids information loss and error propagation in the intermediate links through end-to-end modeling with a single model; reduces the uncertainty caused by model switching or combination, improves the overall system's prediction stability and result reliability; and demonstrates higher accuracy and lower error rates in practical applications.

[0110] The model interface is well-encapsulated, with a simple and intuitive user interface. Business personnel only need to enter basic parameters or upload raw data to obtain the desired results. This significantly reduces the barrier to entry for users, allowing even non-technical personnel to efficiently apply AI technology to their daily work and enhance overall business intelligence. This effectively reduces labor costs, allowing a single model to address all scenarios, making training, deployment, and maintenance easier. Data from different scenarios can be combined to maximize its potential. A single model can achieve the desired results, reducing error accumulation and increasing accuracy. It is simple to use, and business personnel do not need to understand any relevant technical details.

[0111] Figure 4 This is a structural diagram of an optical character recognition device provided in an embodiment of the present application, such as Figure 4 As shown, based on the optical character recognition method provided in the above embodiment, the present application also provides an optical character recognition device including:

[0112] The acquisition module is used to acquire unformatted files; the unformatted files at least include PDF files, pictures and tables.

[0113] A format conversion module is configured to input the unformatted file into a graphic text model to obtain a formatted file corresponding to the unformatted file; the formatted file includes at least a Word file; the graphic text model is obtained through two trainings and one verification; the two trainings include training the graphic text model using a first training data set and training the graphic text model using a second training data set; the one verification includes verifying and training the graphic text model using a verification model and a second training data set; the first training data set includes a formatted training file and an unformatted training file corresponding to the formatted training file; the second training data set includes an unformatted training file and a formatted training file corresponding to the unformatted training file.

[0114] The verification module is used to verify the formatted file using a large language model to obtain a verified file.

[0115] As an optional embodiment, the device further includes:

[0116] The data set construction module is used to construct the first training data set, the second training data set and the third training data set. The third training data set includes a formatted training file and a verified formatted training file.

[0117] The model building module is used to build graph-to-text models, large language models, text-to-graph models, and verification models.

[0118] The first training module is used to use the second training data set to perform parameter adjustment training on the graph-to-text model with the goal of minimizing the first loss value, so as to obtain the graph-to-text model after the first parameter adjustment.

[0119] The second training module is used to use the first training data set to perform parameter adjustment training on the graph-to-text model after the first parameter adjustment with the purpose of minimizing the second loss value, so as to obtain the graph-to-text model after the second parameter adjustment.

[0120] The third training module is used to use the third training data set to perform parameter adjustment training on the large language model with the purpose of minimizing the third loss value, so as to obtain the large language model with adjusted parameters.

[0121] The fourth training module is used to use the first training data set to perform parameter adjustment training on the Wensheng graph model with the goal of minimizing a fourth loss value, thereby obtaining the Wensheng graph model after the first parameter adjustment.

[0122] The fifth training module is used to perform parameter adjustment training on the Wensheng graph model after the first parameter adjustment using the second training data set with the goal of minimizing the fifth loss value, so as to obtain the Wensheng graph model after the second parameter adjustment.

[0123] The sixth training module is used to verify and adjust the image-to-text model and the text-to-image model after the second parameter adjustment using the verification model and the second training data set to obtain the final image-to-text model and the final text-to-image model.

[0124] As an optional embodiment, the second training module is specifically configured to:

[0125] The formatted training files corresponding to the unformatted training files in the first training data set are used as inputs to the graph-based text model after the first parameter adjustment, and the unformatted training files in the first training data set are used as target outputs of the graph-based text model after the first parameter adjustment. Parameter adjustment training is performed on the graph-based text model after the first parameter adjustment with the goal of minimizing a second loss value, thereby obtaining a graph-based text model after the second parameter adjustment. The second loss value is calculated based on the target output and the actual output of the graph-based text model after the first parameter adjustment.

[0126] As an optional embodiment, the sixth training module specifically includes:

[0127] The first input unit is used to use the unformatted training files in the second training data set as the input of the graph-to-text model after the second parameter adjustment, and use the formatted training files corresponding to the unformatted training files in the second training data set as the input of the text-to-graph model after the second parameter adjustment, to respectively obtain the output corresponding to the graph-to-text model after the second parameter adjustment and the preset layer of the text-to-graph model after the second parameter adjustment.

[0128] The second input unit is used to use the output corresponding to the preset layer of the graph-text model after the second parameter adjustment as the first input of the verification model, and use the output corresponding to the preset layer of the text-graph model after the second parameter adjustment as the second input of the verification model to obtain the output of the verification model.

[0129] The first judgment unit is configured to terminate the iteration if the output of the verification model satisfies an iteration termination condition, thereby obtaining a final image-to-text model and a final text-to-image model.

[0130] The second judgment unit is configured to adjust the parameters of the graph-text model after the second parameter adjustment and the text-graph model after the second parameter adjustment if the output of the verification model does not meet the iteration end condition, and return to the first input unit.

[0131] As an optional embodiment, the graph-to-text model includes a convolutional neural network and a recurrent neural network.

[0132] As an optional embodiment, the text graph model includes a Transformer model.

[0133] An embodiment of the present application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement an optical character recognition method.

[0134] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, an optical character recognition method is implemented.

[0135] An embodiment of the present application provides a computer program product, including a computer program, which implements an optical character recognition method when executed by a processor.

[0136] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments. The device and apparatus embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0137] The above is merely one specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An optical character recognition method, characterized in that: The optical character recognition method comprises: Obtaining an unformatted file; the unformatted file includes at least a PDF file, a picture, and a table; Inputting the unformatted file into a graphic text model to obtain a formatted file corresponding to the unformatted file; the formatted file includes at least a Word file; the graphic text model is obtained through two trainings and one verification; the two trainings include training the graphic text model using a first training data set and training the graphic text model using a second training data set; the one verification includes verifying and training the graphic text model using a verification model and a second training data set; the first training data set includes a formatted training file and an unformatted training file corresponding to the formatted training file; the second training data set includes the unformatted training file and a formatted training file corresponding to the unformatted training file; The formatted file is verified using a large language model to obtain a verified file.

2. The optical character recognition method according to claim 1, wherein: The method further comprises: Constructing the first training data set, the second training data set, and the third training data set; the third training data set includes a formatted training file and a verified formatted training file; Build graph-to-text models, large language models, text-to-graph models, and verification models; Using the second training data set, with the goal of minimizing the first loss value, to perform parameter adjustment training on the graph-to-text model, thereby obtaining a graph-to-text model after the first parameter adjustment; Using the first training data set, with the goal of minimizing the second loss value, performing parameter adjustment training on the graph-to-text model after the first parameter adjustment, to obtain a graph-to-text model after the second parameter adjustment; Using the third training data set, with the goal of minimizing the third loss value, to perform parameter adjustment training on the large language model, to obtain a large language model after parameter adjustment; Using the first training data set, with the goal of minimizing the fourth loss value, performing parameter adjustment training on the Vincent graph model to obtain the Vincent graph model after the first parameter adjustment; Using the second training data set, with the goal of minimizing the fifth loss value, performing parameter adjustment training on the Vincent graph model after the first parameter adjustment, to obtain the Vincent graph model after the second parameter adjustment; The verification model and the second training data set are used to verify and adjust the image-to-text model after the second parameter adjustment and the text-to-image model after the second parameter adjustment to obtain a final image-to-text model and a final text-to-image model.

3. The optical character recognition method according to claim 2, wherein: The step of performing parameter adjustment training on the graph-to-text model after the first parameter adjustment using the first training data set with the goal of minimizing the second loss value to obtain the graph-to-text model after the second parameter adjustment specifically includes: The formatted training files corresponding to the unformatted training files in the first training data set are used as the input of the graph-based text model after the first parameter adjustment, and the unformatted training files in the first training data set are used as the target output of the graph-based text model after the first parameter adjustment. With the purpose of minimizing the second loss value, parameter adjustment training is performed on the graph-based text model after the first parameter adjustment to obtain a graph-based text model after the second parameter adjustment; the second loss value is calculated based on the target output and the actual output of the graph-based text model after the first parameter adjustment.

4. The optical character recognition method according to claim 2, wherein: The step of using the verification model and the second training data set to verify and adjust the image-to-text model after the second parameter adjustment and the text-to-image model after the second parameter adjustment to obtain a final image-to-text model and a final text-to-image model specifically includes: Using the unformatted training files in the second training data set as input to the graph-to-text model after the second parameter adjustment, and using the formatted training files corresponding to the unformatted training files in the second training data set as input to the text-to-graph model after the second parameter adjustment, respectively obtaining the output corresponding to the graph-to-text model after the second parameter adjustment and the preset layer of the text-to-graph model after the second parameter adjustment; Using the output corresponding to the preset layer of the graph-text model after the second parameter adjustment as the first input of the verification model, and using the output corresponding to the preset layer of the text-graph model after the second parameter adjustment as the second input of the verification model to obtain the output of the verification model; If the output of the verification model meets the iteration end condition, the iteration ends to obtain the final graph-text model and the final text-graph model; If the output of the verification model does not meet the iteration end condition, the parameters of the graph-text model after the second parameter adjustment and the culture-graph model after the second parameter adjustment are adjusted, and the step of returning to the step of using the unformatted training file in the second training data set as the input of the graph-text model after the second parameter adjustment, and using the formatted training file corresponding to the unformatted training file in the second training data set as the input of the culture-graph model after the second parameter adjustment, and obtaining the output corresponding to the preset layer of the graph-text model after the second parameter adjustment and the culture-graph model after the second parameter adjustment respectively.

5. The optical character recognition method according to claim 1, wherein: The graph-to-text model includes a convolutional neural network and a recurrent neural network.

6. The optical character recognition method according to claim 2, wherein: The text graph model includes a Transformer model.

7. An optical character recognition device, characterized in that: The optical character recognition device comprises: An acquisition module is used to acquire unformatted files; the unformatted files include at least PDF files, pictures and tables; A format conversion module is configured to input the unformatted file into a graphic text model to obtain a formatted file corresponding to the unformatted file; the formatted file includes at least a Word file; the graphic text model is obtained through two trainings and one verification; the two trainings include training the graphic text model using a first training data set and training the graphic text model using a second training data set; the one verification includes verifying and training the graphic text model using a verification model and a second training data set; the first training data set includes a formatted training file and an unformatted training file corresponding to the formatted training file; the second training data set includes the unformatted training file and a formatted training file corresponding to the unformatted training file; The verification module is used to verify the formatted file using a large language model to obtain a verified file.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the optical character recognition method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the optical character recognition method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the optical character recognition method according to any one of claims 1 to 6 is implemented.

Images