A method and apparatus for correcting text content

Abstract

This application discloses a method and apparatus for text content correction, addressing the technical problem that existing technologies cannot meet the scalability and customization requirements of different application scenarios when performing text content verification. The method includes: acquiring an image or document to be recognized; inputting the image or document into an OCR library to obtain editable electronic text; parsing the editable electronic text using grammar rules contained in ANTLR4 to generate a syntax tree and intermediate representation; learning the intermediate representation using a machine learning algorithm, and traversing the syntax tree according to the learning results to perform content verification on the nodes of the syntax tree; and automatically correcting the verified editable electronic text according to business verification rules to obtain the final text. By organically combining OCR, ANTLR4, and machine learning technologies, an efficient, accurate, and intelligent content verification solution is provided, with significant scalability and flexibility.

Description

Technical Field

[0001] This application relates to the field of text processing technology, and in particular to a method and apparatus for correcting text content. Background Technology

[0002] Traditional text content verification methods primarily rely on manual data entry, checking, and proofreading, which is time-consuming and error-prone. While OCR technology can quickly and accurately convert images to electronic text, it only provides surface-level information without in-depth understanding and verification of the content. ANTLR4 is a powerful parsing tool capable of structural and syntactic analysis of text based on grammatical rules. However, ANTLR4 itself does not involve content verification mechanisms or intelligent implementation.

[0003] Currently, users cannot customize different verification and syntax rules based on the characteristics of different domains when performing text content verification. They also cannot freely choose whether to use extensions to further process the content to obtain verified text based on specific scenarios, thus failing to achieve comprehensive, efficient, and intelligent content verification. Therefore, how to provide a correction system that can automatically identify and learn the characteristics and patterns of text content, enabling users to customize verification rules and extension requirements according to their needs, thereby improving the system's scalability and flexibility, is a key issue currently faced. Summary of the Invention

[0004] This application provides a text content correction method and device to achieve efficient and intelligent text content verification and correction, meeting the needs of high scalability and customization.

[0005] On one hand, embodiments of this application provide a method for correcting text content, the method comprising:

[0006] Obtain the image or document to be recognized;

[0007] Input the image or document to be recognized into the OCR library to obtain editable electronic text;

[0008] The editable electronic text is parsed using the grammar rules contained in ANTLR4 to generate a syntax tree and an intermediate representation; wherein the intermediate representation is intermediate code containing the content of the editable electronic text.

[0009] The intermediate representation is learned using a machine learning algorithm, and the syntax tree is traversed based on the learning results to perform content validation on the nodes of the syntax tree.

[0010] The editable electronic text is automatically corrected according to the business verification rules to obtain the final text.

[0011] Furthermore, the image or document to be identified is in PNG image format, JPEG image format, GIF image format, JPG image format, or PDF document format.

[0012] Furthermore, the OCR library is implemented based on the open-source Tess4J library, which uses the Tesseract OCR engine by encapsulating it in Java and calling the Tess4J API.

[0013] Furthermore, before parsing the editable electronic text using the syntax rules included in ANTLR4, the method further includes:

[0014] The editable electronic text is preprocessed, and the preprocessing includes at least one of the following: regular expression validation, spell checking, grammar checking and context analysis of the text content using text correction tools.

[0015] Furthermore, the syntax tree is traversed using the Visitor pattern of ANTLR4.

[0016] Furthermore, the Visitor mode of ANTLR4 supports manual control of the traversal order.

[0017] Furthermore, before obtaining the final text, the method also includes:

[0018] Machine learning algorithms are used to perform rule-based and statistical error correction on the content of the editable electronic text.

[0019] Furthermore, the method also includes:

[0020] The machine learning algorithm is used to automatically complete the content of the editable electronic text.

[0021] Furthermore, after obtaining the final text, the method further includes:

[0022] The final text is displayed in the business system, and the displayed content of the final text includes the error-marked parts. On the other hand, embodiments of this application also provide a text content correction device, the device comprising:

[0023] At least one processor; and,

[0024] A memory communicatively connected to the at least one processor; wherein,

[0025] The memory stores instructions that can be executed by the at least one processor, enabling the at least one processor to perform the following:

[0026] Obtain the image or document to be recognized;

[0027] Input the image or document to be recognized into the OCR library to obtain editable electronic text;

[0028] The editable electronic text is parsed using the grammar rules contained in ANTLR4 to generate a syntax tree and an intermediate representation; wherein the intermediate representation is intermediate code containing the content of the editable electronic text.

[0029] The intermediate representation is learned using a machine learning algorithm, and the syntax tree is traversed based on the learning results to perform content validation on the nodes of the syntax tree.

[0030] The editable electronic text is automatically corrected according to the business verification rules to obtain the final text.

[0031] The text content correction method and device provided in this application have the following beneficial effects:

[0032] By organically combining OCR, ANTLR4, and machine learning technologies, this solution provides a highly efficient and accurate content verification solution. OCR technology replaces traditional manual input, enabling automatic and efficient conversion of images and documents into editable electronic text. ANTLR4 performs lexical and grammatical analysis, allowing users to customize ANTLR4 grammar rules according to different business scenarios, extracting and verifying the structure and rules of text content. This allows for flexible and highly scalable implementation tailored to different business needs. The combination of machine learning and ANTLR4 enables the automatic learning and reinforcement of text content features, ultimately achieving intelligent and automated content verification. Based on machine learning, it can also self-process and correct or complete text content that fails verification, providing users with the correct text content to choose from, offering convenience and significant scalability and flexibility. Attached Figure Description

[0033] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0034] Figure 1 A flowchart illustrating a text content correction method provided in this application embodiment;

[0035] Figure 2 This is a schematic diagram of a text content correction device provided in an embodiment of this application. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] This application provides a text content correction method to address the shortcomings of existing technologies in meeting the scalability and customization requirements of different usage scenarios when performing text content verification.

[0038] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0039] Figure 1 This is a flowchart illustrating a text content correction method provided in an embodiment of this application. Figure 1 As shown in the embodiment of this application, a method for correcting text content specifically includes the following steps:

[0040] Step 1: Obtain the image or document to be recognized.

[0041] The image or document to be identified can be in PNG, JPEG, GIF, JPG, or PDF format.

[0042] Step 2: Input the image or document to be recognized into the OCR library to obtain editable electronic text.

[0043] Among them, OCR (Optical Character Recognition) refers to the process by which electronic devices (such as scanners or digital cameras) examine characters printed on paper, determine their shapes by detecting dark and light patterns, and then translate the shapes into computer text using character recognition methods.

[0044] Typically, general-purpose OCR models based on neural networks simulate image text in real-world scenes, combining background images, dictionaries, glossaries, and font libraries for various language styles to artificially synthesize labeled image text into a training dataset. This training dataset is then used to initially train a constructed OCR language recognition neural network, resulting in an OCR text recognition model. However, using this type of general-purpose OCR model based on neural networks often results in insufficient accuracy in recognizing text due to the strong generalization ability of neural networks, failing to meet application requirements.

[0045] Therefore, this embodiment selects a stable, easy-to-use development toolkit with a complete API for OCR operation. This patent is based on the open-source Tess4J library, a Java wrapper for the Tesseract OCR engine. This allows the use of Tesseract OCR by calling the Tess4J API, supporting various common image formats such as PNG, JPEG, GIF, and JPG, as well as PDF document formats. The acquired image or document to be recognized is used as input, and the OCR library converts the input into editable electronic text, thus making the acquired text data more accurate.

[0046] To convert the content of a PDF document into editable electronic text, the PDF document must first be parsed to obtain its text paragraph structure, table structure, and figures in order to further extract structured information. Current open-source parsing tools such as xpdf, pdfminer, and pypdf2 cannot perform this task well and have various problems, such as: inability to distinguish between figures, tables, and text paragraphs; inaccurate extraction of table content; inability to identify the number of rows and columns in a table, or the content of cells in specific rows and columns; and for multi-column PDF documents, text paragraphs can sometimes appear disorganized.

[0047] This embodiment applies a PDF file parsing method based on Tesseract OCR. The method first uses the Fitz toolkit to convert the PDF file into an image sequence. Then, for each image in the image sequence, the TableBank tool is used to obtain multiple area locations of the table, TableArea[1,...n], where n is the total number of tables in the image. Finally, Tesseract OCR is used to extract the image from the image, extract and recognize the table cells. Using Tesseract OCR to extract images, extract and recognize table cells, the specific steps are as follows: (1) Perform morphological preprocessing and connected component analysis to obtain lines, image regions and text blocks; (2) Detect the end position of text lines to obtain the start and end positions of each line of text; (3) Discover column layout to obtain the order of text; (4) Obtain each block, which has various types, including text blocks, image blocks and table blocks, each block has multiple text lines; (5) Process the table, use TableBank to obtain the multiple area positions of the table TableArea to replace the table blocks in step (4); use the lines in the TableArea[i] area to separate the text lines in TableArea[i] into each cell to complete the recognition of table cells; (6) Use the LSTM model to recognize each line of each text block and each line of each cell of the table to obtain the position and content of each word. At this time, the text in the table has the table index and the cell row and column index.

[0048] When using Tesseract OCR to parse the text content of PNG, JPEG, GIF, and JPG image files, the step of converting the file into an image sequence using the Fitz toolkit is omitted; the remaining steps are basically the same as using Tesseract OCR to parse PDF documents.

[0049] Furthermore, the text content recognized by OCR undergoes simple preprocessing and correction, mainly including simple regular expression verification, and the use of some text correction tools for simple spell checking, grammar checking, and context analysis to ensure recognition accuracy.

[0050] Step 3: Use the grammar rules contained in ANTLR4 to parse the editable electronic text and generate a syntax tree and intermediate representation; wherein, the intermediate representation is intermediate code containing the content of the editable electronic text.

[0051] In this context, intermediate representation is a data structure or code used to represent source code in a way that facilitates further processing. Typically, source code needs to be parsed each time a tool analyzes it, and intermediate representations are usually stored permanently, eliminating the need for further source code parsing.

[0052] This embodiment uses ANTLR4, a powerful parser tool capable of recognizing, processing, and translating structured text, binary files, and even XML files. It provides a framework that can contain grammatical descriptions of Java and C++ actions to construct language recognizers, compilers, and parsers. ANTLR4 automatically generates a lexer and a parser by parsing user-defined grammar files, and automatically processes the input text into a parse tree using a tree parser. The automatic creation and traversal mechanism of parse trees introduced by ANTLR4 greatly improves the development efficiency of language recognition programs, freeing developers from complex compiler theory.

[0053] This embodiment performs lexical and syntactic analysis on a custom grammar, converting it into an abstract syntax tree (AST), and then performs semantic analysis based on the AST. Specifically, the ANTLR4 tool is used to perform syntactic analysis and parsing on OCR-recognized text content such as serial numbers, manufacturers, production dates, and assembly parts according to custom grammar rules (including lexical and syntactic rules), generating a syntax tree and intermediate representations for convenient data analysis and processing. This includes two aspects: generating the grammar rule file corresponding to the lexical and syntactic analyzers, and generating the syntax tree and intermediate representations.

[0054] Furthermore, users can customize grammar and lexical rule files according to the characteristics of different domains. Using the ANTLR4 tool, they can generate lexical analyzers and code parsers for analyzing and editing electronic text. The lexical and parsing tools then perform lexical analysis and syntax parsing on the preprocessed editable electronic text, generating a parsing tree and intermediate representation. After inputting a user-defined grammar rule file, the ANTLR4 tool receives and executes the processing instructions within the grammar rule file. The lexical and parsing tools then generate the corresponding parsing tree from the processed file and produce structured data containing the parsed content.

[0055] Furthermore, the ANTLR4 Visitor mode is used to traverse the syntax tree. The ANTLR4 Visitor mode supports manual control of the traversal order, which can flexibly control the access to tree nodes and supports simple operations and modifications to the traversed nodes. Data assembly is completed through a depth-first traversal of each node.

[0056] Step 4: Use machine learning algorithms to learn the intermediate representation, and traverse the syntax tree according to the learning results to perform content validation on the nodes of the syntax tree.

[0057] This process involves using machine learning algorithms to train and classify the intermediate text content, identifying and verifying its correctness, compliance, or other constraints, such as ensuring a standard part only has one serial number and one production date. Different machine learning algorithms are selected for different scenarios; for example, a simple logistic regression algorithm is used for binary classification problems. Data preprocessing and cleaning are required before model training to ensure data quality. Providing as much historical data as possible ensures the dataset contains representative and diverse examples so the model can learn from a wide range of situations.

[0058] When using machine learning algorithms to learn from intermediate text representations, the data is divided into training and test sets. The training set serves as the sample set for training the parameters of the neural network model; the test set is used to test the trained model and objectively evaluate its performance. To ensure good performance on the test set, the data in both the training and test sets must be consistent. Similarly, to ensure good performance in real-world applications, the distribution of the training set must be consistent with the data distribution used in actual applications. Therefore, training the selected neural network model requires collecting as much relevant historical data as possible. However, sometimes, when data volume is insufficient, a training set must be generated from existing datasets. Otherwise, an insufficient number of samples in the training set will negatively impact the model's training effectiveness.

[0059] If the number of samples is insufficient, samples can be drawn from both the test set and each of the training sets. Features of the test set and each training set are then determined based on the drawn samples. The feature distribution of the test set and multiple training sets is determined based on these features. Samples are then drawn from the multiple training sets based on the feature distribution to form a new training set, generating a new training set with a feature distribution aligned with the feature distribution of the test set. This method ensures that the data distribution of the new training set is aligned with the data distribution of the test set, guaranteeing the model's training effectiveness and achieving good performance on the test set. Furthermore, a new training set can be generated from the existing training set, eliminating the need to acquire additional training sets and saving on training data acquisition costs.

[0060] Step 5: Automatically correct the editable electronic text after verification according to the business verification rules to obtain the final text.

[0061] Specifically, machine learning techniques are used to automatically process or complete text that fails content verification, enabling it to comply with relevant business verification rules. When automatically correcting editable electronic text, different generation model algorithms can be selected for rule-based or statistical error correction. Alternatively, text content can be automatically completed based on training a large-scale text corpus, resulting in correct text content that users can choose from. The generated new text is then presented to the user as a suggested option.

[0062] Furthermore, this step can be interconnected with business systems, allowing users to freely choose whether to use extensions to further process text content based on specific scenarios. By adapting to different extensions, the correct text content required by the user can be obtained, demonstrating great scalability and flexibility.

[0063] Furthermore, after obtaining the final text, different categories of final text can be displayed in the business system. If the verification fails, the error-marked portion is presented to the user in the business system, along with further processing suggestions. The final text can be the correct text automatically processed using the method in this embodiment or the verified text manually processed by the user based on the error markings and processing suggestions.

[0064] This embodiment combines OCR recognition with the ANTLR4 syntax parsing tool and machine learning technology to provide an efficient and accurate content verification solution. During data processing, it utilizes extensions of the parsing tool combined with machine learning to correct and revise text content. This embodiment uses OCR technology to convert text in images or scanned documents into editable electronic text. ANTLR4 is used for lexical and syntactic analysis to extract and verify the structure and rules of the text content. Machine learning is used to train and classify the text content, ultimately achieving intelligent and automated content verification. This solution can be widely applied to various scenarios requiring content verification, such as factory label checks, document proofreading, and code review. Users can freely choose whether to use extensions of the parsing tool to replace or restore text that does not meet verification requirements, ultimately generating supplementary text content that conforms to the verification rules for the user to choose whether to apply.

[0065] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a text content correction device, the structure of which is as follows: Figure 2 As shown.

[0066] Figure 2 This is a schematic diagram of a text content correction device provided in an embodiment of this application. Figure 2 As shown, the device includes:

[0067] At least one processor 201; and,

[0068] The memory 202 is communicatively connected to the at least one processor 201; wherein,

[0069] The memory 202 stores instructions that can be executed by the at least one processor 201, so that the at least one processor 201 can execute:

[0070] Obtain the image or document to be recognized;

[0071] Input the image or document to be recognized into the OCR library to obtain editable electronic text;

[0072] The editable electronic text is parsed using the grammar rules contained in ANTLR4 to generate a syntax tree and an intermediate representation; wherein the intermediate representation is intermediate code containing the content of the editable electronic text.

[0073] The intermediate representation is learned using a machine learning algorithm, and the syntax tree is traversed based on the learning results to perform content validation on the nodes of the syntax tree.

[0074] The editable electronic text is automatically corrected according to the business verification rules to obtain the final text.

[0075] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0076] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0077] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method of correcting text content, characterized by, The method includes: Obtain the image or document to be recognized; Input the image or document to be recognized into the OCR library to obtain editable electronic text; The editable electronic text is parsed using the grammar rules contained in ANTLR4 to generate a syntax tree and an intermediate representation; wherein the intermediate representation is intermediate code containing the content of the editable electronic text. The intermediate representation is learned using a machine learning algorithm, and the syntax tree is traversed using the Visitor mode of ANTLR4 based on the learning results to perform content validation on the nodes of the syntax tree. The editable electronic text after verification is automatically corrected according to the business verification rules to obtain the final text; Before obtaining the final text, the process also includes: using machine learning algorithms to perform rule-based and statistical error correction on the content of the editable electronic text, and using machine learning algorithms to automatically complete the content of the editable electronic text.

2. The method of claim 1, wherein, The image to be identified is in PNG, JPEG, GIF, or JPG format, and the document to be identified is in PDF format.

3. The method of claim 1, wherein the step of correcting the text content comprises the steps of: The OCR library is implemented based on the open-source Tess4J library, which uses TesseractOCR by wrapping the TesseractOCR engine in Java and calling the Tess4J API. ​ 4. The method of claim 1, wherein the step of correcting the text content comprises the steps of: Before parsing the editable electronic text using the syntax rules included in ANTLR4, the method further includes: ​ The editable electronic text is preprocessed, and the preprocessing includes at least one of the following: regular expression validation, spell checking, grammar checking and context analysis of the text content using text correction tools.

5. The method of claim 1, wherein the step of correcting the text content comprises the steps of: determining a word in the text content; and correcting the word in the text content. The Visitor mode of ANTLR4 supports manual control of the traversal order.

6. The method of claim 1, wherein, After obtaining the final text, the method further includes: The final text is displayed in the business system, and the displayed content of the final text includes the error-marked parts.

7. A text content correction device characterized by comprising: The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform a text content correction method according to any one of claims 1-6.

Images