A Multimodal Document Automatic Proofreading Method and System Based on Artificial Intelligence

Abstract

This application discloses an automatic multimodal document proofreading method and system based on artificial intelligence, relating to the field of data processing technology. The disclosed method includes: receiving an original document and converting it into structured data composed of multiple data types in different formats; selecting a corresponding verification model to proofread the structured data according to each data type to obtain the differences corresponding to each data type; determining the risk level of each difference through a preset quality grader; and correcting each difference based on the risk level. This application achieves automatic correction of structured data by selecting a corresponding verification model to proofread the structured data according to different data types in different formats, determining the risk level of the obtained differences through a preset quality grader, and correcting the differences based on the risk level. Compared with manual proofreading, this effectively improves proofreading efficiency.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and system for automatic proofreading of multimodal documents based on artificial intelligence. Background Technology

[0002] With the current explosion of artificial intelligence and big data technologies, a large number of technical documents need to be processed into structured formats before they can be used. After the documents are structured, the content needs to be proofread to ensure the accuracy of the data. Currently, this mainly relies on manual verification of the structured data word by word against the original documents. However, this is limited by human energy and time, making it difficult to efficiently process large-scale structured data, resulting in low proofreading efficiency. Summary of the Invention

[0003] The main purpose of this application is to provide a multimodal document automatic proofreading method and system based on artificial intelligence, which aims to solve the technical problem that existing technologies rely on manual word-by-word proofreading and have low proofreading efficiency.

[0004] To achieve the above objectives, this application proposes an AI-based multimodal document automatic proofreading method, which includes:

[0005] Receive the original document and convert it into structured data consisting of multiple data types in different formats;

[0006] Based on the data of each of the aforementioned format types, the corresponding verification model is selected to verify the structured data, thereby obtaining the difference items corresponding to the data of each of the aforementioned format types;

[0007] The risk level corresponding to each of the aforementioned differences is determined by a preset quality grader;

[0008] The differences are adjusted according to the respective risk levels.

[0009] In one embodiment, the format types include at least JSON, CSV, LaTeX, and image formats. The step of receiving the original document and converting the original document into structured data composed of data of multiple different format types includes:

[0010] Receive the original document and preprocess it to obtain the preprocessed document;

[0011] Identify text blocks, table areas, image areas, and formula areas in the preprocessed document;

[0012] The text block is identified using an optical character recognition model to obtain the JSON-formatted text data;

[0013] The original tables in the table area are identified by a preset table parsing model to obtain the table data in CSV format, which includes the extracted tables corresponding to each of the original tables.

[0014] The original formula images in the formula area are identified by a preset formula recognition model to obtain the formula data in LaTeX format. The formula data includes the extracted formulas corresponding to each original formula image.

[0015] Each original image in the image region is cropped to obtain image data in the image format, and the image data includes the cropped image corresponding to each original image.

[0016] In one embodiment, the differences include at least text differences, table differences, formula differences, and image differences. The step of selecting the corresponding verification model to verify the structured data according to each of the format types to obtain the differences corresponding to each of the format types includes:

[0017] The text data is validated using a preset text validation model, a pre-trained language model, and preset formatting rules to obtain the text difference items;

[0018] The table data is validated using a preset table validation model and a preset analytical database to obtain the table discrepancies.

[0019] The formula data is validated using the first CLIP validation model to obtain the formula difference items;

[0020] The image data is verified using the second CLIP verification model to obtain the image difference items.

[0021] In one embodiment, the text difference items include at least text semantic difference items and text content difference items. The step of validating the text data using a preset text validation model, a pre-trained language model, and preset formatting rules to obtain the text difference items includes:

[0022] The text data is converted into a document graph composed of interconnected text blocks based on the content arrangement information of the text data by a preset text verification model, and the feature vector of each text graph node is determined. The text graph node in the document graph is a text block in the text data.

[0023] The anomaly score of each of the text graph nodes is determined, and the anomaly score of any text graph node is determined based on the feature vector of the text graph node and the feature vectors of its neighboring text graph nodes.

[0024] Determine whether the anomaly score of each of the text graph nodes reaches the preset score;

[0025] When there is a target text graph node with an abnormal score that reaches the preset score, the target text graph node is marked as the text semantic difference item;

[0026] The text data is validated based on a pre-trained language model and preset formatting rules.

[0027] When there is target text content that fails content validation, the target text content is marked as the text content difference item.

[0028] In one embodiment, the table differences include at least table structure differences and table content differences. The step of validating the table data using a preset table validation model and a preset analytical database to obtain the table differences includes:

[0029] Determine the row and column relationship information of each extracted table in the table data;

[0030] The preset table validation model converts the corresponding extracted table into a structure diagram composed of interconnected cells based on the row and column relationship information. The table nodes in the structure diagram are the cells of the extracted table.

[0031] Calculate the structural similarity between each of the structural diagrams and the preset standard table structural diagram;

[0032] Determine whether the similarity of each structure is lower than a preset structural similarity threshold;

[0033] When there is a target structure diagram with a structural similarity lower than the preset structural similarity threshold, the extraction table corresponding to the target structure diagram is marked as the table structural difference item;

[0034] Data consistency verification is performed on each of the extracted tables based on a pre-defined analytical database.

[0035] When a target extraction table fails data consistency verification, the target extraction table is marked as a table content difference item.

[0036] In one embodiment, the formula difference items include at least formula semantic difference items and formula structural difference items. The step of validating the formula data using a first CLIP validation model to obtain the formula difference items includes:

[0037] Each extracted formula in the formula data is rendered into a corresponding rendered formula image;

[0038] The rendering formula feature vector of each of the rendering formula images is calculated using the first CLIP verification model;

[0039] The original formula feature vector corresponding to each of the original formula images is calculated using the first CLIP verification model.

[0040] Calculate the cosine similarity between the feature vectors of each rendering formula and the corresponding feature vectors of the original formula in the high-dimensional semantic space;

[0041] When there is a first type of extraction formula with a cosine similarity lower than a preset similarity threshold, the first type of extraction formula is marked as the semantic difference item of the formula;

[0042] Calculate the hash value difference between each rendered formula image and the corresponding original formula image;

[0043] When there is a second type of extraction formula with a hash value difference higher than the first preset difference threshold, the second type of extraction formula is marked as the formula structure difference item.

[0044] In one embodiment, the image difference items include at least image semantic difference items, image structural difference items, and image cropping difference items. The step of verifying the image data using a second CLIP verification model to obtain the image difference items includes:

[0045] Preprocess each of the cropped images in the image data to obtain the extracted image corresponding to each cropped image;

[0046] The semantic vector of each extracted image is calculated using the second CLIP verification model;

[0047] The original image semantic vector of each original image is calculated using the second CLIP verification model;

[0048] Calculate the cosine similarity between each extracted image semantic vector and the corresponding original image semantic vector;

[0049] When there is a first type of extracted image with a cosine similarity lower than a preset similarity threshold, the first type of extracted image is marked as the image semantic difference item;

[0050] Calculate the hash value difference between each extracted image and its corresponding original image;

[0051] When there is a second type of extracted image with a hash value difference higher than the second preset difference threshold, the second type of extracted image is marked as the image structure difference item;

[0052] Each extracted image is cropped and verified;

[0053] When there is a third type of extracted image that fails the cropping verification, the third type of extracted image is marked as the image cropping difference item.

[0054] In one embodiment, the step of cropping and verifying each of the extracted images includes:

[0055] Extract scale-invariant feature transform feature points from each of the original images.

[0056] Extract scale-invariant feature points from each of the extracted images;

[0057] The scale-invariant feature transform feature points in each extracted image are matched with the corresponding scale-invariant feature transform feature points in the original image to obtain multiple matching feature point pairs;

[0058] Determine the homography matrix based on each of the matching feature point pairs;

[0059] The extracted images are aligned with their corresponding original images using the homography matrix to perform cropping and verification of each extracted image;

[0060] The step of marking the third type of extracted image as the image cropping difference item when there is a third type of extracted image that fails the cropping verification includes:

[0061] When there is a third type of extracted image that is not aligned with the corresponding original image, the third type of extracted image is marked as the image cropping difference item.

[0062] In one embodiment, the step of determining the risk level corresponding to each difference item by using a preset quality grader includes:

[0063] Obtain multi-source difference information composed of the text difference items, the table difference items, the formula difference items, and the image difference items;

[0064] The risk level of each difference item in the multi-source difference information is obtained by scoring each difference item using a preset quality grader.

[0065] The step of correcting each difference item according to each of the risk levels includes:

[0066] Determine whether each of the aforementioned risk levels reaches a preset level;

[0067] When a first-class risk level exists that is lower than the preset level, the first-class difference item corresponding to the first-class risk level is corrected by a preset correction strategy;

[0068] When a second type of risk level exists that is not lower than the preset level, the second type of difference item corresponding to the second type of risk level is displayed on the preset interface so that the user can correct the second type of difference item.

[0069] Furthermore, to achieve the above objectives, this application also proposes an artificial intelligence-based multimodal document automatic proofreading system, the system comprising:

[0070] The input layer is used to receive the original document;

[0071] The parsing layer is used to convert the original document into structured data consisting of multiple data of different format types;

[0072] An artificial intelligence engine cluster is used to select the corresponding verification model according to the data of each of the aforementioned format types to verify the structured data and obtain the difference items corresponding to the data of each of the aforementioned format types;

[0073] The decision-making layer is used to determine the risk level corresponding to each of the aforementioned differences through a preset quality grader.

[0074] The decision-making layer is also used to correct each of the difference items according to each of the risk levels.

[0075] One or more technical solutions proposed in this application have at least the following technical effects:

[0076] This application receives original documents and converts them into structured data composed of multiple data types in different formats. It then selects the corresponding verification model for each data format type to proofread the structured data, obtaining the discrepancies for each format type. A preset quality grader determines the risk level of each discrepancy, and corrections are made based on the risk level. Compared to manual proofreading, which is limited by human effort and time, this application, by combining artificial intelligence (i.e., applying verification models), achieves automatic correction of structured data, efficiently processing large-scale structured data and effectively improving proofreading efficiency. Attached Figure Description

[0077] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0078] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0079] Figure 1 This is a flowchart illustrating the first embodiment of the AI-based multimodal document automatic proofreading method of this application;

[0080] Figure 2 This is a flowchart illustrating the second embodiment of the AI-based multimodal document automatic proofreading method of this application;

[0081] Figure 3 This is a flowchart illustrating the third embodiment of the AI-based multimodal document automatic proofreading method of this application;

[0082] Figure 4 This is a schematic diagram of the module structure of the AI-based multimodal document automatic proofreading system of this application;

[0083] Figure 5 This is a schematic diagram of the structure of the AI-based multimodal document automatic proofreading device of this application.

[0084] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0085] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0086] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0087] The main solution of this application embodiment is: receiving the original document and converting the original document into structured data composed of multiple data of different format types; selecting the corresponding verification model according to the data of each format type to verify the structured data and obtain the difference items corresponding to the data of each format type; determining the risk level corresponding to each difference item through a preset quality grader; and correcting each difference item according to each risk level.

[0088] Because existing technologies rely on manual verification of structured data word by word against the original documents, they are limited by human energy and time, making it difficult to efficiently process large-scale structured data, resulting in low verification efficiency.

[0089] This application provides a solution that selects corresponding verification models to proofread structured data of different format types, determines the risk level of discrepancies obtained through verification using a preset quality grader, and corrects discrepancies based on risk levels. Compared to manual proofreading, which is limited by human effort and time, this application can achieve automatic correction of structured data by combining artificial intelligence, i.e., applying verification models. It can efficiently process large-scale structured data and effectively improve proofreading efficiency.

[0090] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, such as an artificial intelligence-based multimodal document automatic proofreading device. The following description uses an artificial intelligence-based multimodal document automatic proofreading device (hereinafter referred to as the proofreading device) as an example to illustrate this embodiment and the following embodiments.

[0091] Based on this, embodiments of this application provide an artificial intelligence-based method for automatic proofreading of multimodal documents, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the AI-based multimodal document automatic proofreading method of this application.

[0092] In this embodiment, the AI-based multimodal document automatic proofreading method includes steps S10 to S40:

[0093] Step S10: Receive the original document and convert the original document into structured data consisting of multiple data of different format types.

[0094] It should be noted that the original document mentioned above can be a document containing various elements such as text, tables, formulas, and images.

[0095] In practice, the aforementioned proofreading equipment can use open-source layout parsing libraries such as Unstructured to parse the original document, convert each element in the original document into data of the corresponding format type, and the data of each format type constitutes structured data.

[0096] In one feasible implementation, step S10 includes steps S11 to S16:

[0097] Step S11: Receive the original document and preprocess it to obtain the preprocessed document.

[0098] Step S12: Identify text blocks, table areas, image areas, and formula areas in the preprocessed document.

[0099] In its implementation, the aforementioned proofreading device can preprocess the element document. For example, for scanned documents, a Generative Adversarial Network (GAN) model can be applied to denoise and enhance the scanned documents to improve the clarity of the text and images. Then, the preprocessed document is input into an object detection model, such as the YOLOv8 model, which identifies text blocks, table areas, formula areas, and image areas in the preprocessed document. Among these, text blocks can be composed of plain text, table areas can be composed of tables, image areas can be composed of images, and formula areas can be composed of formulas.

[0100] Step S13: The text block is identified using an optical character recognition model to obtain the JSON-formatted text data.

[0101] It should be noted that the output of the above object detection model can be the location information, such as coordinates, of text blocks, table regions, image regions, and formula regions within the original document. The proofreading device can then use this location information to crop the preprocessed document, thereby extracting the text blocks, table regions, image regions, and formula regions from the preprocessed document. The corresponding structured strategy is then selected based on the text blocks, table regions, image regions, and formula regions.

[0102] In practice, the aforementioned proofreading device can use an Optical Character Recognition (OCR) model to recognize text blocks and convert them into plain text JSON format text data.

[0103] The OCR model can use either PaddleOCR or Tesseract as its engine, and supports mixed Chinese and English text.

[0104] Step S14: Identify each original table in the table area using a preset table parsing model to obtain the CSV format table data, which includes the extracted tables corresponding to each original table.

[0105] It should be noted that the above-mentioned preset table parsing model can be a model for structuring tables and exporting data in comma-separated values ​​(CSV) format. It can be obtained by training machine learning models or deep learning models using historical table data.

[0106] In a specific implementation, the aforementioned proofreading device can identify the structure and cell content of each original table in the table area through a preset table parsing model, and export the structure and cell content of each original table as an extracted table in CSV format. Each extracted table constitutes the aforementioned table data.

[0107] Step S15: Recognize each original formula image in the formula area using a preset formula recognition model to obtain the formula data in LaTeX format. The formula data includes the extracted formula corresponding to each original formula image.

[0108] It should be noted that the above-mentioned preset formula recognition model can be a model for structuring the format and exporting LaTeX format data, such as the MathPix API model.

[0109] In a specific implementation, the aforementioned proofreading device can call a preset formula recognition model to recognize each original formula image in the formula area, convert each original format into an extracted formula in LaTeX format, and each extracted formula constitutes the aforementioned formula data.

[0110] Step S16: Capture each original image in the image area to obtain image data in the image format. The image data includes the cropped image corresponding to each original image.

[0111] In a specific implementation, the aforementioned proofreading device can perform image cropping operations on the image area, extract each original image from the image area, and obtain cropped images corresponding to each original image. The format of each cropped image is consistent with the format of the original image, and each cropped image constitutes the aforementioned image data.

[0112] It should be understood that the above structuring process supports multi-threaded parallel processing, thereby improving parsing efficiency. For example, four CPU threads can be used to process text blocks, table areas, formula areas, and image areas in parallel. Through this structuring process, complex documents can be digitized into structured data for subsequent automatic proofreading. Furthermore, the content and location information obtained from the structured parsing can be encapsulated into a hierarchical intermediate representation, such as a JSON Schema tree, preserving metadata such as page numbers and paragraph numbers so that the results correspond to the original text. Moreover, the above structuring process supports multi-threaded parallel processing to accelerate parsing (e.g., four CPU threads processing different pages or blocks in parallel).

[0113] Furthermore, the structured data output by the aforementioned proofreading equipment can be temporarily stored in a structured data pool. This pool serves as a shared intermediate data warehouse, employing a combination of in-memory database and persistent storage to achieve both high-speed reading and persistent storage. Text data is stored in JSON format, and tabular data in CSV format for convenient on-demand random access. Formula and image data are stored as file path references. Within the data pool, rendered images of frequently occurring mathematical formulas can be cached to avoid performance impact from frequent formula rendering calls. This cache uses a Least Recently Used (LRU) strategy to evict old entries and sets a lifespan to ensure that formulas not used for a long time automatically expire, thereby controlling memory usage limits (e.g., 4GB). In implementation, the cache can be queried using the hash of the calculated formula as the key. When a formula with the same hash is identified, the existing rendered result is directly retrieved as the extracted formula. Monitoring metrics can also be integrated, such as using Prometheus to monitor cache hit rate, memory usage, and read latency. By using a structured data pool as the data hub, the modules in each proofreading device can be decoupled and work in parallel, while ensuring that the parsing results are persistent and traceable, thus improving the scalability of the proofreading device.

[0114] Step S20: Select the corresponding verification model according to the data of each format type to verify the structured data and obtain the difference items corresponding to the data of each format type.

[0115] In practical implementation, multiple historical JSON text data, historical CSV table data, historical LaTeX formula data, and historical image data can be collected in advance. Different verification models are trained for each of these structured data formats. After obtaining the structured data, the verification device can call the corresponding verification model to verify the data of different format types, obtaining the differences for each format type. If the content of the structured data obtained from these differences differs from the content of the original document, it needs to be corrected.

[0116] Step S30: Determine the risk level corresponding to each difference item using a preset quality grader.

[0117] It should be noted that the aforementioned preset quality grader can be a gradient boosting machine learning framework based on decision trees, used to classify and score differences, such as Light Gradient Boosting Machine (LightGBM).

[0118] In its implementation, the aforementioned proofreading equipment can input the discrepancies obtained from the proofreading into a preset quality grader. The quality grader then classifies and scores each discrepancy to obtain its corresponding risk level. A higher risk level indicates a higher discrepancy score and a greater difference from the original document.

[0119] Step S40: Correct each of the differences according to each of the risk levels.

[0120] It should be noted that the aforementioned proofreading equipment can perform corresponding correction operations based on the risk level of different discrepancies. In real-world scenarios, the number of high-risk discrepancies is lower than the number of low-risk discrepancies. Therefore, when the proofreading equipment detects discrepancies with a risk level lower than a preset level, it determines that the risk of such discrepancies is low, such as typos or formatting errors. The equipment can then automatically correct these low-risk discrepancies based on the content of the original document. When discrepancies with a risk level higher than the preset level are detected, they are determined to be high-risk, and these discrepancies can be displayed for manual review. This achieves a complementary advantage between artificial intelligence and human expertise: humans correct a small number of high-risk discrepancies, while the proofreading equipment automatically corrects a large number of low-risk discrepancies, thereby improving proofreading efficiency and accuracy.

[0121] This embodiment receives the original document and converts it into structured data composed of multiple data types in different formats. It then selects the corresponding verification model for each data format type to proofread the structured data, obtaining the discrepancies for each format type. A preset quality grader determines the risk level of each discrepancy. Finally, each discrepancy is corrected based on its risk level. This embodiment uses different verification models to proofread the structured data in different formats and determines the risk level of the discrepancies using a preset quality grader. Based on the risk level, the discrepancies are corrected. Compared to manual proofreading, which is limited by human effort and time, this embodiment, by combining artificial intelligence (i.e., applying verification models), can automatically correct structured data, efficiently processing large-scale structured data and effectively improving proofreading efficiency.

[0122] Based on the first embodiment of this application, a second embodiment of this application is proposed. In the second embodiment, content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the AI-based multimodal document automatic proofreading method of this application.

[0123] In this embodiment, the differences include at least text differences, table differences, formula differences, and image differences. Step S20 includes steps S21 to S24:

[0124] Step S21: The text data is validated using a preset text validation model, a pre-trained language model, and preset formatting rules to obtain the text difference items.

[0125] It should be noted that the above-mentioned preset text verification model can be a graph neural network (GNN) model that converts text fragments into graph representations.

[0126] Understandably, the aforementioned pre-trained language model can be used to validate the syntax and semantics of text.

[0127] It should be noted that the above preset formatting rules can be pre-defined standard rules used for standard formats and their usage information. Examples include consistent number and unit formats, standardized use of Chinese and English punctuation, paired matching of quotation marks and brackets, and correct capitalization of proper nouns.

[0128] In specific implementation, the above-mentioned verification device can convert text data into a document graph represented by a preset text verification model, verify the semantics of the text data through the document graph, and verify the content of the text data by combining a pre-trained language model and preset format rules. The text difference item is obtained by combining the semantic verification result and the content verification result. The text difference item represents the part of the extracted text data that differs from the original text.

[0129] In one feasible implementation, the text difference items include at least text semantic difference items and text content difference items, and step S21 includes steps S211 to S216:

[0130] Step S211: Based on the content arrangement information of the text data, the text data is converted into a document graph composed of interconnected text blocks by a preset text verification model, and the feature vector of each text graph node is determined. The text graph node in the document graph is a text block in the text data.

[0131] It should be noted that the above-mentioned content arrangement information can represent the document reading order and spatial proximity of text data. The document reading order can be the order in which the text content in the text data is arranged according to conventional reading habits such as from front to back or from top to bottom. Spatial proximity can be the distance and relative positional relationship between different text blocks, paragraphs, words, and other elements in the text data in two-dimensional space.

[0132] In its implementation, the aforementioned verification device can determine the content layout information of the text data. It identifies text blocks (such as paragraphs and headings) within the text data as text graph nodes using a preset text verification model. Based on the content layout information, it connects adjacent text graph nodes with edges to form a document graph. For example, text areas containing vertically adjacent text blocks will be connected by edges, and adjacent text blocks will be connected sequentially from left to right and from top to bottom. Text areas that are close to each other in the page layout can also be connected, thus forming a complete document graph.

[0133] Furthermore, the aforementioned proofreading device can perform word embedding encoding on text graph nodes, and use the encoded semantic vector as the semantic feature of the text graph node. This semantic feature reflects the semantic meaning expressed by the text graph node. The correction recognition can also identify layout features such as font size and position of text graph nodes, and use the semantic features and layout features as feature vectors of the text graph node.

[0134] Step S212: Determine the anomaly score of each text graph node. The anomaly score of any text graph node is determined based on the feature vector of the text graph node and the feature vectors of its adjacent text graph nodes.

[0135] It should be noted that the anomaly score mentioned above measures the degree of semantic difference between a text graph node and its neighboring text graph nodes. A higher anomaly score indicates a greater semantic difference between the text graph node and its neighboring text graph nodes.

[0136] In its implementation, the aforementioned proofreading device can send the feature vectors of each text graph node as messages to adjacent text graph nodes. For example, taking paragraph nodes to text graph nodes as an example, the feature vectors of any paragraph can be sent to adjacent nodes such as the previous paragraph node, the next paragraph node, and the adjacent title node. After receiving the message from any text graph node, the aforementioned proofreading device can merge and update the feature vectors of the adjacent text graph node with the feature vectors of any text graph node, so that the adjacent text graph node can more accurately reflect its role in the entire document's logical structure and its correlation with the content of surrounding text graph nodes.

[0137] Furthermore, after updating the feature vectors of each text graph node, the aforementioned proofreading device can calculate the semantic similarity, such as cosine similarity, between the feature vectors of any text graph node and its neighboring text graph nodes. Then, the semantic similarity is normalized, and the normalized semantic similarity is converted into an anomaly score, where a larger value indicates a smaller semantic similarity. For example, the semantic similarity can be normalized to the range of 0 to 1, and the anomaly score can be obtained by subtracting the normalized semantic similarity from 1.

[0138] Step S213: Determine whether the abnormal score of each text graph node has reached the preset score.

[0139] It should be noted that the above preset score is a pre-set value used to measure whether the abnormal score is too large.

[0140] Step S214: When there is a target text graph node with an abnormal score that reaches the preset score, mark the target text graph node as the text semantic difference item.

[0141] In its implementation, the aforementioned verification device compares the anomaly scores of each text graph node with preset scores to determine whether the anomaly scores of each text graph node reach the preset scores. When a target text graph node with an anomaly score reaching the preset scores is detected, it is determined that the semantic difference between the target text graph node and its adjacent text graph nodes is large, and the target text graph node is marked as a text semantic difference item that needs to be corrected.

[0142] It should be understood that, compared to word-by-word and sentence-by-sentence verification, the above-mentioned verification method using a pre-set text verification model can capture the relationships within and between paragraphs, thus improving the accuracy of error detection.

[0143] Step S215: Perform content verification on the text data based on the pre-trained language model and preset format rules.

[0144] Step S216: When there is target text content that fails content verification, mark the target text content as the text content difference item.

[0145] In specific implementation, the pre-trained language model can have multiple built-in grammar rule bases, such as part-of-speech tagging rules and syntactic structure rules. The proofreading device can input text blocks of text data into the pre-trained semantic model, and the pre-trained language model can match each text block with the grammar rule base to check whether each text block conforms to the grammar rules in the grammar rule base. For example, it can check whether the subject and verb are consistent, whether the tense is correct, and whether the sentence structure is complete. When a segment that does not conform to the grammar is detected or a certain judgment has a contextual logical conflict, the segment can be determined as the target text content that failed the content verification, and the target text content can be marked as the text content difference item that needs to be corrected.

[0146] Furthermore, the aforementioned proofreading device can match each text block with preset formatting rules to detect whether there are formatting errors in each text block. For example, it can check whether the format of numbers and units is consistent, whether Chinese and English punctuation is used correctly, whether quotation marks and brackets are matched in pairs, and whether proper nouns are spelled correctly in capitalization. The aforementioned proofreading device can mark text blocks with formatting errors as target text content to obtain the text content differences that need to be corrected.

[0147] Step S22: The table data is validated using a preset table validation model and a preset analytical database to obtain the table difference items.

[0148] It should be noted that the above-mentioned preset table validation model can be a GNN model that converts the table structure into a graph structure.

[0149] Understandably, the aforementioned pre-defined analytical database can be a model that performs aggregate validation queries on tables, such as a lightweight embedded OLAP database that can support performing Structured Query Language (SQL) query operations on large datasets in memory.

[0150] In specific implementation, the above-mentioned verification device can convert each extracted table in the table data into a corresponding graph structure diagram through a preset table verification model. The structure of the table data is verified through the structure diagram, and the table data is aggregated and verified in combination with a preset analytical database to check whether the table data is accurate. The table difference items are obtained by combining the structure verification results and the data verification results. The table difference items represent the parts of the extracted table data that differ from the tables in the original document.

[0151] In one feasible implementation, the table differences include at least table structure differences and table content differences, and step S22 includes steps S221 to S227:

[0152] Step S221: Determine the row and column relationship information of each extracted table in the table data.

[0153] It should be noted that the above row and column relationship information can represent the proximity relationship between the rows and columns of a cell.

[0154] In its implementation, the aforementioned verification device can traverse each extracted table in the table data to determine the row and column relationship information of each extracted table.

[0155] Step S222: Based on the row and column relationship information, the corresponding extracted table is converted into a structure diagram composed of interconnected cells by a preset table verification model. The table nodes in the structure diagram are the cells of the extracted table.

[0156] In its specific implementation, for any extracted table, the aforementioned verification device can use the cells of the extracted table as nodes in the table graph, and construct edges between the corresponding table graph nodes based on row and column relationship information. For example, if two cells are in the same row and adjacent columns, or in the same column and adjacent rows, then an edge is established between the nodes of the two cells. The aforementioned verification device can repeat the above operation for each extracted table to obtain the structure graph corresponding to each extracted table, thereby completely transforming the structure of the two-dimensional table into a graph structure representation, forming a structure graph.

[0157] Step S223: Calculate the structural similarity between each of the structural diagrams and the preset standard table structural diagram.

[0158] It should be noted that the above-mentioned preset standard table structure diagram can be pre-built and includes a diagram representation of various common and correct table structure forms, which can be used as a reference benchmark for verification.

[0159] In its implementation, the aforementioned verification device compares the structure diagrams of each extracted table with the structure diagrams of preset representation tables, calculating the structural similarity between the structure diagrams of each extracted table and the preset representation table. This structural similarity reflects the degree of structural matching between the structure of the extracted table and the preset standard table structure diagram. The calculation methods for this structural similarity include, but are not limited to, structural edit distance similarity.

[0160] Step S224: Determine whether the similarity of each structure is lower than a preset structural similarity threshold.

[0161] It should be noted that the above-mentioned preset structural similarity threshold can be a pre-set similarity threshold used to measure whether the structural similarity is large, such as 0.92.

[0162] Step S225: When there is a target structure diagram with a structural similarity lower than the preset structural similarity threshold, the extraction table corresponding to the target structure diagram is marked as the table structural difference item.

[0163] In its implementation, the aforementioned verification device can compare the structural similarity of each extracted table with a preset structural similarity threshold to determine whether each structural similarity is lower than the preset structural similarity threshold. If there is a target structural diagram with a structural similarity lower than the preset structural similarity threshold, it is determined that the extracted table corresponding to the target structural diagram has structural anomalies, such as incorrect cell splitting, merging errors, or incorrect table header hierarchy. The extracted table corresponding to the target structural body can be marked as a table structural difference item that needs to be corrected. Conversely, if there are other structural diagrams with a structural similarity higher than the preset structural similarity threshold, it is determined that the extracted tables corresponding to the other structural diagrams have the same structure as the preset standard table structure and have no structural anomalies.

[0164] Step S226: Perform data consistency verification on each of the extracted tables based on a preset analytical database.

[0165] Step S227: When there is a target extraction table that fails the data consistency check, mark the target extraction table as a table content difference item.

[0166] It should be noted that the aforementioned preset analytical database can be pre-configured with multiple SQL verification rules, and the aforementioned verification device can perform data consistency verification on each extracted table based on each SQL verification rule.

[0167] Specifically, the aforementioned SQL validation rules may include row and column summary consistency rules. The validation device can respond to these rules by using SQL aggregate functions (such as SUM) to calculate the sum of its sub-items and comparing it with the value of the total cell. For example, for a sales table, it verifies whether the value of the total row is equal to the sum of sales data for each month. If the value of a total row or column in any extracted table is not equal to the sum of its sub-items, the extracted table is determined to be a target extracted table where data consistency validation has failed, and this target extracted table is marked as an item requiring correction of table content discrepancies.

[0168] Furthermore, the aforementioned SQL verification rules may also include key value verification rules. The verification device can respond to these rules and verify the key values ​​in each table's data. It can check whether any extracted table contains outliers exceeding a preset multiple (e.g., 3 times) of the standard deviation or duplicate primary keys. For example, in an employee information table, it checks whether employee IDs are unique and whether duplicate records exist. If any extracted table contains outliers exceeding a preset multiple (e.g., 3 times) of the standard deviation or duplicate primary keys, the extracted table is determined to be the target extracted table where data consistency verification fails, and this target extracted table is marked as the table whose content discrepancies require correction.

[0169] The SQL verification rules mentioned above may also include reference consistency rules. The verification device can respond to the reference consistency rules and check whether the reference relationship between the main table and the sub-table in each extracted table is consistent. When the reference relationship between the main table and the sub-table in any extracted table is inconsistent, the extracted table is determined to be the target extracted table where the data consistency verification fails, and the target extracted table is marked as the table content difference item that needs to be corrected.

[0170] In addition, the aforementioned verification device can also identify the data type of each column in each extracted table and determine whether the data format of each data type is correct. For example, if a column is expected to be numeric but contains non-numeric characters, the non-numeric characters are marked as table content differences that need to be corrected. Entries with incorrect formatting in the date column can also be marked as table content differences that need to be corrected. For purely numeric columns, the null value rate of the column can be calculated and compared with a threshold that varies with the data size. If the rate exceeds the threshold, it is determined that the column has an abnormal proportion of missing values ​​and is marked as a table content difference that needs to be corrected.

[0171] Step S23: Verify the formula data using the first CLIP verification model to obtain the formula difference items.

[0172] It should be noted that the first CLIP verification model can be a Contrastive Language-Image Pre-training (CLIP) model for verifying the consistency of formula content.

[0173] In a specific implementation, the aforementioned verification device can verify each extracted formula in the formula data through a pre-trained first CLIP verification model, and mark the extracted formulas that fail verification as formula difference items that need to be corrected.

[0174] In one feasible implementation, step S23 includes steps S231 to S237:

[0175] Step S231: Render each of the extracted formulas in the formula data into a corresponding rendered formula image.

[0176] In practice, the aforementioned proofreading device can render each extracted formula into a formula image using a LaTeX engine, while simultaneously acquiring the original formula image from the preprocessed document.

[0177] Step S232: Calculate the rendering formula feature vector of each of the rendering formula images using the first CLIP verification model.

[0178] Step S233: Calculate the original formula feature vector corresponding to each of the original formula images using the first CLIP verification model.

[0179] In a specific implementation, the aforementioned calibration device can calculate the image feature vectors of each rendered formula image and the original formula image through the first CLIP verification model, thereby obtaining the rendered formula feature vector of each rendered formula image and the original formula feature vector corresponding to each original formula image.

[0180] Step S234: Calculate the cosine similarity between the feature vectors of each rendering formula and the corresponding feature vectors of the original formula in the high-dimensional semantic space.

[0181] Step S235: When there is a first type of extraction formula with a cosine similarity lower than a preset similarity threshold, the first type of extraction formula is marked as the semantic difference item of the formula.

[0182] It should be noted that the aforementioned preset similarity threshold can be a pre-set cosine similarity threshold, used to measure whether the cosine similarity is large, such as 0.85.

[0183] In a specific implementation, the aforementioned proofreading device can calculate the cosine similarity between the feature vector of each rendered formula and the corresponding feature vector of the original formula in the high-dimensional semantic space, and compare each cosine similarity with a preset similarity threshold. If there is a first-class extracted formula with a cosine similarity lower than the preset similarity threshold, it is determined that the first-class extracted formula is semantically different from the original formula, and the first-class extracted formula is marked as the formula semantic difference item that needs to be corrected.

[0184] Step S236: Calculate the hash value difference between each rendered formula image and the corresponding original formula image.

[0185] Step S237: When there is a second type of extraction formula with a hash value difference higher than the first preset difference threshold, the second type of extraction formula is marked as the formula structure difference item.

[0186] In practical implementation, to ensure accuracy, the aforementioned calibration device can calculate the differences in perceptual hash values, differential hash values, and wavelet hash values ​​between each rendered formula image and its corresponding original formula image. For any rendered formula image, the sum of these differences—perceptual hash values, differential hash values, and wavelet hash values—between it and its corresponding original formula image is calculated to obtain the hash value difference corresponding to that rendered formula image. The aforementioned calibration device can obtain a pre-set first preset difference threshold (e.g., 15). When a second type of extraction formula exists with a hash value difference higher than the first preset difference threshold, it is determined that the rendered formula image corresponding to the second type of extraction formula differs significantly from the original formula image in pixel details, and the second type of extraction formula is marked as a formula structure difference item that needs to be corrected.

[0187] In addition, the aforementioned proofreading equipment can also perform syntax checks on each extracted formula, such as detecting whether there are unclosed parentheses, missing paired symbols, illegal commands, etc. Furthermore, it can perform format checks on each extracted formula based on the formula layout (inline formulas or block-level formulas) in the original document, such as checking whether dollar signs are correctly added or removed, and then mark the extracted formatting that fails syntax or format checks as formula structure differences that need to be corrected.

[0188] It should be understood that by using the above-mentioned multiple verification methods for formula data, errors in formula extraction can be identified and differences marked with high confidence. For example, when an error in the superscript range causes an abnormal formula image after rendering, it can be identified through the difference in the obtained hash value; or when an error in the recognition of a mathematical symbol causes a semantic change, it can be identified through cosine similarity, thereby improving the accuracy of formula data verification.

[0189] Step S24: Verify the image data using the second CLIP verification model to obtain the image difference item.

[0190] It should be noted that the second CLIP verification model can be a small CLIP model for image consistency verification, such as the CLIP-ViT-Tiny model.

[0191] In a specific implementation, the aforementioned verification device can, after acquiring image data containing non-textual content such as illustrations, photographs, and screenshots, verify each cropped image in the image data using a pre-trained second CLIP verification model. This verifies the clarity, integrity, and consistency of each cropped image with the original image content, and marks cropped images that fail verification as image differences that need to be corrected.

[0192] In one feasible implementation, the image difference items include at least image semantic difference items, image structural difference items, and image cropping difference items, and step S24 includes steps S241 to S249:

[0193] Step S241: Preprocess each of the cropped images in the image data to obtain the extracted image corresponding to each cropped image.

[0194] In practice, the aforementioned calibration device can call the corresponding GAN model to preprocess each cropped image, including adjusting the image size, normalizing pixel values, and enhancing image contrast. For example, it can use DeblurGAN-v2 to eliminate blur and DemoireNet to remove scanning mesh noise, thereby obtaining the extracted image corresponding to each cropped image, improving the image clarity, and thus improving the efficiency and accuracy of subsequent processing.

[0195] In addition, the aforementioned proofreading device can also input each extracted image into a target detection model, such as the YOLOv8 model, to detect whether each extracted image contains text or other regions of interest, thus avoiding missed detections due to noise or blur.

[0196] Step S242: Calculate the extracted image semantic vector of each extracted image using the second CLIP verification model.

[0197] Step S243: Calculate the original image semantic vector of each original image using the second CLIP verification model.

[0198] In a specific implementation, the aforementioned proofreading device can extract features from each extracted image using the second CLIP verification model to obtain the semantic vector of each extracted image; the aforementioned proofreading device can also extract features from each original image using the second CLIP verification model to obtain the semantic vector of each original image.

[0199] Furthermore, when the preprocessed document provides a text description of the original image (as shown in the figure note), the aforementioned proofreading device can extract features from the text description using the second CLIP verification model to obtain the corresponding text semantic vector, and then merge the text semantic vector into the corresponding original image semantic vector.

[0200] Step S244: Calculate the cosine similarity between each extracted image semantic vector and the corresponding original image semantic vector.

[0201] Step S245: When there is a first type of extracted image with a cosine similarity lower than a preset similarity threshold, the first type of extracted image is marked as the image semantic difference item.

[0202] In a specific implementation, the aforementioned calibration device can calculate the cosine similarity between each extracted image semantic vector and the corresponding original image semantic vector, and compare each cosine similarity with a preset similarity threshold (consistent with the values ​​involved in the formula correction, such as 0.85). If there is a first-class extracted image with a cosine similarity lower than the preset similarity threshold, it is determined that the first-class extracted image has a semantic difference from the original image semantic vector, and the first-class extracted image is marked as the image semantic difference item that needs to be corrected.

[0203] Step S246: Calculate the hash value difference between each extracted image and the corresponding original image.

[0204] Step S247: When there is a second type of extracted image with a hash value difference higher than the second preset difference threshold, the second type of extracted image is marked as the image structure difference item.

[0205] In its implementation, the aforementioned calibration device can calculate the differences in perceptual hash values, differential hash values, and wavelet hash values ​​between each extracted image and its corresponding original image. For any extracted image, the sum of these differences—perceptual hash values, differential hash values, and wavelet hash values—between it and its corresponding original image is calculated to obtain the hash value difference corresponding to that extracted image. The calibration device can obtain a pre-set first preset difference threshold. When a second type of extracted image exists with a hash value difference higher than the first preset difference threshold, it is determined that the extracted image corresponding to the second type of extracted image differs significantly from the original image in pixel details, and the second type of extracted image is marked as a formula structure difference term requiring correction.

[0206] It should be understood that through the above-mentioned image semantic correction and structural correction, it is possible to effectively determine whether the extracted image is consistent with the original image or its text description, thereby enabling quality control of the image extraction or processing process.

[0207] Step S248: Perform cropping and verification on each of the extracted images.

[0208] In its implementation, the cropping verification performed by the aforementioned calibration device includes detecting whether each extracted image is complete during the cropping process.

[0209] In one feasible implementation, step S248 may include steps S2481 to S2485:

[0210] Step S2481: Extract scale-invariant feature transformation feature points from each of the original images.

[0211] Step S2482: Extract scale-invariant feature transformation feature points from each of the extracted images.

[0212] In its implementation, the aforementioned calibration device can extract Scale-Invariant Feature Transform (SIFT) feature points from each original image, as well as SIFT feature points from each extracted image. The SIFT feature points in the original image effectively describe the local features of the image and are robust to changes in scale, rotation, and illumination. The SIFT feature points in the extracted image are used for matching with the SIFT feature points in the original image.

[0213] Step S2483: Match the scale-invariant feature transform feature points in each extracted image with the corresponding scale-invariant feature transform feature points in the original image to obtain multiple matching feature point pairs.

[0214] In a specific implementation, the aforementioned calibration device can use a feature matching algorithm (such as a FLANN matcher) to calculate the distance between SIFT feature points in each extracted image and the corresponding SIFT feature points in the original image. This distance metric includes, but is not limited to, Euclidean distance and Hamming distance. A pre-set distance threshold is obtained. If the distance between any two SIFT feature points is less than the set distance threshold, it is determined that the two SIFT feature points have similar features, and the SIFT feature points are associated to form a matching feature point pair.

[0215] Step S2484: Determine the homography matrix based on each of the matching feature point pairs.

[0216] In a specific implementation, the aforementioned calibration device can use the Random Sample Consensus (RANSAC) algorithm to calculate the homography matrix from each matching feature point pair. This homography matrix describes the geometric transformation relationship between the original image and the feature image.

[0217] Step S2485: Align each extracted image with the corresponding original image using the homography matrix to perform cropping verification on each extracted image.

[0218] Step S249: When there is a third type of extracted image that fails to be extracted, mark the third type of extracted image as the image extraction difference item.

[0219] Accordingly, step S249 includes step S2491:

[0220] Step S249: When there is a third type of extracted image that is not aligned with the corresponding original image, mark the third type of extracted image as the image cropping difference item.

[0221] In its implementation, the aforementioned calibration device utilizes a homography matrix to project and transform each extracted image onto the coordinate system of the original image, achieving alignment. It then checks if the aligned extracted image algorithm completely covers the original image area. If the extracted image aligns with only a portion of the original image area, it is determined that the extracted image has been cropped or is missing. This extracted image is classified as a third type of failed cropping verification and marked as an image cropping difference requiring correction. If the extracted image completely covers the original image area, the cropping verification is considered successful. Through this cropping correction operation, it is possible to accurately determine whether the extracted image remains completely consistent with the original image, ensuring that no cropping or missing images occurred during the extraction and processing.

[0222] This embodiment verifies text data using a preset text verification model, a pre-trained language model, and preset formatting rules to obtain text discrepancies; it verifies table data using a preset table verification model and a preset analytical database to obtain table discrepancies; it verifies formula data using a first CLIP verification model to obtain formula discrepancies; and it verifies image data using a second CLIP verification model to obtain image discrepancies. This achieves verification based on different selections of the text content, effectively improving the accuracy of the correction.

[0223] Based on the first and second embodiments of this application, a third embodiment of this application is proposed. In this third embodiment, content that is the same as or similar to the first and second embodiments described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 , Figure 3 This is a flowchart illustrating the third embodiment of the AI-based multimodal document automatic proofreading method of this application.

[0224] In this embodiment, step 30 includes steps S31 to S32:

[0225] Step S31: Obtain multi-source difference information consisting of the text difference items, the table difference items, the formula difference items, and the image difference items.

[0226] In practice, the aforementioned proofreading device can add the text differences, table differences, formula differences, and image differences obtained from the above correction operations to the same list to form multi-source difference information.

[0227] Step S32: The risk level of each difference item in the multi-source difference information is obtained by scoring each difference item using a preset quality grader.

[0228] In practice, the aforementioned calibration device can input multi-source difference information into a preset quality classifier. The preset quality classifier can output the corresponding risk level based on the characteristics of each difference item, and at the same time output the corresponding confidence score to help judge uncertainty.

[0229] It should be noted that the risk levels output by the aforementioned preset quality classifier range from low to high, including low risk, medium risk, and high risk.

[0230] Accordingly, step 40 includes steps S41 to S43:

[0231] Step S41: Determine whether each of the risk levels has reached a preset level.

[0232] Step S42: When there is a first type of risk level lower than the preset level, the first type of difference item corresponding to the first type of risk level is corrected by a preset correction strategy.

[0233] In practice, the aforementioned calibration equipment can set the preset level to medium risk level, compare the risk level corresponding to each difference item with the medium risk level, and when there is a first-class risk level that is lower than the medium risk level (i.e., low risk level), the first-class difference item corresponding to the first-class risk level can be corrected through a preset correction strategy.

[0234] The preset correction strategies include: For text discrepancies, spelling dictionaries, language models, and regular expressions can be used to batch correct typos and formatting errors identified by OCR. For example, for issues such as mixed use of full-width / half-width characters, mismatched punctuation pairs, capitalization errors, and redundant spaces, automatic normalization replacement can be performed. For words suspected of being spelled incorrectly, a pre-trained language model can provide suggestions for correction based on the context. For table discrepancies, when table structure or content discrepancies are detected, automatic completion or recalculation is performed according to preset rules. For example, when a "total" is not equal to the sum of its sub-items, the correct value is recalculated and entered, or a marker is added to the cell. If a column is expected to contain dates but contains non-date strings, it is converted to a standard date format. For formula discrepancies, when minor formatting errors (such as missing curly braces in superscripts) are detected, automatic completion and correction can be performed based on LaTeX syntax. For image differences, if the semantic differences are caused by low image resolution or noise, image enhancement algorithms (such as deblurring and demoiring) can be automatically invoked to improve clarity.

[0235] Step S43: When there is a second type of risk level that is not lower than the preset level, the second type of difference item corresponding to the second type of risk level is displayed on the preset interface so that the user can correct the second type of difference item.

[0236] In practice, the aforementioned proofreading equipment compares the risk level corresponding to each difference item with the medium risk level. When there is a second type of risk level that is not lower than the medium risk level (i.e., the medium risk level or the high risk level), the second type of difference item can be rendered into the corresponding manual proofreading interface and displayed. The specific problems of the second type of difference item can be displayed by means of highlighting, side notes, or displaying the original text and the extracted content side by side.

[0237] It should be noted that on the manual proofreading interface, users can interactively process the displayed second-category discrepancies. Specifically, users can choose to accept automatic correction (i.e., trigger the proofreading device to automatically correct based on a preset correction strategy), manually adjust and modify, or mark the second-category discrepancy as a false alarm and ignore it. The manual proofreading interface supports convenient navigation functions, allowing users to filter by discrepancy type, page number, etc., and jump to the corresponding location in the original document to view the original context. All manual operations during the entire review process are recorded in the background log and fed back to the proofreading device in real time. The proofreading device can store the received feedback data in the training database for subsequent model iteration and learning, thereby continuously improving the accuracy of the relevant verification model.

[0238] For example, in a real-world scenario, once all content fragments of an original document have been proofread, the aforementioned proofreading device can automatically generate a final proofreading report. Specifically, this application can employ Kafka-Streams stream processing to merge the proofreading results from each parallel channel as they arrive: that is, multi-source difference information consisting of text differences, table differences, formula differences, and image differences can be merged into a unified data structure in real time through a streaming aggregator after correction. The output data structure, i.e., the standardized result, follows a predefined JSON Schema standard format, including fields such as the document's hierarchical structure, the proofreading conclusion for each element, the location of the differences, and suggested correction operations. This standardized result can be used to generate user-readable reports and is also easy for machine systems to parse, enabling integration with backend business processes. While generating the standardized result, the aforementioned proofreading device can use a front-end visualization library (such as Vega-Lite) to render the proofreading result into a human proofreading interface. The human proofreading interface restores the layout of the original document, with each difference item marked with a highlighted background or warning icon in its corresponding position. Users can click on markers to expand and view detailed information and corresponding modification suggestions for each discrepancy. The human proofreading interface also supports quick filtering and locating based on page number or discrepancy category, helping users efficiently browse proofreading results. Furthermore, this human proofreading interface, combined with human review, allows users to directly process each discrepancy (accepting / rejecting AI modifications, etc.), and the executed human actions are recorded by the backend service and used as a reference for subsequent model learning. Ultimately, the proofreading equipment outputs two forms of results: a visually appealing proofreading report for human review (in PDF or web page format) and a machine-readable JSON format discrepancy list and correction scheme. This dual output format ensures that the proofreading results can be easily reviewed by humans and automatically processed by other systems, achieving a closed-loop human-machine interface. This enables fully automated workflow processing of document proofreading tasks from parsing, detection, decision-making, correction to reporting, with human interaction introduced at relevant decision points, achieving a balance between high efficiency and high reliability.

[0239] Furthermore, to continuously improve the performance of the proofreading equipment over time, an adaptive continuous optimization mechanism is incorporated. This includes an active learning framework: automatically collecting discrepancy samples with uncertain model predictions and case data with significant manual corrections, adding them to the key training set to enhance the learning of each verification model in relevant situations. Specifically, this strategy involves pre-setting discrepancies with high prediction entropy (high uncertainty) or those classified as high-risk by the quality grader. These are recorded, manually confirmed, and added to the model training library as new training samples, given higher weights for priority learning, thereby improving the accuracy of subsequent verification models. The proofreading equipment also includes an online fine-tuning mechanism: after processing a set number of documents (e.g., 10,000 pages), newly accumulated labeled data can be used to perform small incremental training updates on each verification model for fine-tuning. To control the learning rate and training rounds during fine-tuning, rolling training based on recent data can be configured to adapt to gradual changes in document distribution (e.g., changes in document formats across different industries), while retaining all historical versions of the verification model. Semantic version number management ensures that each verification model update is traceable and supports rollback. The proofreading equipment also includes a feature repository: it stores the features of each processed document (such as document type, language, scan clarity score, etc.) and the corresponding verification results. Analyzing this repository reveals weaknesses in certain types of documents (e.g., high error rates for certain fonts), allowing for targeted training with more data of that type. Simultaneously, when feature statistics indicate a significant change in the distribution of recently processed documents, it triggers specific fine-tuning of the corresponding model to maintain performance stability. The proofreading equipment also includes rule monitoring and optimization: it continuously tracks the hit rate and false alarm rate of its built-in proofreading rules. If a rule is frequently triggered but mostly false alarms (e.g., manually determined to be error-free), its priority is automatically lowered or it is disabled. Conversely, for manually identified issues not covered by existing rules, the equipment prompts users or developers to add new rules. Through the closed-loop optimization of active learning, online fine-tuning, feature repository, and rule adaptation, we ensure that each verification model and rule base can continuously evolve with use to adapt to increasingly diverse document proofreading needs. Furthermore, each major model or rule update undergoes offline validation set testing and manual sampling verification to ensure that the overall system performance is not reduced.

[0240] This embodiment uses a preset quality grader to score each difference item in the multi-source difference information to obtain the risk level of each difference item. When there is a first-class risk level lower than the preset level, the first-class difference item corresponding to the first-class risk level is corrected by a preset correction strategy. When there is a second-class risk level not lower than the preset level, the second-class difference item corresponding to the second-class risk level is displayed on a preset interface so that the user can correct the second-class difference item. This combines artificial intelligence and human intervention in the correction operation, effectively improving the accuracy of the correction.

[0241] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the artificial intelligence-based multimodal document automatic proofreading method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0242] This application also provides an artificial intelligence-based multimodal document automatic proofreading system; please refer to [reference needed]. Figure 4 , Figure 4 This is a schematic diagram of the module structure of the AI-based multimodal document automatic proofreading system of this application. The AI-based multimodal document automatic proofreading system includes:

[0243] Input layer 10 is used to receive the original document.

[0244] Parsing layer 20 is used to convert the original document into structured data consisting of multiple data of different format types.

[0245] The artificial intelligence engine cluster 30 is used to select the corresponding verification model according to the data of each of the aforementioned format types to verify the structured data and obtain the difference items corresponding to the data of each of the aforementioned format types.

[0246] The decision layer 40 is used to determine the risk level corresponding to each of the aforementioned differences through a preset quality grader.

[0247] The decision layer 40 is also used to correct each of the difference items according to each of the risk levels.

[0248] The AI-based multimodal document automatic proofreading system provided in this application, employing the AI-based multimodal document automatic proofreading method described in the above embodiments, can solve the technical problem of low proofreading efficiency caused by reliance on manual word-by-word verification in existing technologies. Compared with existing technologies, the beneficial effects of the AI-based multimodal document automatic proofreading system provided in this application are the same as those of the AI-based multimodal document automatic proofreading method provided in the above embodiments, and other technical features of the AI-based multimodal document automatic proofreading system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0249] This application provides an AI-based multimodal document automatic proofreading device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the AI-based multimodal document automatic proofreading method in Embodiment 1 above.

[0250] The following is for reference. Figure 5 , Figure 5 This is a schematic diagram of the structure of the AI-based multimodal document automatic proofreading device of this application. The AI-based multimodal document automatic proofreading device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), vehicle terminals (such as vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers.

[0251] like Figure 5 As shown, the AI-based multimodal document automatic proofreading device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the AI-based multimodal document automatic proofreading device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the AI-based multimodal document automatic proofreading device to wirelessly or wiredly communicate with other devices to exchange data. Although the figure shows an AI-based multimodal document automatic proofreading device with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0252] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0253] The AI-based multimodal document automatic proofreading device provided in this application, employing the AI-based multimodal document automatic proofreading method described in the above embodiments, can solve the technical problem of low proofreading efficiency caused by reliance on manual word-by-word verification in existing technologies. Compared with the prior art, the beneficial effects of the AI-based multimodal document automatic proofreading device provided in this application are the same as those of the AI-based multimodal document automatic proofreading method provided in the above embodiments, and other technical features in this AI-based multimodal document automatic proofreading device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0254] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0255] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An artificial intelligence-based multi-modal document automatic proofreading method, characterized in that, The artificial intelligence-based multi-modal document automatic proofreading method comprises: receiving an original document and converting the original document into structured data composed of a plurality of different format types of data, including text data, table data, formula data and image data; selecting a corresponding verification model according to each format type of data to proofread the structured data, and obtaining a difference item corresponding to each format type of data; determining the risk level corresponding to each difference item through a preset quality grader; correcting each difference item according to each risk level; The difference item at least includes a text difference item, a table difference item, a formula difference item and an image difference item, and the step of selecting a corresponding verification model according to each format type of data to proofread the structured data and obtaining a difference item corresponding to each format type of data comprises: verify the text data through a preset text verification model, a pre-trained language model and a preset format rule to obtain the text difference item; verify the table data through a preset table verification model and a preset analytical database to obtain the table difference item; verify the formula data through a first CLIP verification model to obtain the formula difference item; verify the image data through a second CLIP verification model to obtain the image difference item; The text difference item at least includes a text semantic difference item and a text content difference item, and the step of verifying the text data through a preset text verification model, a pre-trained language model and a preset format rule to obtain the text difference item comprises: convert the text data into a document graph composed of text blocks connected to each other based on the content layout information of the text data through a preset text verification model, and determine the feature vector of each text graph node, wherein the text graph node in the document graph is a text block in the text data; determine the anomaly score of each text graph node, wherein the anomaly score of any text graph node is determined based on the feature vector of the any text graph node and the feature vector of the adjacent text graph node; determine whether the anomaly score of each text graph node reaches a preset score; when there is a target text graph node with an anomaly score reaching the preset score, mark the target text graph node as the text semantic difference item; perform content verification on the text data based on a pre-trained language model and a preset format rule; when there is a target text content with a failed content verification, mark the target text content as the text content difference item. 2.The artificial intelligence-based multi-modal document proofreading method of claim 1, wherein, The format type at least includes JSON format, CSV format, LaTeX format and image format, and the step of receiving an original document and converting the original document into structured data composed of a plurality of different format types of data comprises: receive an original document and pre-process the original document to obtain a pre-processed document; identify the text block, table area, picture area and formula area in the pre-processed document; The text block is recognized by an optical character recognition model to obtain the JSON format text data; Each original table in the table region is recognized by a preset table analysis model to obtain the CSV format table data, and the table data includes extracted tables corresponding to the original tables; Each original formula image in the formula region is recognized by a preset formula recognition model to obtain the LaTeX format formula data, and the formula data includes extracted formulas corresponding to the original formula images; Each original image in the picture region is subjected to image cropping to obtain image data in the image format, and the image data includes cropped images corresponding to the original images. 3.The AI-based multi-modal document proofreading method of claim 2, wherein, The table difference items at least include table structure difference items and table content difference items, and the step of checking the table data by a preset table checking model and a preset analytical database to obtain the table difference items includes: Determining row and column relationship information of each extracted table in the table data; Converting the corresponding extracted table into a structure diagram composed of interconnected cells based on each row and column relationship information by a preset table checking model, and a table diagram node in the structure diagram is a cell of the extracted table; Calculating structure similarity between each structure diagram and a preset standard table structure diagram; Determining whether each structure similarity is lower than a preset structure similarity threshold; When there is a target structure diagram with a structure similarity lower than the preset structure similarity threshold, marking the extracted table corresponding to the target structure diagram as the table structure difference item; Performing data consistency checking on each extracted table based on a preset analytical database; When there is a target extracted table with failed data consistency checking, marking the target extracted table as the table content difference item. 4.The AI-based multi-modal document proofreading method of claim 2, wherein, The formula difference items at least include formula semantic difference items and formula structure difference items, and the step of checking the formula data by a first CLIP checking model to obtain the formula difference items includes: Rendering each extracted formula in the formula data into a corresponding rendered formula image; Calculating a rendered formula feature vector of each rendered formula image by a first CLIP checking model; Calculating an original formula feature vector corresponding to each original formula image by the first CLIP checking model; Calculating a cosine similarity between each rendered formula feature vector and the corresponding original formula feature vector in a high-dimensional semantic space; When there is a first type of extracted formula with a cosine similarity lower than a preset similarity threshold, marking the first type of extracted formula as the formula semantic difference item; Calculating a hash value difference between each rendered formula image and the corresponding original formula image; When there is a second type of extracted formula with a hash value difference higher than a first preset difference threshold, marking the second type of extracted formula as the formula structure difference item. 5.The AI-based multi-modal document proofreading method of claim 2, wherein, The image difference item at least includes an image semantic difference item, an image structure difference item and an image interception difference item, the step of verifying the image data by the second CLIP verification model to obtain the image difference item, comprising: Preprocessing each of the intercepted images in the image data to obtain an extracted image corresponding to each of the intercepted images; Calculating an extracted image semantic vector of each of the extracted images by a second CLIP verification model; Calculating an original image semantic vector of each of the original images by the second CLIP verification model; Calculating the cosine similarity between each of the extracted image semantic vectors and the corresponding original image semantic vectors; When there is a first type of extracted image with a cosine similarity lower than a preset similarity threshold, marking the first type of extracted image as the image semantic difference item; Calculating the hash value difference between each of the extracted images and the corresponding original images; When there is a second type of extracted image with a hash value difference higher than a second preset difference threshold, marking the second type of extracted image as the image structure difference item; Intercepting verification of each of the extracted images; When there is a third type of extracted image with failed intercepting verification, marking the third type of extracted image as the image interception difference item. 6.The AI-based multi-modal document proofreading method of claim 5, wherein, The step of intercepting verification of each of the extracted images, comprising: extracting a scale-invariant feature transform feature point in each of the original images, extracting a scale-invariant feature transform feature point in each of the extracted images; matching the scale-invariant feature transform feature points in each of the extracted images with the scale-invariant feature transform feature points in the corresponding original images to obtain a plurality of matching feature point pairs; determining a homography matrix according to each of the matching feature point pairs; aligning each of the extracted images with the corresponding original images by the homography matrix to perform intercepting verification on each of the extracted images; The step of marking the third type of extracted image as the image interception difference item when there is a third type of extracted image with failed intercepting verification, comprising: When there is a third type of extracted image that is not aligned with the corresponding original image, marking the third type of extracted image as the image interception difference item.

7. The artificial intelligence-based multi-modal document proofreading method according to any one of claims 2 to 6, wherein, The step of determining a risk level corresponding to each of the difference items by a preset quality classifier, comprising: obtaining multi-source difference information composed of the text difference item, the table difference item, the formula difference item and the image difference item; scoring each of the difference items in the multi-source difference information by a preset quality classifier to obtain a risk level of each of the difference items; The step of correcting each of the difference items according to each of the risk levels, comprising: determining whether each of the risk levels reaches a preset level; When there is a first type of risk level lower than the preset level, correcting a first type of difference item corresponding to the first type of risk level by a preset correction strategy; When there is a second type of risk level not lower than the preset level, displaying a second type of difference item corresponding to the second type of risk level on a preset interface to enable a user to correct the second type of difference item.

8. An artificial intelligence-based multi-modal document auto-correction system, characterized by, The system comprises: an input layer for receiving an original document; An analysis layer is configured to convert the original document into structured data composed of data of multiple different format types, including text data, table data, formula data, and image data; An artificial intelligence engine cluster is configured to select a corresponding verification model according to each format type of data to correct the structured data, and obtain a difference item corresponding to each format type of data; the difference item at least includes a text difference item, a table difference item, a formula difference item, and an image difference item; the step of selecting a corresponding verification model according to each format type of data to correct the structured data, and obtaining a difference item corresponding to each format type of data, includes: verifying the text data through a preset text verification model, a pre-trained language model, and a preset format rule to obtain the text difference item; verifying the table data through a preset table verification model and a preset analytical database to obtain the table difference item; verifying the formula data through a first CLIP verification model to obtain the formula difference item; verifying the image data through a second CLIP verification model to obtain the image difference item; the text difference item at least includes a text semantic difference item and a text content difference item; the step of verifying the text data through a preset text verification model, a pre-trained language model, and a preset format rule to obtain the text difference item, includes: converting the text data into a document graph composed of text blocks connected to each other based on content layout information of the text data through a preset text verification model, and determining a feature vector of each text graph node; the text graph node in the document graph is a text block in the text data; determining an abnormal score of each text graph node, the abnormal score of any text graph node being determined based on a feature vector of the any text graph node and a feature vector of an adjacent text graph node; determining whether the abnormal score of each text graph node reaches a preset score; when there is a target text graph node with an abnormal score reaching the preset score, marking the target text graph node as the text semantic difference item; performing content verification on the text data based on a pre-trained language model and a preset format rule; when there is a target text content with a failed content verification, marking the target text content as the text content difference item; A decision layer is configured to determine a risk level corresponding to each difference item through a preset quality grader; The decision layer is further configured to correct each difference item according to each risk level.

Images