Document content restoration method and system based on artificial intelligence

Abstract

The invention relates to the technical field of document processing, and discloses a document content restoration method and system based on artificial intelligence, and the method comprises the steps: collecting document image information as an input source through a camera, and then carrying out the page scanning of the preprocessed document image information through a multi-modal layout analysis model, positioning various layout elements, obtaining physical layout parameters, deducing a geometric constraint relation through a geometric constraint algorithm, and generating a structured layout model; carrying out multi-language character recognition, picture graph classification and metadata extraction, formula format conversion and table structure detection and reconstruction, and outputting corresponding structured data; distinguishing different semantic blocks and constructing a tree hierarchical structure by combining a context semantic encoder with a conditional random field and text features, and extracting and restoring text fonts and paragraph format styles; and reading the basic metadata of the document, and restoring and complementing the associated information of the document elements based on the large model to complete the information improvement work. And efficient and accurate restoration of the document content is realized.

Description

Technical Field

[0001] This invention relates to the field of document processing technology, specifically to a document content restoration method and system based on artificial intelligence. Background Technology

[0002] In the process of document digitization, there is often a need to restore damaged paper documents, low-quality scanned copies, and unstructured electronic documents. Traditional document restoration technologies have the following shortcomings:

[0003] 1. Limited layout parsing capabilities, making it difficult to handle complex layouts such as multi-column mixed layouts and nested text and images;

[0004] Second, the accuracy of multimodal content recognition is insufficient, especially in the recognition of handwriting, complex formulas, and tables spanning multiple pages.

[0005] Third, the lack of hierarchical logic in semantic structure reconstruction leads to problems such as confusion between title and body text and loss of formatting.

[0006] Therefore, there is an urgent need for a document content restoration solution that integrates artificial intelligence technology to achieve accurate restoration of layout, content, format and semantics in multiple dimensions. Summary of the Invention

[0007] (a) Technical problems to be solved

[0008] To address the shortcomings of existing technologies, this invention provides a document content restoration method and system based on artificial intelligence. It has the advantages of solving problems such as inaccurate layout parsing, weak multimodal recognition capabilities, and lack of semantic structure reconstruction in existing technologies, and achieving efficient and accurate restoration of document content.

[0009] (II) Technical Solution

[0010] To achieve the above objectives, the present invention provides the following technical solution: a document content restoration method based on artificial intelligence, comprising the following steps:

[0011] Step 1: Acquire the original document image through the image acquisition unit and perform preprocessing operations to generate a standardized visual feature tensor. Then, input the visual feature tensor into the multimodal layout analysis model, utilizing its cross-modal attention mechanism to extract the visual and textual features of the document, and output the category labels and candidate region coordinates for each layout element. Based on this, a geometric constraint logic layer is introduced. Combining the extracted physical layout parameters, a preset relative position algorithm is used to logically correct and fine-tune the boundaries of the candidate regions output by the model, ultimately generating a structured layout model that combines the generalization ability of deep learning with the accuracy of geometric logic.

[0012] Step 2: Perform multilingual character recognition, image and graphic classification and metadata extraction, formula format conversion, table structure detection and reconstruction, and output the corresponding structured data;

[0013] Step 3: Use the BERT-CRF sequence labeling model to distinguish different semantic blocks and construct a tree-like hierarchical structure, while extracting and restoring the text font and paragraph formatting styles;

[0014] Step Four: Semantic Reasoning and Logical Reconstruction Based on a Large Language Model. The structured semantic hierarchy data generated in Step Three and the physical layout parameters from Step One are transformed into structured prompt words and input into a pre-trained generative large model. Utilizing the large model's semantic understanding capabilities and extensive knowledge base, the OCR recognition results are proofread and corrected based on semantic fluency; according to the semantic relevance of the text content, images and captions, tables and titles, and body text and cross-page paragraphs are automatically associated; completing the final refinement and output of the information.

[0015] Preferably, in step one, the preprocessed document image information includes geometric correction, illumination normalization, background removal, and resolution optimization performed on the acquired document image information in advance. For the preprocessed document image information, text is extracted in advance by OCR. The preprocessed document image is first converted into a visual feature tensor and a text feature sequence, and then input into LayoutLMv3. The visual region and text content are associated through a cross-modal attention mechanism. Then, the categories of different layout elements are identified through a sequence labeling task, and the candidate region coordinates of each element are output.

[0016] Preferably, in step one, locating various layout elements and obtaining physical layout parameters includes:

[0017] Set the coordinate system of the document image, with the top left corner of the document image as the origin. , represent The axis is in the horizontal right direction. represent The axis is in the vertical downward direction. Represents the width of the image. Represents the height of the image;

[0018] S11.1 Global layout parameter extraction and derivation;

[0019] Let any text-type layout element be... , Represents the total number of layout elements. In the image coordinate system, the bounding box is determined by the coordinates of four vertices, which can be simplified to the coordinates of the top left corner and the bottom right corner.

[0020] ;

[0021] in, Represents text-based layout elements The coordinates of the top left corner, Represents text-based layout elements The coordinates of the upper right corner, , , , ;

[0022] Deducing text layout elements width ,high Center coordinates ;

[0023] ;

[0024] ;

[0025] ;

[0026] S11.2, Define two elements and image / graphic layout elements Relative position:

[0027] S11.2a, Text-based layout elements and image / graphic layout elements Horizontal distance, horizontal spacing ;

[0028] like exist Left side:

[0029] ;

[0030] like exist Right side:

[0031] ;

[0032] S11.2b, Text-based layout elements and image / graphic layout elements Vertical distance, vertical spacing ;

[0033] like exist Above:

[0034] ;

[0035] like exist Below:

[0036] ;

[0037] S11.2c, Text-based layout elements and image / graphic layout elements overlapping area With overlap rate Define the bounding box of the overlapping region as:

[0038] ;

[0039] ;

[0040] ;

[0041] in, Represents text-based layout elements area, Represents image and graphic layout elements The area;

[0042] S11.3 Normalize the global layout parameters:

[0043] ;

[0044] in, , , , This represents the normalized layout parameters, which are directly used for cross-document layout logic comparison.

[0045] Preferably, in step one, inferring geometric constraint relationships using a geometric constraint algorithm includes:

[0046] S12, Horizontal alignment constraint;

[0047] ;

[0048] in, The error threshold representing the level;

[0049] S13, Vertical alignment constraint;

[0050] ;

[0051] in, The vertical error threshold;

[0052] S14, Includes constraints;

[0053] and and and

[0054] S15, Adjacency Constraints:

[0055] or

[0056] S16, Hierarchical Constraints:

[0057] and and

[0058] S17, Multi-column constraints;

[0059] ;

[0060] in, The threshold for approximating the width is represented. The threshold is approximated by the central ordinate.

[0061] Preferably, generating a structured layout model involves determining the geometric constraints of each preprocessed document image, associating element attributes with constraint results, aggregating global layout rules, and finally integrating and encapsulating them into a structured layout model in a standardized data format.

[0062] Preferably, step two includes:

[0063] S2.1 Multilingual character recognition:

[0064] The text positions are extracted from the structured layout model, and the text content is directly recognized and output by the multilingual OCR recognition tool on the extracted text region image. The language of the text is directly output by the language determination module built into the recognition tool, and the recognition confidence score is directly output by the recognition tool. At the same time, the hierarchical relationship of the text and the paragraph relationship are determined by geometric constraints, and the final output is structured text data.

[0065] S2.2. Image and Graphic Classification and Metadata Extraction:

[0066] The positions of images and graphics are extracted from the structured layout model. The image classification tool directly outputs the type and classification confidence. Based on the adjacency constraint results in step one, the text element IDs and contents that are adjacent to the image and graphic elements are directly extracted. The color extraction tool directly analyzes the image and graphic regions and outputs the main color tone in the core information. Finally, the structured image and graphic data is output.

[0067] S2.3. Formula Format Conversion:

[0068] Formula locations are extracted from the structured layout model. The formula recognition tool extracts features and parses the structure of the extracted formula region image, directly outputting standardized text formulas and recognition confidence scores, and finally outputting structured formula data.

[0069] S2.4. Table Structure Detection and Reconstruction:

[0070] Extract the table area locations from the structured layout model in step one. The table detection tool directly outputs the row boundaries, column boundaries, and cell position coordinates of the table. Based on the row and column boundaries, calculate and restore the total number of rows and columns of the table. Combine the cell position and size to determine the cell merging situation. The OCR tool directly recognizes and outputs the internal text and numerical content of each cell area, and finally outputs the structured table data.

[0071] Preferably, step three includes:

[0072] S3.1 Semantic block partitioning and tree-like hierarchical construction:

[0073] Extract text sequences from the structured text data, structured image data, structured formula data, and structured table data output in step two, and integrate them with the global layout parameters from the structured layout model in step one. , , , , , , The geometric constraint results are used to construct a multi-dimensional input feature vector. The constructed multi-dimensional feature vector is input into the BERT-CRF sequence labeling model. The semantic features of the text context are encoded by BERT and the sequence label dependencies are decoded by CRF. The semantic label of each text token is output. Independent semantic blocks are generated by aggregating the label sequences and assigned a unique ID. The relationship between semantic blocks is determined based on the geometric constraint rules in step one, forming structured semantic hierarchical data.

[0074] S3.2 Extraction and restoration of text font and paragraph formatting styles:

[0075] The visual region coordinates of text elements are extracted from the structured layout model in Step 1. The stroke outlines and character shape features of the extracted text region images are extracted using a font recognition tool. The font type, font size, whether it is bold, italic, underlined, and the recognition confidence are directly output. The RGB values ​​of the text region pixels are analyzed by a color extraction tool, and the main color tone of the text is directly output. The multi-column layout attributes are determined by combining the multi-column constraints in Step 1 and associated with the corresponding nodes of the structured semantic hierarchy data in S3.1 to form complete structured document data.

[0076] Preferably, step four includes:

[0077] S4.1 Structured Data Aggregation and Prompt Formatting:

[0078] Extract core data from the complete structured document data output in step three, including the global layout parameters from step one, the metadata of various types of structured data from step two, and the complete structured document data from step three. At the same time, read the original metadata of the document file, classify and encode the above aggregated data, organize it using key-value pairs and hierarchical indentation format, construct structured prompt words in combination with task instructions, and finally output a standardized prompt word dataset.

[0079] S4.2 Configuring the Large Language Model and Setting Up the Inference Environment:

[0080] Based on the core task requirements of step four, select an appropriate pre-trained generative large language model, configure the core parameters of model inference, build the model inference environment, complete the input adaptation processing of the prompt word dataset, construct the entire process execution link from input to inference to output, and form the basic configuration for model inference.

[0081] S4.3, Large Language Model-Driven Semantic Optimization and Association Completion:

[0082] The standardized prompt word dataset of S4.1 is input into the configured large language model. Combining the model's semantic understanding and knowledge reasoning capabilities, a multi-dimensional optimization and completion task is performed, including:

[0083] Perform semantic proofreading on OCR-recognized text, correcting typos and omissions, optimizing sentence fluency, and re-judging low-confidence content;

[0084] Based on content relevance and layout constraints, complete the missing relationships between formulas and body text, tables and titles, and images and text descriptions;

[0085] Connecting paragraphs across pages with semantic blocks,

[0086] Optimize the semantic hierarchy; label the matching basis and confidence level of each type of optimized completion result, and finally output structured semantic optimization and related completion data;

[0087] S4.4 Information Integration, Verification, and Standardized Output:

[0088] By integrating the aggregated structured data of S4.1 with the semantically optimized and associated data of S4.3, verifying the consistency of the association logic, associating the optimized and associated information with the corresponding nodes of the basic structured data, encapsulating it into a standardized format according to application requirements, generating data version numbers and check codes to ensure integrity, and finally outputting complete and perfect document data to form a standardized document information system.

[0089] An AI-based document content restoration system, applied to document content restoration methods, includes an image information acquisition module, a layout parsing module, a multimodal recognition module, a semantic reconstruction module, and an information restoration module.

[0090] Preferably, the image information acquisition module uses a camera to acquire document image information as the input source;

[0091] The layout parsing module is used to locate various layout elements and obtain physical layout parameters, and then infer geometric constraint relationships through geometric constraint algorithms to generate a structured layout model.

[0092] The multimodal recognition module performs multilingual text recognition, image and graphic classification and metadata extraction, formula format conversion, table structure detection and reconstruction, and outputs corresponding structured data.

[0093] The semantic reconstruction module combines text features to distinguish different semantic blocks and constructs a tree-like hierarchical structure, while extracting and restoring the text font and paragraph formatting styles.

[0094] The information restoration module reads the basic metadata of the document and restores and completes the document element association information based on the large model, thus completing the information improvement work.

[0095] Compared with existing technologies, the present invention provides a document content restoration method and system based on artificial intelligence, which has the following beneficial effects:

[0096] 1. This invention has multi-dimensional restoration capabilities: it integrates layout analysis, multimodal recognition, semantic reconstruction and information restoration to achieve a comprehensive restoration of the document's physical structure, content elements, format styles and logical relationships;

[0097] 2. This invention is adaptable to complex scenarios: it supports complex scenarios such as multilingual mixed documents, handwriting, cross-page tables, and vertically formatted ancient books, with high recognition accuracy;

[0098] 3. This invention has automated processing efficiency: by reducing human intervention through artificial intelligence algorithms, batch processing efficiency is improved, and the cost of document digitization is significantly reduced. Attached Figure Description

[0099] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0100] Figure 2 This is a schematic diagram of the system of the present invention. Detailed Implementation

[0101] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention. It is worth noting that the parts not disclosed in this application are all prior art, and will not be elaborated on here.

[0102] Please see Figures 1-2 A document content restoration method based on artificial intelligence includes the following steps:

[0103] Step 1: Acquire the original document image through the image acquisition unit and perform preprocessing operations to generate a standardized visual feature tensor. Then, input the visual feature tensor into the multimodal layout analysis model, utilizing its cross-modal attention mechanism to extract the visual and textual features of the document, and output the category labels and candidate region coordinates for each layout element. Based on this, a geometric constraint logic layer is introduced. Combining the extracted physical layout parameters, a preset relative position algorithm is used to logically correct and fine-tune the boundaries of the candidate regions output by the model, ultimately generating a structured layout model that combines the generalization ability of deep learning with the accuracy of geometric logic.

[0104] The preprocessed document image information includes geometric correction, illumination normalization, background removal and resolution optimization of the acquired document image information in advance. For the preprocessed document image information, text is extracted in advance by OCR. The preprocessed document image needs to be converted into visual feature tensors and text feature sequences, and then input into LayoutLMv3. The visual region and text content are associated through a cross-modal attention mechanism. Then, the category of different layout elements is identified through a sequence labeling task, and the candidate region coordinates of each element are output.

[0105] Geometric correction: Based on edge detection algorithm, document outline is identified, and perspective transformation is used to correct image tilt and distortion. Then, the tilted image is mapped to a standard rectangular plane to ensure that the document content is parallel to the image coordinate system.

[0106] Illumination normalization: An adaptive histogram equalization algorithm is used to deal with uneven illumination, eliminate reflective and shadow areas, and process image brightness in blocks to improve details in dark areas while avoiding overexposure in bright areas, so that the overall grayscale value distribution of the image is uniform.

[0107] Background removal: Based on threshold segmentation and morphological operations, the document body and background are separated. First, a grayscale threshold is set to binarize the image, then small noise areas are removed, and finally the clean document body area is retained, reducing the interference of irrelevant pixels on the model.

[0108] Resolution optimization: The preprocessed image is scaled to the resolution adapted to the LayoutLMv3 model. Bilinear interpolation algorithm is used to ensure the clarity of the scaled image, while the aspect ratio of the image is unified to facilitate batch processing of the model.

[0109] Locate various layout elements and obtain physical layout parameters, including:

[0110] Set the coordinate system of the document image, with the top left corner of the document image as the origin. , represent The axis is in the horizontal right direction. represent The axis is in the vertical downward direction. Represents the width of the image. Represents the height of the image;

[0111] S11.1 Extract and derive global layout parameters;

[0112] S11.2, Define two elements and image / graphic layout elements Relative position:

[0113] S11.2a, Text-based layout elements and image / graphic layout elements Horizontal distance, horizontal spacing ;

[0114] S11.2b, Text-based layout elements and image / graphic layout elements Vertical distance, vertical spacing ;

[0115] S11.2c, Text-based layout elements and image / graphic layout elements overlapping area With overlap rate Define the bounding box of the overlapping region;

[0116] S11.3 Normalize the global layout parameters;

[0117] Inferring geometric constraint relationships through geometric constraint algorithms includes:

[0118] S12, Horizontal alignment constraint;

[0119] S13, Vertical alignment constraint;

[0120] S14, Includes constraints;

[0121] S15, Adjacency Constraints:

[0122] S16, Hierarchical Constraints:

[0123] S17, Multi-column constraints;

[0124] The geometric constraints of each preprocessed document image are determined one by one, then the element attributes and constraint results are associated, global layout rules are aggregated, and finally integrated and encapsulated into a structured layout model with standardized data format.

[0125] Step one involves capturing raw document images using a camera and using the LayoutLMv3 model to perform layout parsing. This transforms disordered image pixel information into a structured layout model. The preprocessing stage eliminates interference factors such as distortion and uneven lighting during image acquisition, ensuring the accuracy of subsequent analysis. LayoutLMv3 achieves deep correlation between visual features and text features, accurately locates various layout elements and extracts physical parameters. The geometric constraint algorithm further explores the logical relationships between elements, such as alignment, adjacency, and inclusion, providing a unified spatial reference framework for subsequent multimodal recognition and semantic reconstruction. This avoids misidentification of elements or semantic fragmentation caused by layout chaos.

[0126] Step Two: Perform multilingual character recognition, image and graphic classification and metadata extraction, formula format conversion, table structure detection and reconstruction, and output corresponding structured data, including:

[0127] S2.1 Multilingual character recognition:

[0128] The text positions are extracted from the structured layout model, and the text content is directly recognized and output by the multilingual OCR recognition tool on the extracted text region image. The language of the text is directly output by the language determination module built into the recognition tool, and the recognition confidence score is directly output by the recognition tool. At the same time, the hierarchical relationship of the text and the paragraph relationship are determined by geometric constraints, and the final output is structured text data.

[0129] S2.2. Image and Graphic Classification and Metadata Extraction:

[0130] The positions of images and graphics are extracted from the structured layout model. The image classification tool directly outputs the type and classification confidence. Based on the adjacency constraint results in step one, the text element IDs and contents that are adjacent to the image and graphic elements are directly extracted. The color extraction tool directly analyzes the image and graphic regions and outputs the main color tone in the core information. Finally, the structured image and graphic data is output.

[0131] S2.3. Formula Format Conversion:

[0132] Formula locations are extracted from the structured layout model. The formula recognition tool extracts features and parses the structure of the extracted formula region image, directly outputting standardized text formulas and recognition confidence scores, and finally outputting structured formula data.

[0133] S2.4. Table Structure Detection and Reconstruction:

[0134] Extract the table area location from the structured layout model in step one. The table detection tool directly outputs the row boundaries, column boundaries, and cell position coordinates of the table. Based on the row and column boundaries, calculate and restore the total number of rows and columns of the table. Combine the cell position and size to calculate and determine the cell merging situation. The OCR tool directly recognizes and outputs the internal text and numerical content of each cell area, and finally outputs the structured table data.

[0135] Step two achieves accurate recognition and structured output for different types of document elements. Multilingual text recognition breaks through language limitations. Combined with geometric constraints to determine hierarchy and paragraph relationships, it gives text information spatial semantic attributes. Image and graphic classification and metadata extraction not only clarify the types of non-text elements, but also associate adjacent text information, enriching the semantic dimension of elements. Formula format conversion transforms image-based formulas into standardized text formulas, solving the pain point that formulas cannot be directly retrieved and edited. Table structure detection and reconstruction restores the row and column logic and cell content of the table, realizing the structured extraction of table data, and providing multi-type, high-precision structured data support for subsequent semantic reconstruction and information improvement.

[0136] Step 3: Using the BERT-CRF sequence labeling model combined with text features, different semantic blocks are distinguished and a tree-like hierarchical structure is constructed. Simultaneously, the text font and paragraph formatting styles are extracted and restored, including:

[0137] S3.1 Semantic block partitioning and tree-like hierarchical construction:

[0138] Extract text sequences from the structured text data, structured image data, structured formula data, and structured table data output in step two, and integrate them with the global layout parameters from the structured layout model in step one. , , , , , , The geometric constraint results are used to construct a multi-dimensional input feature vector. The constructed multi-dimensional feature vector is input into the BERT-CRF sequence labeling model. The semantic features of the text context are encoded by BERT and the sequence label dependencies are decoded by CRF. The semantic label of each text token is output. Independent semantic blocks are generated by aggregating the label sequences and assigned a unique ID. The relationship between semantic blocks is determined based on the geometric constraint rules in step one, forming structured semantic hierarchical data.

[0139] S3.2 Extraction and restoration of text font and paragraph formatting styles:

[0140] The visual region coordinates of text elements are extracted from the structured layout model in Step 1. The stroke outlines and character shape features of the extracted text region images are extracted using a font recognition tool. The font type, font size, whether it is bold, italic, underline, and recognition confidence are directly output. The RGB values ​​of the text region pixels are analyzed by a color extraction tool, and the main color tone of the text is directly output. The multi-column layout attributes are determined by combining the multi-column constraints in Step 1 and associated with the corresponding nodes of the structured semantic hierarchy data in S3.1 to form complete structured document data.

[0141] Step 3 uses the BERT-CRF model to upgrade from content recognition to semantic understanding, solving the problem that simple layout-based element division cannot reflect semantic hierarchy. The tree-like hierarchical structure built based on geometric constraint rules makes the document content form a clear semantic system. At the same time, the extraction and restoration of font and paragraph formatting completely preserves the original layout information of the document, which not only meets the needs of semantic understanding, but also takes into account the needs of document format restoration.

[0142] Step Four: Semantic Reasoning and Logical Reconstruction Based on a Large Language Model. The structured semantic hierarchy data generated in Step Three and the physical layout parameters from Step One are converted into structured prompt words and input into a pre-trained generative large model. Utilizing the large model's semantic understanding capabilities and extensive knowledge base, the OCR recognition results are proofread and corrected based on semantic fluency; based on the semantic relevance of the text content, images and captions, tables and titles, and body text and cross-page paragraphs are automatically associated; the final information is completed and output, including:

[0143] S4.1 Structured Data Aggregation and Prompt Formatting:

[0144] Extract core data from the complete structured document data output in step three, including the global layout parameters from step one, the metadata of various types of structured data from step two, and the complete structured document data from step three. At the same time, read the original metadata of the document file, classify and encode the above aggregated data, organize it using key-value pairs and hierarchical indentation format, construct structured prompt words in combination with task instructions, and finally output a standardized prompt word dataset.

[0145] S4.2 Configuring the Large Language Model and Setting Up the Inference Environment:

[0146] Based on the core task requirements of step four, select an appropriate pre-trained generative large language model, configure the core parameters of model inference, build the model inference environment, complete the input adaptation processing of the prompt word dataset, construct the entire process execution link from input to inference to output, and form the basic configuration for model inference.

[0147] S4.3, Large Language Model-Driven Semantic Optimization and Association Completion:

[0148] The standardized prompt word dataset of S4.1 is input into the configured large language model. Combining the model's semantic understanding and knowledge reasoning capabilities, a multi-dimensional optimization and completion task is performed, including:

[0149] Perform semantic proofreading on OCR-recognized text, correcting typos and omissions, optimizing sentence fluency, and re-judging low-confidence content;

[0150] Based on content relevance and layout constraints, complete the missing relationships between formulas and body text, tables and titles, and images and text descriptions;

[0151] Connecting paragraphs across pages with semantic blocks,

[0152] Optimize the semantic hierarchy; label the matching basis and confidence level of each type of optimized completion result, and finally output structured semantic optimization and related completion data;

[0153] S4.4 Information Integration, Verification, and Standardized Output:

[0154] Integrate the aggregated structured data of S4.1 with the semantically optimized and associated data of S4.3, verify the consistency of the association logic, associate the optimized and associated information with the corresponding nodes of the basic structured data, encapsulate it into a standardized format according to application requirements, generate data version numbers and check codes to ensure integrity, and finally output complete and perfect document data to form a standardized document information system.

[0155] Step four leverages the semantic understanding and reasoning capabilities of the large language model to accurately correct text errors identified in the preceding OCR process, ensuring content accuracy. Simultaneously, it breaks down the isolation of multimodal elements, achieving semantic association completion for formulas and body text, tables and titles, and cross-page logical connections. Finally, through integration, verification, and standardized encapsulation, a complete and standardized document information system is formed, providing high-quality structured data support for subsequent document storage, retrieval, and secondary applications, significantly improving the completeness, logic, and practical value of the overall restoration results.

[0156] An artificial intelligence-based document content restoration system, applied to document content restoration methods, includes an image information acquisition module, a layout parsing module, a multimodal recognition module, a semantic reconstruction module, and an information restoration module;

[0157] The image information acquisition module uses a camera to capture document image information as the input source;

[0158] The layout parsing module is used to locate various layout elements and obtain physical layout parameters, and then infer geometric constraint relationships through geometric constraint algorithms to generate a structured layout model;

[0159] The multimodal recognition module performs multilingual text recognition, image and graphic classification and metadata extraction, formula format conversion, table structure detection and reconstruction, and outputs corresponding structured data.

[0160] The semantic reconstruction module combines text features to distinguish different semantic blocks and constructs a tree-like hierarchical structure, while extracting and restoring the text font and paragraph formatting styles;

[0161] The information restoration module reads the basic metadata of the document and restores and completes the document element association information based on the large model, thus completing the information improvement work.

[0162] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A document content restoration method based on artificial intelligence, characterized in that, Includes the following steps: Step 1: Acquire the original document image through the image acquisition unit and perform preprocessing operations to generate a standardized visual feature tensor. Then, input the visual feature tensor into the multimodal layout analysis model, utilizing its cross-modal attention mechanism to extract the visual and textual features of the document, and output the category labels and candidate region coordinates for each layout element. Based on this, a geometric constraint logic layer is introduced. Combining the extracted physical layout parameters, a preset relative position algorithm is used to logically correct and fine-tune the boundaries of the candidate regions output by the model, ultimately generating a structured layout model that combines the generalization ability of deep learning with the accuracy of geometric logic. Step 2: Perform multilingual character recognition, image and graphic classification and metadata extraction, formula format conversion, table structure detection and reconstruction, and output the corresponding structured data; Step 3: Use a contextual semantic encoder combined with a conditional random field and text features to distinguish different semantic blocks and construct a tree-like hierarchical structure, while extracting and restoring the text font and paragraph formatting styles; Step Four: Semantic Reasoning and Logical Reconstruction Based on a Large Language Model. The structured semantic hierarchy data generated in Step Three and the physical layout parameters from Step One are transformed into structured prompt words and input into a pre-trained generative large model. Utilizing the large model's semantic understanding capabilities and extensive knowledge base, the OCR recognition results are proofread and corrected based on semantic fluency; according to the semantic relevance of the text content, images and captions, tables and titles, and body text and cross-page paragraphs are automatically associated; completing the final refinement and output of the information.

2. The document content restoration method based on artificial intelligence according to claim 1, characterized in that, In step one, the preprocessed document image information includes geometric correction, illumination normalization, background removal, and resolution optimization of the acquired document image information in advance. For the preprocessed document image information, text is extracted in advance by OCR. The preprocessed document image needs to be converted into a visual feature tensor and a text feature sequence, and then input into LayoutLMv3. The visual region and text content are associated through a cross-modal attention mechanism. Then, the category of different layout elements is identified through a sequence labeling task, and the candidate region coordinates of each element are output.

3. The document content restoration method based on artificial intelligence according to claim 2, characterized in that, In step one, locating various layout elements and obtaining physical layout parameters includes: Set the coordinate system of the document image, with the top left corner of the document image as the origin. , represent The axis is in the horizontal right direction. represent The axis is in the vertical downward direction. Represents the width of the image. Represents the height of the image; S11.1 Global layout parameter extraction and derivation; Let any text-type layout element be... , Represents the total number of layout elements. In the image coordinate system, the bounding box is determined by the coordinates of four vertices, which can be simplified to the coordinates of the top left corner and the bottom right corner. ； in, Represents text-based layout elements The coordinates of the top left corner, Represents text-based layout elements The coordinates of the upper right corner, , , , ; Deducing text layout elements width ,high Center coordinates ; ； ； ； S11.2, Define two elements and image / graphic layout elements Relative position: S11.2a, Text-based layout elements and image / graphic layout elements Horizontal distance, horizontal spacing ; like exist Left side: ； like exist Right side: ； S11.2b, Text-based layout elements and image / graphic layout elements Vertical distance, vertical spacing ; like exist Above: ； like exist Below: ； S11.2c, Text-based layout elements and image / graphic layout elements overlapping area With overlap rate Define the bounding box of the overlapping region as: ； ； ； in, Represents text-based layout elements area, Represents image and graphic layout elements The area; S11.3 Normalize the global layout parameters: ； in, , , , This represents the normalized layout parameters, which are directly used for cross-document layout logic comparison.

4. The document content restoration method based on artificial intelligence according to claim 3, characterized in that, In step one, the geometric constraint relationship is inferred using a geometric constraint algorithm, including: S12, Horizontal alignment constraint; ； in, The error threshold representing the level; S13, Vertical alignment constraint; ； in, The vertical error threshold; S14, Includes constraints; and and and ; S15, Adjacency Constraints: or ; S16, Hierarchical Constraints: and and ; S17, Multi-column constraints; ； in, The threshold for approximating the width is represented. The threshold is approximated by the central ordinate.

5. The document content restoration method based on artificial intelligence according to claim 4, characterized in that, In step one, generating a structured layout model includes determining the geometric constraints of each preprocessed document image, associating element attributes with constraint results, aggregating global layout rules, and finally integrating and encapsulating them into a structured layout model in a standardized data format.

6. The document content restoration method based on artificial intelligence according to claim 5, characterized in that, Step two includes: S2.1 Multilingual character recognition: The text positions are extracted from the structured layout model, and the text content is directly recognized and output by the multilingual OCR recognition tool on the extracted text region image. The language of the text is directly output by the language determination module built into the recognition tool, and the recognition confidence score is directly output by the recognition tool. At the same time, the hierarchical relationship of the text and the paragraph relationship are determined by geometric constraints, and the final output is structured text data. S2.

2. Image and Graphic Classification and Metadata Extraction: The positions of images and graphics are extracted from the structured layout model. The image classification tool directly outputs the type and classification confidence. Based on the adjacency constraint results in step one, the text element IDs and contents that are adjacent to the image and graphic elements are directly extracted. The color extraction tool directly analyzes the image and graphic regions and outputs the main color tone in the core information. Finally, the structured image and graphic data is output. S2.

3. Formula Format Conversion: Formula locations are extracted from the structured layout model. The formula recognition tool extracts features and parses the structure of the extracted formula region image, directly outputting standardized text formulas and recognition confidence scores, and finally outputting structured formula data. S2.

4. Table Structure Detection and Reconstruction: Extract the table area locations from the structured layout model in step one. The table detection tool directly outputs the row boundaries, column boundaries, and cell position coordinates of the table. Based on the row and column boundaries, calculate and restore the total number of rows and columns of the table. Combine the cell position and size to determine the cell merging situation. The OCR tool directly recognizes and outputs the internal text and numerical content of each cell area, and finally outputs the structured table data.

7. The document content restoration method based on artificial intelligence according to claim 6, characterized in that, Step three includes: S3.1 Semantic block partitioning and tree-like hierarchical construction: Extract text sequences from the structured text data, structured image data, structured formula data, and structured table data output in step two, and integrate them with the global layout parameters from the structured layout model in step one. , , , , , , The geometric constraint results are used to construct a multi-dimensional input feature vector. The constructed multi-dimensional feature vector is input into the BERT-CRF sequence labeling model. The semantic features of the text context are encoded by BERT and the sequence label dependencies are decoded by CRF. The semantic label of each text token is output. Independent semantic blocks are generated by aggregating the label sequences and assigned a unique ID. The relationship between semantic blocks is determined based on the geometric constraint rules in step one, forming structured semantic hierarchical data. S3.2 Extraction and restoration of text font and paragraph formatting styles: The visual region coordinates of text elements are extracted from the structured layout model in Step 1. The stroke outlines and character shape features of the extracted text region images are extracted using a font recognition tool. The font type, font size, whether it is bold, italic, underlined, and the recognition confidence are directly output. The RGB values ​​of the text region pixels are analyzed by a color extraction tool, and the main color tone of the text is directly output. The multi-column layout attributes are determined by combining the multi-column constraints in Step 1 and associated with the corresponding nodes of the structured semantic hierarchy data in S3.1 to form complete structured document data.

8. The document content restoration method based on artificial intelligence according to claim 1, characterized in that, Step four includes: S4.1 Structured Data Aggregation and Prompt Formatting: Extract core data from the complete structured document data output in step three, including the global layout parameters from step one, the metadata of various types of structured data from step two, and the complete structured document data from step three. At the same time, read the original metadata of the document file, classify and encode the above aggregated data, organize it using key-value pairs and hierarchical indentation format, construct structured prompt words in combination with task instructions, and finally output a standardized prompt word dataset. S4.2 Configuring the Large Language Model and Setting Up the Inference Environment: Based on the core task requirements of step four, select an appropriate pre-trained generative large language model, configure the core parameters of model inference, build the model inference environment, complete the input adaptation processing of the prompt word dataset, construct the entire process execution link from input to inference to output, and form the basic configuration for model inference. S4.3, Large Language Model-Driven Semantic Optimization and Association Completion: The standardized prompt word dataset of S4.1 is input into the configured large language model. Combining the model's semantic understanding and knowledge reasoning capabilities, a multi-dimensional optimization and completion task is performed, including: Perform semantic proofreading on OCR-recognized text, correcting typos and omissions, optimizing sentence fluency, and re-judging low-confidence content; Based on content relevance and layout constraints, complete the missing relationships between formulas and body text, tables and titles, and images and text descriptions; Connecting paragraphs across pages with semantic blocks, Optimize the semantic hierarchy; label the matching basis and confidence level of each type of optimized completion result, and finally output structured semantic optimization and related completion data; S4.4 Information Integration, Verification, and Standardized Output: By integrating the aggregated structured data of S4.1 with the semantically optimized and associated data of S4.3, verifying the consistency of the association logic, associating the optimized and associated information with the corresponding nodes of the basic structured data, encapsulating it into a standardized format according to application requirements, generating data version numbers and check codes to ensure integrity, and finally outputting complete and perfect document data to form a standardized document information system.

9. An artificial intelligence-based document content restoration system, applied to the document content restoration method according to any one of claims 1 to 8, characterized in that, It includes an image information acquisition module, a layout parsing module, a multimodal recognition module, a semantic reconstruction module, and an information restoration module.

10. A document content restoration system based on artificial intelligence according to claim 9, characterized in that: The image information acquisition module uses a camera to acquire document image information as the input source; The layout parsing module is used to locate various layout elements and obtain physical layout parameters, and then infer geometric constraint relationships through geometric constraint algorithms to generate a structured layout model. The multimodal recognition module performs multilingual text recognition, image and graphic classification and metadata extraction, formula format conversion, table structure detection and reconstruction, and outputs corresponding structured data. The semantic reconstruction module combines text features to distinguish different semantic blocks and constructs a tree-like hierarchical structure, while extracting and restoring the text font and paragraph formatting styles. The information restoration module reads the basic metadata of the document and restores and completes the document element association information based on the large model, thus completing the information improvement work.

Images