AI intelligent document processing method based on large language model

Abstract

The invention relates to an AI intelligent document processing method based on a large language model, and the method comprises the steps: receiving the input of an original document, and obtaining the text features, visual layout features and semantic structure features of the document through a multi-modal feature extraction module; processing the multi-modal features by using a layout-aware attention mechanism, and generating an incidence matrix of document elements; inputting the extracted features into a pre-trained large language model, and generating an intermediate representation containing an association relationship between document elements; and dynamically selecting a processing path according to task requirements, wherein the processing path comprises but is not limited to document classification, information extraction, content generation or question and answer response. According to the AI intelligent document processing method based on the large language model, through an attention mechanism of layout perception and an experiment on a standard data set, table structure recognition F1-score is improved, the cross-page element association accuracy is greatly improved, the vision-text feature alignment loss is reduced, the processing time of a 100-page PDF document is greatly shortened through a dynamic partitioning processing strategy, and the processing efficiency of the 100-page PDF document is greatly improved. And the memory occupancy peak value of the GPU is reduced.

Description

Technical Field

[0001] The present invention relates to the field of document processing technology, and specifically to an AI intelligent document processing method based on a large language model. Background Art

[0002] Intelligent document processing technology is an integrated application of multiple advanced technologies such as artificial intelligence and machine learning. It aims to automatically parse, understand and process information in unstructured and semi-structured documents, thereby converting this information into structured data. Error correction through intelligent document processing can automatically detect and correct spelling errors, grammatical errors, punctuation errors, and irregular wording in documents, thereby improving the accuracy of document content. Polishing through intelligent document processing can fine-tune documents, including improving sentence structure, correcting grammatical errors, and optimizing expressions.

[0003] For example, Chinese patent 202410402965.7 proposes an AI intelligent document processing method based on a large language model, which annotates text data to obtain annotated documents; uses training documents and corresponding annotated documents to train artificial intelligence documents to obtain a large language model; uses verification documents and corresponding annotated documents to evaluate the large language model to obtain an evaluation coefficient. When the evaluation coefficient is not greater than the coefficient threshold, the large language model is corrected and optimized according to the evaluation coefficient and the coefficient threshold.

[0004] However, existing AI intelligent document processing methods still have the following defects:

[0005] 1. The traditional OCR+rule engine method cannot handle heterogeneous document layouts;

[0006] 2. When a single LLM processes long documents, the attention mechanism fails.

[0007] 3. Existing document classification methods need to be retrained when migrating across domains.

[0008] Therefore, an AI intelligent document processing method based on a large language model is proposed to solve the above problems. Summary of the Invention

[0009] In response to the shortcomings of the existing technology, the present invention provides an AI intelligent document processing method based on a large language model, which has the advantages of improving the accuracy of structured extraction, reducing memory consumption when processing long documents of more than 100 pages, and supporting zero-sample learning for cross-domain migration. It solves the problems that the traditional AI intelligent document processing method, the practical OCR+rule engine method, cannot process heterogeneous document layouts, the attention mechanism fails when a single LLM processes long documents, and the existing document classification method needs to be retrained when migrating across domains.

[0010] To achieve the above objectives, the present invention provides the following technical solution: an AI intelligent document processing method based on a large language model, comprising the following steps:

[0011] S1. Receive the original document input and obtain the document's text features, visual layout features, and semantic structure features through a multimodal feature extraction module;

[0012] S2. Use layout-aware attention mechanism to process multimodal features and generate the correlation matrix of document elements;

[0013] S3. Input the extracted features into the pre-trained large language model to generate an intermediate representation containing the associations between document elements;

[0014] S4. Dynamically select processing paths based on task requirements, including but not limited to: document classification, information extraction, content generation, or question-answering response;

[0015] S5. Output the structured processing results and update the domain knowledge base. The domain knowledge base is updated using a differential privacy mechanism:

[0016] Add noise that satisfies (ε,δ)-differential privacy to the newly added knowledge items:

[0017]

[0018] New knowledge is compressed into a lightweight model through knowledge distillation.

[0019] Furthermore, the multimodal feature extraction module specifically includes:

[0020] The text encoding submodule uses character-level and word-level dual embedding representations;

[0021] The visual feature submodule uses an improved Swin Transformer to extract document layout features, and its window division strategy automatically adjusts to the density of document elements;

[0022] The structural analysis submodule constructs the topological relationship between document elements through graph neural network.

[0023] Furthermore, the visual feature submodule performs the following operations:

[0024] Calculate the relative position encoding matrix of document elements:

[0025]

[0026] Generate visual association weights between document elements, which are calculated as follows:

[0027]

[0028] The improved Swin Transformer includes:

[0029] Dynamic window division module, adjusts the window size according to the number of elements nn in the document area;

[0030] Cross-window information interaction layer, using learnable shift parameters Δx, Δy.

[0031] Furthermore, the layout-aware attention mechanism includes:

[0032] Calculate the geometric relationship weights between elements:

[0033] a geo =MLP([x i -x j ,y i -y j ,w i ,h j ])

[0034] Generate an attention score that fuses geometry and semantics:

[0035]

[0036] Furthermore, the dynamic selection of the processing path includes:

[0037] The code for real-time monitoring of document complexity indicators is as follows:

[0038] defcompute_complexity(doc):

[0039] returnλ1*entropy(text)+λ2*layout_variance+λ3*relation_density

[0040] When the complexity exceeds the threshold T, the distributed processing mode is started and the document blocks are assigned to multiple LLM instances for collaborative processing.

[0041] Furthermore, the updated domain knowledge base adopts a differential privacy mechanism:

[0042] Add noise that satisfies (ε,δ)-differential privacy to the newly added knowledge items:

[0043]

[0044] New knowledge is compressed into a lightweight model through knowledge distillation.

[0045] Furthermore, it also includes an intelligent document processing system for implementing the method, and the intelligent document processing system is specifically as follows:

[0046] a document pre-processing unit configured to perform document format conversion and standardization;

[0047] Front-end interactive module, supporting document drag-and-drop upload and real-time preview;

[0048] A multimodal understanding engine, consisting of a parallel visual encoder and a text encoder;

[0049] Distributed computing engine, including a master node and multiple worker nodes;

[0050] Dynamic routing controller that allocates computing resources based on real-time load conditions;

[0051] The knowledge management module enables the storage and retrieval of multi-version knowledge graphs.

[0052] Furthermore, the front-end interaction module includes:

[0053] Visual annotation tool, supports correction of LLM output results;

[0054] The feedback loop adds user correction data to the model fine-tuning training set;

[0055] The distributed computing engine includes:

[0056] Dynamic load balancer: Using an improved consistent hashing algorithm:

[0057] h(u)=(a·u+b)modp

[0058] Where a and b are random prime numbers, and p is the number of nodes;

[0059] Fault-tolerant recovery module: uses a checkpoint mechanism to save intermediate states every t seconds

[0060] Furthermore, the dynamic routing controller includes:

[0061] Resource monitoring submodule: real-time tracking of GPU memory utilization and computing latency;

[0062] Load balancing algorithm: An improved Consistent Hashing method is used to achieve dynamic addition and deletion of processing nodes.

[0063] Compared with the existing technology, the technical solution of this application has the following beneficial effects:

[0064] 1. This AI intelligent document processing method based on a large language model uses a layout-aware attention mechanism. Experiments on standard datasets show that the F1-score of table structure recognition is improved, the accuracy of cross-page element association is greatly improved, and the loss of visual-text feature alignment is reduced. The dynamic block processing strategy significantly shortens the processing time of a 100-page PDF document and reduces the peak GPU memory usage.

[0065] 2. This AI intelligent document processing method based on a large language model uses an improved Swin Transformer window partitioning strategy to reduce the layout analysis error rate in medical report processing. While differential privacy knowledge updates meet GDPR requirements, the accuracy rate only decreases after model fine-tuning. It automatically constructs a domain knowledge graph through the document processing process and uses differential privacy fine-tuning to achieve continuous model learning. In the FinDoc dataset test, the accuracy of structured extraction increased by 27.6%, and memory consumption was reduced by 58% when processing long documents of more than 100 pages. It supports zero-shot learning capabilities for cross-domain migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a flow chart of the AI ​​intelligent document processing method based on a large language model of the present invention;

[0067] Figure 2 This is a framework diagram of the intelligent document processing system of the present invention. DETAILED DESCRIPTION

[0068] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0069] See also Figure 1-2 The AI ​​intelligent document processing method based on the large language model in this embodiment includes the following steps:

[0070] S1. Receive the original document input and obtain the document's text features, visual layout features, and semantic structure features through a multimodal feature extraction module. The multimodal feature extraction module specifically includes:

[0071] The text encoding submodule uses character-level and word-level dual embedding representations;

[0072] The visual feature submodule uses an improved Swin Transformer to extract document layout features, and its window division strategy automatically adjusts to the density of document elements;

[0073] The structural analysis submodule constructs the topological relationship between document elements through graph neural networks;

[0074] S2. Use the layout-aware attention mechanism to process multimodal features and generate a correlation matrix of document elements. The layout-aware attention mechanism includes:

[0075] Calculate the geometric relationship weights between elements:

[0076] a geo =MLP([x i -x j ,y i -y j ,w i ,h j ])

[0077] Generate an attention score that fuses geometry and semantics:

[0078]

[0079] S3. Input the extracted features into the pre-trained large language model to generate an intermediate representation containing the associations between document elements;

[0080] S4. Dynamically select processing paths based on task requirements, including but not limited to: document classification, information extraction, content generation, or question-answering response;

[0081] S5. Output the structured processing results and update the domain knowledge base.

[0082] Specifically, the visual feature submodule performs the following operations:

[0083] Calculate the relative position encoding matrix of document elements:

[0084]

[0085] Generate visual association weights between document elements, which are calculated as follows:

[0086]

[0087] The improved Swin Transformer includes:

[0088] Dynamic window division module, adjusts the window size according to the number of elements nn in the document area;

[0089] Cross-window information interaction layer, using learnable shift parameters Δx, Δy.

[0090] In this embodiment, the dynamic selection of a processing path includes:

[0091] The code for real-time monitoring of document complexity indicators is as follows:

[0092] defcompute_complexity(doc):

[0093] returnλ1*entropy(text)+λ2*layout_variance+λ3*relation_density

[0094] When the complexity exceeds the threshold T, the distributed processing mode is started and the document blocks are assigned to multiple LLM instances for collaborative processing.

[0095] Specifically, the association matrix of the document elements includes: constructing a topological graph G = (V, E) of the document elements, where the vertex set V = {v1, ..., vn} represents the document elements, and the edge set E includes:

[0096] Explicit connection edges, such as lines between table cells;

[0097] Implicit semantic edges, cross-paragraph associations calculated by LLM;

[0098] Calculate the incidence matrix A∈R n×n , whose element values ​​are jointly determined by the following multimodal features:

[0099]

[0100] in

[0101] h i ,h j is the text feature vector;

[0102] φ(pi j ) is the spatial position encoding function;

[0103] is the visual similarity feature.

[0104] Specifically, it is characterized in that the spatial position encoding function adopts:

[0105] Fourier transform of relative position offset (Δx, Δy, Δw, Δh):

[0106]

[0107] The fundamental frequency ω k Distributed in geometric progression:

[0108] Visual similarity feature calculation includes:

[0109] Extract element region image features f through pre-trained ResNet-50 i , f j ;

[0110] Calculate the weighted combination of cosine similarity and IoU:

[0111]

[0112] Among them B i The coordinates of the element's bounding box.

[0113] In this embodiment, an intelligent document processing system for implementing the method is also included. The intelligent document processing system is specifically as follows:

[0114] a document pre-processing unit configured to perform document format conversion and standardization;

[0115] Front-end interactive module, supporting document drag-and-drop upload and real-time preview;

[0116] A multimodal understanding engine, consisting of a parallel visual encoder and a text encoder;

[0117] Distributed computing engine, including a master node and multiple worker nodes;

[0118] Dynamic routing controller that allocates computing resources based on real-time load conditions;

[0119] The knowledge management module enables the storage and retrieval of multi-version knowledge graphs.

[0120] Specifically, the front-end interaction module includes:

[0121] Visual annotation tool, supports correction of LLM output results;

[0122] The feedback loop adds user correction data to the model fine-tuning training set;

[0123] The distributed computing engine includes:

[0124] Dynamic load balancer: Using an improved consistent hashing algorithm:

[0125] h(u)=(a·u+b)modp

[0126] Where a and b are random prime numbers, and p is the number of nodes;

[0127] Fault-tolerant recovery module: uses a checkpoint mechanism to save intermediate states every t seconds

[0128] Specifically, the dynamic routing controller includes:

[0129] Resource monitoring submodule: real-time tracking of GPU memory utilization and computing latency;

[0130] Load balancing algorithm: An improved Consistent Hashing method is used to achieve dynamic addition and deletion of processing nodes.

[0131] Example 2: Intelligent Analysis of Financial Contracts

[0132] Application Scenario

[0133] A bank's automated credit contract processing system handles PDF contracts containing heterogeneous elements such as forms, clauses, signatures, etc.

[0134] Technical Implementation

[0135] 1. Multimodal feature extraction

[0136] Using a hierarchical BERT model, character-level embedding captures special symbols, paragraph-level embedding identifies clause relevance (F1-score = 97.3%), and the Swin Transformer window is dynamically adjusted, with an 8×8 window used in the dense table area and a 16×16 window used in the text area.

[0137] 2. Generate correlation matrix

[0138] Establish an association between the "Breach of Contract" clause and the signature area:

[0139] A_{ij}=0.82#text semantic similarity 0.7+spatial position weight 0.12(Δy<200px)

[0140] Detect the correspondence between the table and the description text. When IoU>0.6, the association weight is +0.15.

[0141] 3. Dynamic processing

[0142] Complexity assessment:

[0143] C=0.6*H_t(1.2)+0.3*V_l(0.8)+0.1*R_g(1.5)=1.11>T(1.0)

[0144] Trigger distributed processing and distribute the contract to three computing nodes for parallel processing.

[0145] The implementation results are shown in the following table:

[0146] index This program Traditional OCR+NLP promote Accuracy of key terms extraction 98.5％ 72.1％ +36.4％ Cross-page association recognition rate 93.2％ 61.8％ +50.8％ Processing speed (pages / minute) 45 12 +275％

[0147] Example 2: Structural processing of medical reports

[0148] Application Scenario

[0149] CT examination reports from tertiary hospitals are automatically structured, including diagnostic conclusions, image descriptions, test data, etc.

[0150] Technical Implementation

[0151] 1. Visual-Text Alignment

[0152] Visual similarity calculation using claim 18:

[0153] The feature cosine similarity between the image description paragraph and the corresponding image area is greater than 0.85;

[0154] Correct the description of directional words such as "left lung" through relative position coding;

[0155] 2. Incremental update of knowledge base

[0156] New medical term recognition: When the LLM output probability P("ground-glass opacity of lung") = 0.91 > θ_h(0.9), it is automatically stored and Laplace perturbation with noise ε = 0.3 is added;

[0157] Conflict detection: The expert voting mechanism is triggered when the new and old diagnostic criteria conflict.

[0158] 3. Effect verification

[0159] Diagnostic keyword extraction accuracy: 96.7%;

[0160] New term discovery cycle shortened from 14 days to 2 hours;

[0161] The accuracy of misdiagnosis risk warning increased by 42%.

[0162] In summary, this AI-powered document processing method based on a large language model, through a layout-aware attention mechanism, demonstrates improved F1-score performance in table structure recognition, significantly increased cross-page element association accuracy, and reduced visual-text feature alignment loss in standard datasets. A dynamic block processing strategy significantly shortens the processing time for 100-page PDF documents and reduces peak GPU memory usage. An improved Swin Transformer window partitioning strategy reduces layout analysis error rates in medical report processing, while differential privacy knowledge updates meet GDPR requirements, with only a decrease in accuracy after model fine-tuning. The document processing process automatically constructs a domain knowledge graph, and differential privacy fine-tuning enables continuous model learning. In FinDoc dataset testing, the accuracy of structured extraction increased by 27.6%, memory consumption decreased by 58% when processing documents over 100 pages, and zero-shot learning capabilities were supported for cross-domain transfer.

[0163] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0164] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. An AI intelligent document processing method based on a large language model, characterized in that: The following steps are involved: S1. Receive the original document input and obtain the document's text features, visual layout features, and semantic structure features through a multimodal feature extraction module; S2. Use layout-aware attention mechanism to process multimodal features and generate the correlation matrix of document elements; S3. Input the extracted features into the pre-trained large language model to generate an intermediate representation containing the associations between document elements; S4. Dynamically select processing paths based on task requirements, including but not limited to: document classification, information extraction, content generation, or question-answering response; S5. Output the structured processing results and update the domain knowledge base.

2. The AI ​​intelligent document processing method based on a large language model according to claim 1 is characterized in that: The multimodal feature extraction module specifically includes: The text encoding submodule uses character-level and word-level dual embedding representations; The visual feature submodule uses an improved Swin Transformer to extract document layout features, and its window division strategy automatically adjusts to the density of document elements; The structural analysis submodule constructs the topological relationship between document elements through graph neural network.

3. The AI ​​intelligent document processing method based on a large language model according to claim 2 is characterized in that: The visual feature submodule performs the following operations: Calculate the relative position encoding matrix of document elements: Generate visual association weights between document elements, which are calculated as follows: The improved Swin Transformer includes: Dynamic window division module, adjusts the window size according to the number of elements nn in the document area; Cross-window information interaction layer, using learnable shift parameters Δx, Δy.

4. The AI ​​intelligent document processing method based on a large language model according to claim 1 is characterized in that: The layout-aware attention mechanism includes: Calculate the geometric relationship weights between elements: to geo =MLP([x i -x j ,and i -and j ,w i ,h j ]) Generate an attention score that fuses geometry and semantics:

5. The AI ​​intelligent document processing method based on a large language model according to claim 2 is characterized in that: The dynamic selection processing path includes: The code for real-time monitoring of document complexity indicators is as follows: defcompute_complexity(doc): returnλ1*entropy(text)+λ2*layout_variance+λ3*relation_density When the complexity exceeds the threshold T, the distributed processing mode is started and the document blocks are assigned to multiple LLM instances for collaborative processing.

6. The AI ​​intelligent document processing method based on a large language model according to claim 1 is characterized in that: The updated domain knowledge base adopts a differential privacy mechanism: Add noise that satisfies (ε,δ)-differential privacy to the newly added knowledge items: New knowledge is compressed into a lightweight model through knowledge distillation.

7. The AI ​​intelligent document processing method based on a large language model according to claim 1 is characterized in that: The invention also includes an intelligent document processing system for implementing the method according to any one of claims 1 to 6, wherein the intelligent document processing system is specifically as follows: a document pre-processing unit configured to perform document format conversion and standardization; Front-end interactive module, supporting document drag-and-drop upload and real-time preview; A multimodal understanding engine, consisting of a parallel visual encoder and a text encoder; Distributed computing engine, including a master node and multiple worker nodes; Dynamic routing controller that allocates computing resources based on real-time load conditions; The knowledge management module enables the storage and retrieval of multi-version knowledge graphs.

8. The AI ​​intelligent document processing method based on a large language model according to claim 7 is characterized in that: The front-end interaction module includes: Visual annotation tool, supports correction of LLM output results; The feedback loop adds user correction data to the model fine-tuning training set; The distributed computing engine includes: Dynamic load balancer: Using an improved consistent hashing algorithm: h(u)=(a·u+b)modp Where a and b are random prime numbers, and p is the number of nodes; Fault-tolerant recovery module: uses a checkpoint mechanism to save intermediate states every t seconds.

9. The AI ​​intelligent document processing method based on a large language model according to claim 7, characterized in that: The dynamic routing controller includes: Resource monitoring submodule: real-time tracking of GPU memory utilization and computing latency; Load balancing algorithm: An improved Consistent Hashing method is used to achieve dynamic addition and deletion of processing nodes.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images