Multi-modal file intelligent approval method and system based on large language model

Abstract

The invention relates to the technical field of OA examination and approval, in particular to a multi-modal file intelligent examination and approval method and system based on a large language model, and the method comprises the steps: configuring exclusive examination and approval templates for different examination and approval roles, and generating a role exclusive intelligent examination model; the method comprises the following steps: receiving a to-be-approved multi-modal file, analyzing file content, and outputting structured JSON (JavaScript Object Notation) data; inputting the structured JSON data and a configured approval template into a corresponding intelligent review model, performing step-by-step reasoning through a thinking chain prompt strategy, and outputting a structured review intermediate result; encoding the review intermediate result into a semantic vector, and retrieving Top-K similar historical approval cases from a knowledge base module based on a vector database to generate a reference suggestion set; fusing the review intermediate result with the reference suggestion set to generate a review result; based on the review result and the feedback of the approver, the standardized approval opinions are generated, and the accuracy and consistency of review are improved.

Description

Technical Field

[0001] This application relates to the field of OA approval technology, specifically to a multimodal intelligent document approval method and system based on a large language model. Background Technology

[0002] In existing office automation (OA) systems, the document approval process primarily relies on manual review and verification of each document item. This approach has several significant drawbacks: First, the approval process requires reviewing the entire document, making it difficult for approvers to quickly identify key issues or risks. This results in a longer average processing time for a single document, which is not suitable for the needs of efficient operations in modern enterprises.

[0003] Secondly, due to differences in the professional background, experience level, and focus of different approvers, the approval conclusions for the same document are often highly subjective, which can easily lead to inconsistent approval standards and affect the fairness and reliability of decision-making.

[0004] Third, existing OA systems often struggle to effectively parse and process scanned documents and image-based documents. Their ability to identify and extract complex elements such as tables, signature areas, and handwritten annotations within these documents is particularly limited, resulting in a large amount of valuable information failing to be automatically captured and utilized.

[0005] Finally, the large amount of historical approval data accumulated in the system is usually stored in a scattered manner, lacking a unified structured organization and intelligent retrieval mechanism. This makes it difficult to transform the experience and knowledge gained in the approval process into reusable decision-making resources. Similar problems repeatedly rely on manual judgment, which not only increases the workload but also affects the stability and efficiency of approval quality.

[0006] Therefore, there is an urgent need for a new method that can automate and intelligently process the document approval process, unify review standards, enhance multimodal data analysis capabilities, and effectively utilize historical approval knowledge to overcome the aforementioned shortcomings of existing technologies. Summary of the Invention

[0007] To address the aforementioned issues, this invention provides a multimodal document intelligent approval method and system based on a large language model.

[0008] In a first aspect, the technical solution of the present invention provides a multimodal document intelligent approval method based on a large language model, comprising the following steps: S1. Configure exclusive approval templates for different approval roles. Based on the historical approval data of each approval role, use LoRA parameter fine-tuning technology to perform personalized training on the pre-trained large language model and generate an intelligent review model exclusive to each role. S2. Receive a multimodal file to be approved. The multimodal file includes at least one of PDF documents, Word text files, and image files. Distribute the file to the corresponding parsing channel according to the file type. Parse the file content using OCR recognition technology and layout analysis algorithm. Output structured JSON data containing text content, table structure, signature area, and handwritten content. S3. Input the structured JSON data and the configured approval template into the corresponding intelligent review model, perform step-by-step reasoning through the thinking chain prompt strategy, and output a structured review intermediate result containing question type, text fragment, location information and confidence level. S4. Encode the intermediate review results into a semantic vector; based on the semantic vector, retrieve the Top-K similar historical approval cases from the knowledge base module based on the vector database; and generate a set of reference suggestions based on the retrieved cases; S5. The intermediate review results are merged with the reference suggestion set to generate the review results. The review results are then visualized through the human-computer interaction module, including highlighting the problem items in the original text and receiving confirmation or modification instructions from the approver. S6. Based on the review results and the approver's feedback, generate standardized approval opinions.

[0009] As a further limitation of the technical solution of the present invention, in S1, the step of configuring exclusive approval templates for different approval roles includes: Create and define different approval roles, and assign a unique role identifier to each role; Based on the JSON Schema rule description language, structured review rules are defined for various approval documents. These rules include field type constraints, logical condition judgments, and business thresholds. Associate the defined review rules with the corresponding approval roles, and generate an approval template specific to that role through the web configuration interface; The approval template is serialized into a structured Prompt template, which serves as the fixed context input for the intelligent review model specific to the corresponding role.

[0010] By creating and defining different approval roles and assigning a unique role identifier to each role, personalized configuration of approval templates can be achieved to meet the approval needs of different roles. Approval rules are defined based on the JSON Schema rule description language, including field type constraints, logical condition judgments, and business thresholds, making the approval rules more flexible and precise. Approval templates are generated through a web configuration interface and serialized into structured Prompt templates, facilitating management and updates and improving the maintainability of approval templates.

[0011] As a further limitation of the technical solution of the present invention, in S1, the step of using LoRA parameter efficient fine-tuning technology to perform personalized training on the pre-trained large language model includes: Collect historical approval logs of the target approval role. The logs include the content of the approval documents, marked issues, and the final approval comments. Construct an instruction fine-tuning dataset based on the logs. Based on the pre-trained large language model, a new LoRA adapter module is added. The LoRA adapter is injected into the self-attention layer of the original model through a low-rank decomposition matrix. The instructions are used to fine-tune the dataset, and the parameters of the LoRA adapter module are trained in a distributed manner on the GPU cluster. During the training process, mixed precision computation and gradient checkpointing techniques are used to optimize the use of video memory. Save the trained LoRA adapter weights and combine them with the pre-trained model base to generate a character-specific intelligent censorship model.

[0012] A fine-tuning dataset of instructions is constructed by collecting historical approval logs from the target approval roles. A LoRA adapter module is added to the pre-trained large language model, enabling precise personalized training that better adapts the model to the specific role's approval tasks. The parameters of the LoRA adapter module are trained in a distributed manner on a GPU cluster, employing mixed-precision computation and gradient checkpointing techniques to optimize memory usage, improve training efficiency, and shorten training time. The trained LoRA adapter weights are saved and combined with the pre-trained model base to generate a role-specific intelligent review model, facilitating model updates and deployment and ensuring continuous optimization of the approval system.

[0013] As a further limitation of the technical solution of the present invention, step S2 includes: S21. Receive uploaded documents pending approval and automatically identify their file types; S22. Based on the identified file type, the file is distributed to the corresponding parsing channel, wherein the parsing channel includes: Plain text parsing channel: used for processing Word text files; Layout document parsing channel: used for processing PDF documents; Image parsing channel: used for processing JPG and PNG image files; If the file is distributed to the plain text parsing channel, the text and metadata in the file are extracted directly. If the file is distributed to the document parsing channel or image parsing channel, the OCR recognition algorithm is called to extract the text content, and the layout analysis model is called to recognize the table structure, signature area and handwritten content in the document. S23. Convert the parsed text content, table structure, signature area information, and handwritten content into a structured JSON data format.

[0014] The system automatically identifies the type of documents awaiting approval and distributes them to the corresponding parsing channel based on the document type. It then uses OCR recognition algorithms and layout analysis models to parse the document content, achieving automation and efficiency in document parsing. It supports the parsing of various document types, including plain text, formatted documents, and images, extracting text content, layout structure, and semantic units, and converting them into structured JSON data format. This provides rich input data for subsequent intelligent review. The parsed text, layout structure, and semantic units are uniformly converted into structured JSON data format, facilitating processing and analysis by subsequent modules and improving data utilization and sharing.

[0015] As a further limitation of the technical solution of the present invention, step S3 includes: S31. Integrate the rules defined in the approval template with the document content in the structured JSON data to construct a mind chain prompt containing review context and step-by-step reasoning instructions; S32. Input the prompt words into the role-specific intelligent review model, and guide the model to perform the following reasoning operations step by step according to the preset reasoning chain: Identify and extract all key information fields related to the approval rules from the document content; The extracted key fields are compared one by one with the constraints, logical rules and business thresholds defined in the approval template. Based on the comparison results, determine whether each key field or combination of fields conforms to the rules and generate the corresponding confidence score; For items judged to be inconsistent with the rules or with a confidence level below the confidence threshold, generate natural language explanations; S33. Map and encapsulate the inference results of the model into a structured JSON format review intermediate result, wherein the review intermediate result includes at least the question type, the associated text fragment, the location information of the fragment in the original text, and the confidence level of the judgment.

[0016] By integrating the rules defined in the approval template with the document content in structured JSON data, a thought chain prompt is constructed to guide the model to perform reasoning operations step by step according to a preset reasoning chain. This enables accurate identification and judgment of problems in the document content, improving the accuracy and reliability of the review. Structured intermediate review results containing problem type, text fragments, location information, and confidence levels are generated, providing approvers with detailed review information to quickly understand the problems in the document. Guiding the model step-by-step reasoning through the thought chain prompt strategy improves the model's reasoning ability and logical consistency, enabling the model to better understand and process complex approval rules and document content.

[0017] As a further limitation of the technical solution of the present invention, step S4 specifically includes: S41. Input the text fragments and question types from the intermediate review results into the text encoder to generate semantic vectors; S42. Using the semantic vector as a query vector, perform an approximate nearest neighbor search algorithm in the vector database to calculate its similarity to the historical approval case vectors in the knowledge base, and return the Top-K most similar historical cases. S43. Extract the corresponding approval opinions and decision results from the top-K historical cases retrieved, and form a set of reference suggestions.

[0018] Encoding intermediate review results into semantic vectors and retrieving Top-K similar historical approval cases from a vector database-based knowledge base module to generate a reference suggestion set provides relevant reference opinions for current approval tasks, improving the scientific nature and accuracy of approval decisions. Extracting approval opinions and decision results from retrieved historical cases to form a reference suggestion set provides approvers with rich reference information, helping them better understand and handle current approval tasks. Through knowledge base retrieval and the generation of the reference suggestion set, consistency in approval decisions can be ensured, reducing inconsistencies in approval results caused by subjective differences among approvers.

[0019] As a further limitation of the technical solution of the present invention, step S5 includes: S51. Integrate the intermediate review results with the reference suggestion set to generate the review results; S52. Based on the location information in the review results, highlight the original copy of the multimodal document to be approved, and use different colors or icons to distinguish different problem types. S53. Display the highlighted document copy and corresponding review result details to the approver through a human-computer interaction interface; S54. Receive instructions from the approver regarding the review results, including confirmation of the issue item, ignoring it, modifying the approval opinion, or manually adding a new issue item.

[0020] The system integrates intermediate review results with a set of reference suggestions to generate review outcomes, which are then visualized and presented to approvers through a human-computer interaction interface. This includes highlighting issue items in the original text, allowing approvers to intuitively understand the problems in the documents and improving approval efficiency. The system receives instructions from approvers regarding the review results, including confirming, ignoring, modifying approval comments, or manually adding new issue items, providing approvers with flexible approval operation methods to meet the needs of different approval scenarios. The human-computer interaction module enables efficient collaboration between approvers and the system, fully leveraging the advantages of artificial intelligence and human review to improve approval quality and decision-making efficiency.

[0021] As a further limitation of the technical solution of the present invention, the specific steps of S51 include: S511. Combine the intermediate results of the structured review with the set of reference suggestions to form a comprehensive input context; S512. Submit the comprehensive input context to the role-specific intelligent review model and provide an instruction prompt, requesting the model to conduct a final review and enhancement of the review results; S513. Based on the historical approval opinions in the reference suggestion set, the confidence levels of each issue in the current review intermediate results are weighted and corrected to generate the corrected confidence levels; S514. Extract relevant background information and decision-making basis from the reference recommendations and add them to the explanation of the current review intermediate results; S515. Output an enhanced final review result with confidence level correction and supplementary background information.

[0022] By concatenating structured review intermediate results with a set of reference suggestions to form a comprehensive input context, and submitting it to a role-specific intelligent review model for final review and enhancement, the review results can be further optimized, improving their accuracy and reliability. Based on historical approval opinions in the reference suggestion set, the confidence levels of various issues in the current review intermediate results are weighted and corrected to generate revised confidence levels that more accurately reflect the severity and credibility of the issues, providing approvers with more valuable reference information. Relevant background information and decision-making basis are extracted from the reference suggestion set and added to the explanations of the current review intermediate results, providing approvers with rich background information to help them better understand and handle the current approval task.

[0023] As a further limitation of the technical solution of the present invention, step S6 includes: S61. Extract key information from the review results, including the question type, text fragments, location information, and confidence level; S62. Adjust and optimize the review results based on the approver's confirmation or modification instructions; S63. Retrieve standard approval opinion templates related to the review results from the knowledge base module; S64. Combine the adjusted review results with the standard approval opinion template, and use a large language model to generate standardized approval opinions; S65. Format the generated approval comments.

[0024] The system extracts key information from the review results and, combined with the approver's confirmation or modification instructions, retrieves relevant standard approval opinion templates from the knowledge base module. It then uses a large language model to generate standardized approval opinions, ensuring their standardization and consistency. The adjusted review results are combined with the standard approval opinion templates, and the large language model is used to generate further approval opinions. These opinions are then formatted, supporting multiple output formats and enabling the rapid generation of approval opinions that conform to the company's language style, thus improving approval efficiency. By generating standardized approval opinions, the system provides approvers with clear decision-making basis, reducing approval errors caused by unclear or non-standard opinions and improving the quality and credibility of approval decisions.

[0025] Secondly, the present invention also provides a multimodal document intelligent approval system based on a large language model, used to implement the method described in the first aspect, the system comprising: The approval configuration module is used to configure exclusive approval templates for different approval roles. Based on the historical approval data of each approval role, the LoRA parameter high-efficiency fine-tuning technology is used to perform personalized training on the pre-trained large language model to generate an intelligent review model exclusive to each role. The file parsing module is used to receive multimodal files to be approved, distribute them to the corresponding parsing channels according to the file type, parse the file content through OCR recognition technology and layout analysis algorithm, and output structured JSON data. The intelligent review module is used to input the structured JSON data and the approval template into the corresponding intelligent review model, perform step-by-step reasoning through the thinking chain prompt strategy, and output the intermediate results of the structured review. The knowledge base module is used to encode intermediate review results into semantic vectors, and based on these semantic vectors, retrieve similar historical approval cases from the vector database and generate a set of reference suggestions. The human-computer interaction module is used to integrate intermediate review results with the set of reference suggestions to generate review results, and to visualize and receive instructions. The feedback generation module is used to generate standardized approval feedback based on the review results and the approver's feedback.

[0026] As can be seen from the above technical solutions, this application has the following advantages: By configuring exclusive approval templates for different approval roles and training personalized intelligent review models, key issues in documents can be quickly identified, reducing the time and effort required for item-by-item verification. Personalized training of the pre-trained large language model based on LoRA parameter fine-tuning technology enables the model to learn the review preferences of specific roles, improving the accuracy and consistency of the review. Combining OCR recognition algorithms and thought chain prompting strategies, it can handle multimodal documents of various formats, achieving automatic parsing and intelligent review of document content, improving the intelligence level of the entire approval process. Automatic extraction of non-compliant content and generation of highlighted prompts help approvers quickly locate problems and reduce erroneous approvals due to negligence. Attached Figure Description

[0027] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating the method provided in an embodiment of the present invention.

[0029] Figure 2 A block diagram of a system provided in an embodiment of the present invention. Detailed Implementation

[0030] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this application and in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0032] like Figure 1 As shown, this embodiment of the invention provides a multimodal document intelligent approval method based on a large language model, including the following steps: S1. Configure exclusive approval templates for different approval roles. Based on the historical approval data of each approval role, use LoRA parameter fine-tuning technology to perform personalized training on the pre-trained large language model and generate an intelligent review model exclusive to each role. The steps to configure exclusive approval templates for different approval roles include: Create and define different approval roles, and assign a unique role identifier to each role; Based on the JSON Schema rule description language, structured review rules are defined for various approval documents. These rules include field type constraints, logical condition judgments, and business thresholds. Associate the defined review rules with the corresponding approval roles, and generate an approval template specific to that role through the web configuration interface; The approval template is serialized into a structured Prompt template, which serves as the fixed context input for the intelligent review model specific to the corresponding role.

[0033] The steps for personalized training of a pre-trained large language model include: Collect historical approval logs from the target approval role. The logs include the content of the approval documents, marked issues, and the final approval comments. Construct an instruction fine-tuning dataset based on the logs. Specifically, this includes: de-identifying the log data, removing personal information and confidential content; marking data issue types, and creating triplet samples of document content-issue description-correction suggestions; performing data augmentation on the samples, including synonym replacement, sentence transformation, and negative sample generation, to obtain the fine-tuning dataset.

[0034] Based on the pre-trained large language model, a new LoRA adapter module is added. The LoRA adapter is injected into the self-attention layer of the original model through a low-rank decomposition matrix. The instructions are used to fine-tune the dataset, and the parameters of the LoRA adapter module are trained in a distributed manner on the GPU cluster. During the training process, mixed precision computation and gradient checkpointing techniques are used to optimize the use of video memory. The trained LoRA adapter weights are saved and combined with the pre-trained model base to generate a character-specific intelligent censorship model. The personalized training of the pre-trained large language model using LoRA parameter efficient fine-tuning technology includes: A pre-trained large language model was chosen as the base model, with a model size of 7B to 13B parameters. Configure LoRA parameters: rank r=8, scaling factor α=16, only adapting the Q / K / V matrix of the attention layer; Set the training hyperparameters as follows: learning rate 3e-4, batch size 32, training epochs 3; Mixed precision training (FP16) and gradient checkpointing techniques are used to optimize GPU memory usage; Distributed training is performed on a GPU cluster using a data parallel strategy.

[0035] The generated role-specific review model also includes: Evaluate model performance on the validation set, and calculate accuracy, recall, and F1 score; Set dynamic thresholds: Confidence level ≥ 0.7, directly accept; 0.5 ≤ confidence level < 0.7, manually review; < 0.5, reject. The trained LoRA adapter is combined with the base model and exported as a service-oriented model. Implement a hot-loading mechanism for the model to support updates without downtime; Establish a model version control system to retain historical versions for rollback purposes; Deploy a model monitoring dashboard to track metrics such as inference latency, memory usage, and anomaly rate.

[0036] S2. Receive a multimodal file to be approved, wherein the multimodal file includes at least one of PDF documents, Word text files, and image files. Distribute the file to the corresponding parsing channel according to the file type, parse the file content using OCR recognition technology and layout analysis algorithms, and output structured JSON data containing text content, table structure, signature area, and handwritten content; specifically including: S21. Receive uploaded documents pending approval and automatically identify their file types; S22. Based on the identified file type, the file is distributed to the corresponding parsing channel, wherein the parsing channel includes: Plain text parsing channel: used for processing Word text files; Layout document parsing channel: used for processing PDF documents; Image parsing channel: used for processing JPG and PNG image files; If the file is distributed to the plain text parsing channel, the text and metadata in the file are extracted directly. If the file is distributed to the document parsing channel or image parsing channel, the OCR recognition algorithm is called to extract the text content, and the layout analysis model is called to recognize the table structure, signature area and handwritten content in the document. S23. Convert the parsed text content, table structure, signature area information, and handwritten content into a structured JSON data format.

[0037] The layout analysis model is invoked to identify the physical and logical structure of the document, including: identifying paragraphs, headings, tables, stamp areas, handwritten areas, and signature areas in the document, and marking the coordinate position of each identified element in the original file.

[0038] For files containing tables, the OCR recognition algorithm prioritizes an OCR model with table reconstruction capabilities, organizes the extracted text content into operable table data according to the original row and column structure, and stores it in the JSON data.

[0039] Before generating the final JSON data, natural language processing is performed on the parsed text content, including sentence boundary detection and paragraph segmentation, to identify the semantic units of the text.

[0040] The document parsing channel is specifically for PDF files, while the image parsing channel is specifically for JPG and PNG files.

[0041] S3. Input the structured JSON data and the configured approval template into the corresponding intelligent review model, perform step-by-step reasoning through a thought chain prompt strategy, and output a structured review intermediate result containing question type, text fragment, location information, and confidence level; specifically including: S31. Integrate the rules defined in the approval template with the document content in the structured JSON data to construct a mind chain prompt containing review context and step-by-step reasoning instructions; S32. Input the prompt words into the role-specific intelligent review model, and guide the model to perform the following reasoning operations step by step according to the preset reasoning chain: Identify and extract all key information fields related to the approval rules from the document content; The extracted key fields are compared one by one with the constraints, logical rules and business thresholds defined in the approval template. Based on the comparison results, determine whether each key field or combination of fields conforms to the rules and generate the corresponding confidence score; For items judged to be inconsistent with the rules or with a confidence level below the confidence threshold, generate natural language explanations; S33. Map and encapsulate the inference results of the model into a structured JSON format review intermediate result, wherein the review intermediate result includes at least the question type, the associated text fragment, the location information of the fragment in the original text, and the confidence level of the judgment.

[0042] S4. Encode the intermediate review results into a semantic vector; based on the semantic vector, retrieve the Top-K similar historical approval cases from the knowledge base module based on the vector database; and generate a reference suggestion set based on the retrieved cases; specifically including: S41. Input the text fragments and question types from the generated structured review intermediate results into the text encoder to generate semantic vectors; S42. Using the semantic vector as a query vector, perform an approximate nearest neighbor search algorithm in the vector database to calculate its similarity to the historical approval case vectors in the knowledge base, and return the Top-K most similar historical cases. S43. Extract the corresponding approval opinions and decision results from the top-K historical cases retrieved, and form a set of reference suggestions.

[0043] The text encoder is a pre-trained Sentence-BERT model.

[0044] The approximate nearest neighbor search algorithm is either HNSW or IVF-PQ algorithm.

[0045] The methods for calculating similarity are cosine similarity or inner product similarity.

[0046] The generated set of reference suggestions is also organized in a structured JSON format, containing original text fragments of similar cases, historical approval opinions, and final approval results.

[0047] The knowledge base module automatically adds the vector representations of approved cases and approval results to the database by subscribing to approval completion events in real time, thus enabling incremental updates.

[0048] This application stores historical approval records in a vector database. During retrieval, cosine similarity is calculated to return the most similar case and its approval opinion. The data storage adopts a hierarchical architecture, and incremental learning is achieved by subscribing to approval completion events through Kafka. Specifically, the construction and retrieval process of the knowledge base module includes: The document content, review results, and approval opinions in the historical approval records are combined into text segments, and semantic vectors are generated through a pre-trained text encoding model and stored in a vector database. During retrieval, the text fragments and question types in the intermediate review results are input into the same text encoding model to generate query vectors; By calculating the cosine similarity between the query vector and the vectors in the database, the top-K most similar historical approval cases are returned.

[0049] The knowledge base module adopts a hierarchical storage architecture: frequently accessed case data is stored in an in-memory database to achieve high-speed retrieval; all case data is stored in a vector database and object storage to balance storage cost and access performance.

[0050] The incremental learning process implemented in the knowledge base module includes: Subscribe to approval completion events via the Kafka message queue; The events are consumed by an independent vectorization service, which encodes newly completed approval cases into semantic vectors in real time and stores them incrementally into the vector database. The dynamic index of the vector database supports real-time incremental updates.

[0051] S5. The intermediate review results are merged with the reference suggestion set to generate the review results. The review results are then visualized through the human-computer interaction module, including highlighting the problem items in the original text and receiving confirmation or modification instructions from the approver. S6. Based on the review results and the approver's feedback, generate standardized approval opinions.

[0052] In some embodiments, step S5 includes: S51. Integrate the intermediate review results with the reference suggestion set to generate the review results; S52. Based on the location information in the review results, highlight the original copy of the multimodal document to be approved, and use different colors or icons to distinguish different problem types. S53. Display the highlighted document copy and corresponding review result details to the approver through a human-computer interaction interface; S54. Receive instructions from the approver regarding the review results, including confirmation of the issue item, ignoring it, modifying the approval opinion, or manually adding a new issue item.

[0053] Furthermore, the specific steps of S51 include: S511. Combine the intermediate results of the structured review with the set of reference suggestions to form a comprehensive input context; S512. Submit the comprehensive input context to the role-specific intelligent review model and provide an instruction prompt, requesting the model to conduct a final review and enhancement of the review results; S513. Based on the historical approval opinions in the reference suggestion set, the confidence levels of each issue in the current review intermediate results are weighted and corrected to generate the corrected confidence levels; S514. Extract relevant background information and decision-making basis from the reference recommendations and add them to the explanation of the current review intermediate results; S515. Output an enhanced final review result with confidence level correction and supplementary background information.

[0054] It should be noted here that the instruction prompt in S512 is a thought chain prompt, guiding the model to perform the following sequential reasoning: 1. Understand the problem found in the current review; 2. Refer to the handling opinions of similar historical cases; 3. Determine whether the severity of the current problem is consistent with historical cases; 4. Decide whether to adjust the confidence level and how to supplement the decision basis.

[0055] The specific method for weighted assessment and correction of confidence level in S513 is as follows: calculate the weighted average of the confidence level of the current review result and the confidence level of the approval opinions of the Top-K similar cases, where the weight is determined by the case similarity.

[0056] The final review result output in S515 is a structured JSON object. This object adds the fields final_confidence and reference_basis to the fields of the intermediate review result.

[0057] In some embodiments, step S6 includes: S61. Extract key information from the review results, including the question type, text fragments, location information, and confidence level; S62. Adjust and optimize the review results based on the approver's confirmation or modification instructions; S63. Retrieve standard approval opinion templates related to the review results from the knowledge base module; S64. Combine the adjusted review results with the standard approval opinion template, and use a large language model to generate standardized approval opinions; S65. Format the generated approval comments.

[0058] like Figure 2 As shown in the figure, this embodiment of the invention also provides a multimodal document intelligent approval system based on a large language model, used to implement the methods described in the above embodiments. The system includes: The approval configuration module is used to configure exclusive approval templates for different approval roles. Based on the historical approval data of each approval role, the LoRA parameter high-efficiency fine-tuning technology is used to perform personalized training on the pre-trained large language model to generate an intelligent review model exclusive to each role. The file parsing module is used to receive multimodal files to be approved, distribute them to the corresponding parsing channels according to the file type, parse the file content through OCR recognition technology and layout analysis algorithm, and output structured JSON data. The intelligent review module is used to input the structured JSON data and the approval template into the corresponding intelligent review model, perform step-by-step reasoning through the thinking chain prompt strategy, and output the intermediate results of the structured review. The knowledge base module is used to encode intermediate review results into semantic vectors, and based on these semantic vectors, retrieve similar historical approval cases from the vector database and generate a set of reference suggestions. The human-computer interaction module is used to integrate intermediate review results with the set of reference suggestions to generate review results, and to visualize and receive instructions. The feedback generation module is used to generate standardized approval feedback based on the review results and the approver's feedback.

[0059] In some embodiments, the approval configuration module includes: The role management unit is used to create and define different approval roles and assign a unique role identifier to each role; The rule modeling unit is used to define structured review rules for various approval documents based on the JSON Schema rule description language. The rules include field type constraints, logical condition judgments, and business thresholds. The template building unit is used to associate defined review rules with corresponding approval roles and generate an approval template specific to that role through a web configuration interface. The template serialization unit is used to serialize the approval template into a structured Prompt template, which serves as the fixed context input for the intelligent review model specific to the corresponding role.

[0060] In some embodiments, the approval configuration module further includes: The data acquisition unit is used to collect historical approval logs of the target approval role. The logs include the content of the approval documents, marked issues, and the final approval opinions. An instruction fine-tuning dataset is constructed based on the logs. The model adaptation unit is used to add a LoRA adapter module to the pre-trained large language model. The LoRA adapter is injected into the self-attention layer of the original model through a low-rank decomposition matrix. The training optimization unit is used to fine-tune the dataset using the instructions and to perform distributed training on the parameters of the LoRA adapter module on the GPU cluster. During the training process, mixed precision computation and gradient checkpointing techniques are used to optimize the use of video memory. The model generation unit is used to store the trained LoRA adapter weights, combine them with the pre-trained model base, and generate a character-specific intelligent review model.

[0061] In some embodiments, the file parsing module includes: The file receiving unit is used to receive uploaded files pending approval and automatically identify their file types. The channel distribution unit is used to distribute files to corresponding parsing channels according to the identified file type. The parsing channels include at least plain text parsing channels, layout document parsing channels, and image parsing channels. The content parsing unit is used to: if the file is distributed to the plain text parsing channel, directly extract the text and metadata in the file; if the file is distributed to the layout document parsing channel or the image parsing channel, call the OCR recognition algorithm to extract the text content, and call the layout analysis model to recognize the physical and logical structure of the document. The data standardization unit is used to uniformly convert the parsed text, layout structure, and semantic units into a structured JSON data format. The JSON data includes one or more of the following: text content, font style, paragraph information, table structure, and element coordinate information.

[0062] In some embodiments, the intelligent review module includes: The prompt word construction unit is used to integrate the rules defined in the approval template with the document content in the structured JSON data to construct a mind chain prompt word that includes review context and step-by-step reasoning instructions; The model reasoning unit is used to input the prompt words into the role-specific intelligent review model and guide the model to perform review reasoning step by step according to the preset reasoning chain; The result encapsulation unit is used to map and encapsulate the model's inference results into structured JSON format review intermediate results.

[0063] In some embodiments, the knowledge base module includes: The vector encoding unit is used to input text fragments and question types from the intermediate results of structured review into the text encoder to generate semantic vectors. The vector retrieval unit is used to use the semantic vector as a query vector, and retrieve the top-K most similar historical cases in the vector database through an approximate nearest neighbor search algorithm. The suggestion generation unit is used to extract the corresponding approval opinions and decision results from the top-K retrieved historical cases and form a set of reference suggestions.

[0064] In some embodiments, the human-computer interaction module includes: The results fusion unit is used to merge intermediate review results with the set of reference recommendations to generate review results; A visualization rendering unit is used to highlight and annotate the original copy of the multimodal document to be approved based on the location information in the review results. The interactive display unit is used to show approvers highlighted copies of documents and corresponding review result details; The instruction receiving unit is used to receive instructions from the approver regarding the review results.

[0065] In some embodiments, the result fusion unit includes: The input splicing subunit is used to splice the intermediate results of the structured review with the set of reference suggestions to form a comprehensive input context; The model reasoning subunit is used to submit the comprehensive input context to the role-specific intelligent review model for final review and enhancement. The confidence level correction subunit is used to perform weighted evaluation and correction of the confidence level of each issue in the current review intermediate results based on the historical approval opinions in the reference suggestion set. The background supplement subunit is used to extract relevant background information and decision-making basis from the reference suggestion set and supplement it to the explanation of the current review intermediate results; The result generation sub-unit is used to output the enhanced final review result.

[0066] In some embodiments, the opinion generation module includes: The information extraction unit is used to extract key information from the review results; The results optimization unit is used to adjust and optimize the review results in accordance with the approver's instructions; The template retrieval unit is used to retrieve standard approval opinion templates from the knowledge base module; The opinion generation unit is used to combine the adjusted review results with the standard approval opinion template and generate standardized approval opinions using a large language model. The formatting processing unit is used to format the generated approval comments.

[0067] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal intelligent document approval method based on a large language model, characterized in that, Includes the following steps: S1. Configure exclusive approval templates for different approval roles. Based on the historical approval data of each approval role, use LoRA parameter fine-tuning technology to perform personalized training on the pre-trained large language model and generate an intelligent review model exclusive to each role. S2. Receive a multimodal file to be approved. The multimodal file includes at least one of PDF documents, Word text files, and image files. Distribute the file to the corresponding parsing channel according to the file type. Parse the file content using OCR recognition technology and layout analysis algorithm. Output structured JSON data containing text content, table structure, signature area, and handwritten content. S3. Input the structured JSON data and the configured approval template into the corresponding intelligent review model, perform step-by-step reasoning through the thinking chain prompt strategy, and output a structured review intermediate result containing question type, text fragment, location information and confidence level. S4. Encode the intermediate review results into a semantic vector; based on the semantic vector, retrieve the Top-K similar historical approval cases from the knowledge base module based on the vector database; and generate a set of reference suggestions based on the retrieved cases; S5. The intermediate review results are merged with the reference suggestion set to generate the review results. The review results are then visualized through the human-computer interaction module, including highlighting the problem items in the original text and receiving confirmation or modification instructions from the approver. S6. Based on the review results and the approver's feedback, generate standardized approval opinions.

2. The intelligent approval method for multimodal documents based on a large language model according to claim 1, characterized in that, In S1, the steps for configuring exclusive approval templates for different approval roles include: Create and define different approval roles, and assign a unique role identifier to each role; Based on the JSON Schema rule description language, structured review rules are defined for various approval documents. These rules include field type constraints, logical condition judgments, and business thresholds. Associate the defined review rules with the corresponding approval roles, and generate an approval template specific to that role through the web configuration interface; The approval template is serialized into a structured Prompt template, which serves as the fixed context input for the intelligent review model specific to the corresponding role.

3. The intelligent approval method for multimodal documents based on a large language model according to claim 2, characterized in that, In S1, the steps for personalized training of the pre-trained large language model include: Collect historical approval logs of the target approval role. The logs include the content of the approval documents, marked issues, and the final approval comments. Construct an instruction fine-tuning dataset based on the logs. Based on the pre-trained large language model, a new LoRA adapter module is added. The LoRA adapter is injected into the self-attention layer of the original model through a low-rank decomposition matrix. The instructions are used to fine-tune the dataset, and the parameters of the LoRA adapter module are trained in a distributed manner on the GPU cluster. During the training process, mixed precision computation and gradient checkpointing techniques are used to optimize the use of video memory. Save the trained LoRA adapter weights and combine them with the pre-trained model base to generate a character-specific intelligent censorship model.

4. The intelligent approval method for multimodal documents based on a large language model according to claim 3, characterized in that, The steps in S2 include: S21. Receive uploaded documents pending approval and automatically identify their file types; S22. Based on the identified file type, the file is distributed to the corresponding parsing channel, wherein the parsing channel includes: Plain text parsing channel: used for processing Word text files; Layout document parsing channel: used for processing PDF documents; Image parsing channel: used for processing JPG and PNG image files; If the file is distributed to the plain text parsing channel, the text and metadata in the file are extracted directly. If the file is distributed to the document parsing channel or image parsing channel, the OCR recognition algorithm is called to extract the text content, and the layout analysis model is called to recognize the table structure, signature area and handwritten content in the document. S23. Convert the parsed text content, table structure, signature area information, and handwritten content into a structured JSON data format.

5. The intelligent approval method for multimodal documents based on a large language model according to claim 4, characterized in that, The steps in S3 include: S31. Integrate the rules defined in the approval template with the document content in the structured JSON data to construct a mind chain prompt containing review context and step-by-step reasoning instructions; S32. Input the prompt words into the role-specific intelligent review model, and guide the model to perform the following reasoning operations step by step according to the preset reasoning chain: Identify and extract all key information fields related to the approval rules from the document content; The extracted key fields are compared one by one with the constraints, logical rules and business thresholds defined in the approval template. Based on the comparison results, determine whether each key field or combination of fields conforms to the rules and generate the corresponding confidence score; For items judged to be inconsistent with the rules or with a confidence level below the confidence threshold, generate natural language explanations; S33. Map and encapsulate the inference results of the model into a structured JSON format review intermediate result, wherein the review intermediate result includes at least the question type, the associated text fragment, the location information of the fragment in the original text, and the confidence level of the judgment.

6. The intelligent approval method for multimodal documents based on a large language model according to claim 5, characterized in that, The steps in S4 specifically include: S41. Input the text fragments and question types from the intermediate review results into the text encoder to generate semantic vectors; S42. Using the semantic vector as the query vector, perform an approximate nearest neighbor search algorithm in the vector database to calculate its similarity to the historical approval case vectors in the knowledge base, and return the Top-K most similar historical cases. S43. Extract the corresponding approval opinions and decision results from the top-K historical cases retrieved, and form a set of reference suggestions.

7. The intelligent approval method for multimodal documents based on a large language model according to claim 6, characterized in that, The steps in S5 include: S51. Integrate the intermediate review results with the reference suggestion set to generate the review results; S52. Based on the location information in the review results, highlight the original copy of the multimodal document to be approved, and use different colors or icons to distinguish different problem types. S53. Display the highlighted document copy and corresponding review result details to the approver through a human-computer interaction interface; S54. Receive instructions from the approver regarding the review results, including confirmation of the issue item, ignoring it, modifying the approval opinion, or manually adding a new issue item.

8. The intelligent approval method for multimodal documents based on a large language model according to claim 7, characterized in that, The specific steps of S51 include: S511. Combine the intermediate results of the structured review with the set of reference suggestions to form a comprehensive input context; S512. Submit the comprehensive input context to the role-specific intelligent review model and provide an instruction prompt, requesting the model to conduct a final review and enhancement of the review results; S513. Based on the historical approval opinions in the reference suggestion set, the confidence levels of each issue in the current review intermediate results are weighted and corrected to generate the corrected confidence levels; S514. Extract relevant background information and decision-making basis from the reference recommendations and add them to the explanation of the current review intermediate results; S515. Output an enhanced final review result with confidence level correction and supplementary background information.

9. The intelligent approval method for multimodal documents based on a large language model according to claim 8, characterized in that, The steps in S6 include: S61. Extract key information from the review results, including the question type, text fragments, location information, and confidence level; S62. Adjust and optimize the review results based on the approver's confirmation or modification instructions; S63. Retrieve standard approval opinion templates related to the review results from the knowledge base module; S64. Combine the adjusted review results with the standard approval opinion template, and use a large language model to generate standardized approval opinions; S65. Format the generated approval comments.

10. A multimodal intelligent document approval system based on a large language model, used to implement the method described in any one of claims 1-9, characterized in that, include: The approval configuration module is used to configure exclusive approval templates for different approval roles. Based on the historical approval data of each approval role, the LoRA parameter high-efficiency fine-tuning technology is used to perform personalized training on the pre-trained large language model to generate an intelligent review model exclusive to each role. The file parsing module is used to receive multimodal files to be approved, distribute them to the corresponding parsing channels according to the file type, parse the file content through OCR recognition technology and layout analysis algorithm, and output structured JSON data. The intelligent review module is used to input the structured JSON data and the approval template into the corresponding intelligent review model, perform step-by-step reasoning through the thinking chain prompt strategy, and output the intermediate results of the structured review. The knowledge base module is used to encode intermediate review results into semantic vectors, and based on these semantic vectors, retrieve similar historical approval cases from the vector database and generate a set of reference suggestions. The human-computer interaction module is used to integrate intermediate review results with the set of reference suggestions to generate review results, and to visualize and receive instructions. The feedback generation module is used to generate standardized approval feedback based on the review results and the approver's feedback.

Images