Multi-agent collaborative form information extraction method and system and terminal equipment

Abstract

A multi-agent collaborative form information extraction method comprises the following steps: S1, constructing a task planning layer, and dynamically generating a task framework chain adaptive to a target form by a large model through an iterative optimization mechanism based on a TTA dynamic adjustment thought; s2, constructing an execution reflection layer, scheduling a corresponding agent plug-in in an agent plug-in library to execute a task framework chain by the large model according to the task framework chain output by the task planning layer, and generating an execution result and feedback reflection; and S3, updating the execution result output by the execution reflection layer and the feedback reflection to the task planning layer for iterative optimization of the next round of task framework chain. By dynamically adjusting the frame chain of the form analysis process, the large model can schedule the agent plug-in in the field corresponding to the form, so that the source domain general knowledge of the large model is dynamically aligned with the target domain form features, field adaptation is realized, the accuracy of form information extraction is improved, and the efficiency of form information extraction is improved.

Description

Technical Field

[0001] The present invention relates to the field of multi-agent collaborative technology, and in particular to a multi-agent collaborative form information extraction method, system and terminal device. Background Art

[0002] With the rapid development of artificial intelligence (AI), form information extraction (FIT) has been widely used in fields such as finance, healthcare, and law. While traditional natural language processing (NLP) models (such as the BERT series) have demonstrated some capabilities in document understanding, their performance in FIT is less than ideal.

[0003] Currently, some researchers are attempting to apply large, general-purpose models to document understanding tasks in the hope of improving performance. However, these large, general-purpose models often struggle to adapt to domain-specific form recognition tasks. To address this issue, researchers typically fine-tune the large models. While this fine-tuning approach can improve performance to a certain extent, it has significant drawbacks. First, it requires a large amount of domain-specific annotated data, which is often costly and time-consuming to obtain. Second, the fine-tuning process consumes vast amounts of computing resources, placing high demands on both hardware infrastructure and energy supply.

[0004] In addition to fine-tuning, a large-model agent-based approach has also been applied to form information extraction. However, this approach presents a key issue: the agent may lack domain-specific background knowledge, leading to poor performance. For example, agents are often inaccurate when extracting location information. In such cases, manual intervention is required, using OCR (optical character recognition) tools to extract text content and location information from document images. The large model is then used to perform structured analysis on these extracted results.

[0005] However, even with the introduction of external tools like OCR, extracting form information still faces numerous challenges. Because large models lack understanding of the technical limitations of external tools, the problem of large model hallucinations becomes more serious when performing post-processing operations such as structuring the output of external tools. This means that the content generated by the large model does not match the actual situation and lacks factual basis, thus affecting the accuracy of form information extraction. Summary of the Invention

[0006] The purpose of the present invention is to provide a method, system and terminal device for collaborative extraction of form information by multiple agents, so as to improve the efficiency and accuracy of form information extraction through the collaborative action of multiple agents.

[0007] To achieve the above objectives, the present invention provides a solution: a multi-agent collaborative form information extraction method, comprising the following steps: S1: Build the task planning layer. Based on the TTA dynamic adjustment concept, the large model dynamically generates a task framework chain that adapts to the target form through an iterative optimization mechanism; S2: Build the execution reflection layer. The large model dispatches the corresponding agent plug-in in the agent plug-in library to execute the task framework chain based on the task framework chain output by the task planning layer, and generates execution results and feedback reflections. S3: Update the execution results and feedback reflections output by the execution reflection layer to the task planning layer for iterative optimization of the next round of task framework chain.

[0008] Preferably, the task planning layer includes task background, task requirements, example form result set, agent plug-in library and current framework chain; The task planning layer uses a large model to select an intelligent plug-in from the intelligent plug-in library according to the task background, task requirements and example form result set to optimize the current framework chain, generate a task framework chain that adapts to the target form and transmit it to the execution reflection layer, and the task framework chain is synchronously updated to the current framework chain.

[0009] In the preferred embodiment, the execution reflection layer executes the task framework chain according to the original form according to the corresponding intelligent agent plug-in in the task framework chain scheduling intelligent agent plug-in library to obtain the execution result. The execution result and the original form are input into the large model together to generate feedback reflection of the execution result. The execution result and feedback reflection are updated to the example form result set of the task planning layer for the next round of iterative optimization of the current framework chain.

[0010] In a preferred embodiment, the agent plug-in library includes an OCR tool plug-in, a structured parsing plug-in, a multi-page integration plug-in, a missing location and completion plug-in, and a hallucination field filtering plug-in; When the intelligent agent plug-in executes the task framework chain, the OCR tool plug-in performs text recognition and position coordinate extraction on the original form to obtain recognition and extraction results, and the recognition and extraction results are input into the structured parsing plug-in for JSON structure organization, and the preliminary complete JSON data of the original form is obtained through the multi-page integration plug-in. Then, the missing information positioning and completion plug-in and the hallucination field filtering plug-in are used to process the missing information and large model hallucination field problems of the JSON data, and finally the execution result is obtained.

[0011] Preferably, the task framework chain includes a prompt design module, an analysis module, an intelligent correction module and a final output module; The OCR tool plug-in fills the recognition and extraction results of the original form into the prompt design module, which includes background information, task description, structure perception rules and precautions. The structure perception rules establish a spatial mapping relationship between the recognition and extraction results of the original form and the standard JSON structure template, so that the large model obtains the row and column organization structure of the data in the original form, thereby allowing the prompt design module to generate prompts based on the recognition and extraction results of the original form.

[0012] In a preferred embodiment, in the prompt input parsing module generated by the prompt design module, the structured parsing plug-in organizes the recognition and extraction results of the original form into JSON structures page by page, and recursively merges them through a multi-page integration plug-in to obtain the JSON data of the original form. Each field of the JSON data contains both text and corresponding location coordinate information.

[0013] In a preferred embodiment, the intelligent correction module adopts a two-level verification mechanism. The missing location and completion plug-in locates the missing fields of the JSON data processed by the structured parsing plug-in, and uses local image cropping and visual large model re-recognition technology to complete the missing text content caused by the OCR tool plug-in recognition error. The hallucination field filtering plug-in removes the content in the JSON data that is inconsistent with the original form semantics through coordinate backtracking, thereby obtaining the final execution result and inputting it into the final output module.

[0014] Preferably, the large model is an LLM model.

[0015] A multi-agent collaborative form information extraction system, the system comprising: The task planning module is configured to build the task planning layer. Based on the TTA dynamic adjustment concept, the large model dynamically generates a task framework chain that adapts to the target form through an iterative optimization mechanism; The execution reflection module is configured to build the execution reflection layer. The large model dispatches the corresponding agent plug-in in the agent plug-in library to execute the task framework chain according to the task framework chain output by the task planning module, and generates execution results and feedback reflections; The update feedback module is configured to update the execution results and feedback reflections output by the execution reflection module to the task planning module for iterative optimization of the next round of task framework chain.

[0016] A terminal device includes a memory and a processor, wherein the memory stores a computer program running on the processor, and when the processor executes the computer program, it implements the above-mentioned multi-agent collaborative form information extraction method.

[0017] After adopting the above solution, the beneficial effects of the present invention are: The task planning layer of the present invention is based on the TTA dynamic adjustment idea. The large model dynamically generates a task framework chain that adapts to the target form through an iterative optimization mechanism. In the execution reflection layer, the large model schedules the corresponding intelligent agent plug-in in the intelligent agent plug-in library to execute the task framework chain according to the task framework chain output by the task planning layer, generates execution results and feedback reflections, and finally the execution results and feedback reflections will be updated to the task planning layer for the next round of iterative optimization of the task framework chain.

[0018] Therefore, the present invention dynamically adjusts the framework chain of the form parsing process, and the large model can dispatch the intelligent agent plug-in corresponding to the form field from the intelligent agent plug-in library, so that the general knowledge of the source domain of the general large model is dynamically aligned with the form features of the target domain, realizing domain adaptation to improve the accuracy of form information extraction, avoiding the dependence of traditional large model fine-tuning solutions on labeled data and computing power, and helping to improve the efficiency of form information extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of a two-layer architecture of a task planning layer and an execution reflection layer in an embodiment of the present invention; Figure 2 It is a schematic diagram of the collaborative operation of multiple intelligent agent plug-ins in the task framework chain in an embodiment of the present invention.

[0020] Description of labels: 1. Task planning layer; 11. Task background; 12. Task requirements; 13. Sample form results; 14. Agent plug-in library; 15. Current framework chain; 2. Execution reflection layer; 21. Original form; 22. Feedback reflection; 23. Execution results; 3. Large model; 31. LLM model; 4. Task framework chain; 41. Prompt design module; 411. Background information; 412. Task description; 413. Structural perception rules; 414. Precautions; 42. Parsing module; 43. Intelligent correction module; 44. Final output module. DETAILED DESCRIPTION

[0021] The present invention will now be further described with reference to the accompanying drawings and specific embodiments.

[0022] This embodiment provides a multi-agent collaborative form information extraction method, including the following steps: S1: Constructing the task planning layer 1. Based on the TTA dynamic adjustment concept, the large model 3 dynamically generates a task framework chain 4 that adapts to the target form through an iterative optimization mechanism; S2: Construct the execution reflection layer 2. The large model 3 schedules the corresponding agent plug-in in the agent plug-in library 14 to execute the task framework chain 4 according to the task framework chain 4 output by the task planning layer 1, and generates the execution result 23 and feedback reflection 22; S3: Update the execution result 23 and feedback reflection 22 output by the execution reflection layer 2 to the task planning layer 1 for the next round of iterative optimization of the task framework chain 4.

[0023] The task planning layer 1 of this embodiment is based on the dynamic adjustment concept of TTA (test-time adaptation). The large model 3 can dynamically generate an adaptive task framework chain 4 according to the characteristics and requirements of the target form, so as to cope with forms of different types and structures.

[0024] In the execution reflection layer 2, the large model 3 dispatches the corresponding agent plug-in from the agent plug-in library 14 to execute the task framework chain 4 output by the task planning layer 1. This dynamically aligns the general knowledge of the source domain of the large model 3 with the form features of the target domain, achieving domain adaptation. For example, natural language understanding capabilities are matched with forms such as bank statements and contracts. Furthermore, the execution results 23 and feedback reflections 22 are updated to the task planning layer 1, helping the large model 3 optimize the next round of task framework chain 4, thereby improving the accuracy and robustness of form information extraction.

[0025] This embodiment adopts a two-layer architecture consisting of a task planning layer 1 and an execution reflection layer 2, in conjunction with a large model 3 and a variety of intelligent agent plug-ins in the intelligent agent plug-in library 14. The task planning layer 1 is based on the concept of dynamic adjustment using TTA (test-time adaptation). Rather than directly adjusting the parameters of the large model 3 in the traditional sense, the concept of dynamic adjustment is applied to the dynamic adjustment of the task framework chain 4. This eliminates the need to adjust the parameters of the large model 3 each time the task framework chain 4 is optimized, thus avoiding the reliance of traditional large model 3 fine-tuning solutions on labeled data and computing power, and is conducive to improving the efficiency of form information extraction. Of course, during specific operations, the target form can be set according to actual needs, including but not limited to forms in the fields of finance, medical care, and commercial contracts.

[0026] like Figure 1 As shown, the task planning layer 1 includes task background 11, task requirements 12, set of example form results 13, agent plug-in library 14 and current framework chain 15; The task planning layer 1 uses the big model 3 to select the intelligent agent plug-in from the intelligent agent plug-in library 14 according to the task background 11, task requirements 12 and sample form results 13 to optimize the current framework chain 15, generate a task framework chain 4 that adapts to the target form and transmit it to the execution reflection layer 2, and the task framework chain 4 is synchronously updated to the current framework chain 15.

[0027] Specifically, the task context 11 is used to clarify the specific domain and business rule characteristics of the current parsing task. The task context 11 can be modified according to specific needs. It can be a form in a banking scenario or a form in other scenarios such as medical care. Task requirement 12 instructs the system to optimize the current framework chain 15 based on the available intelligent agent plug-ins in the intelligent agent plug-in library 14 to make it more suitable for actual business needs; The 13 sets of sample form results provide typical samples in the target field, enabling the system to accurately capture the layout characteristics and data organization rules of forms in that field, providing a data-driven decision-making basis for the optimization of task framework chain 4; The current framework chain 15 serves as the basis for iterative optimization. Its initial state can adopt a preset general parsing process, such as the form image to visual large model 3 parsing method. During the operation, it will continue to optimize and improve based on execution feedback.

[0028] The task planning layer 1 of this embodiment comprehensively considers multiple dimensions of information, including task context 11, task requirements 12, and a set of sample form results 13. This allows for a more comprehensive understanding of the characteristics and requirements of the target form. This enables the large model 3 to more accurately match form information extraction tasks when selecting agent plug-ins, generating a task framework chain 4 adapted to the target form. When the target form changes, the task planning layer 1 can quickly adjust the task framework chain 4 to accommodate the new form structure or data format, enhancing flexibility.

[0029] like Figure 1 As shown, the execution reflection layer 2 schedules the original form 21 according to the task framework chain 4 and the corresponding intelligent agent plug-in in the intelligent agent plug-in library 14 to execute the task framework chain 4 to obtain the execution result 23. The execution result 23 and the original form 21 are input into the large model 3 together to generate the feedback reflection 22 of the execution result 23. The execution result 23 and the feedback reflection 22 are updated to the example form result 13 of the task planning layer 1 for the next round of iterative optimization of the current framework chain 15.

[0030] In this embodiment, the execution reflection layer 2 inputs the execution results 23 and the original form 21 into the large model 3 to generate feedback reflections 22, forming a closed-loop feedback mechanism that can promptly detect deviations and errors in the execution process and correct and optimize the task framework chain 4 through feedback reflections 22, thereby improving the accuracy of information extraction. The execution results 23 and feedback reflections 22 are then updated to the example form results 13 of the task planning layer 1, used for the next round of iterative optimization of the current framework chain 15. This enhances the generalization capabilities of the task framework chain 4 and enables it to better handle various complex and changing form information extraction tasks.

[0031] Furthermore, the agent plug-in library 14 includes an OCR tool plug-in, a structured parsing plug-in, a multi-page integration plug-in, a missing location and completion plug-in, and a hallucination field filtering plug-in; Among them, the OCR tool plug-in is responsible for recognizing the text content and sub-pixel text coordinates in the original form 21 image; The structured parsing plug-in combines the text and spatial location information of the original form 21 with the large language model, and establishes a spatial mapping relationship with the predefined standard JSON structure template, so that the large model 3 can accurately understand the row and column organization structure of the table data; The multi-page integration plug-in can be used to integrate the processing results of each page of the original form 21 into a complete document parsing output; The missing field location and completion plug-in is used to repair missing fields. It can accurately complete key information through intelligent positioning of missing areas, local image cropping, and visual large model 3 auxiliary recognition. The illusion field filter plug-in eliminates the illusions produced by the large model 3, that is, the content that does not conform to the actual situation and lacks factual basis.

[0032] When the intelligent agent plug-in executes the task framework chain 4, the OCR tool plug-in performs text recognition and position coordinate extraction on the original form 21 to obtain the recognition and extraction results, and the recognition and extraction results are input into the structured parsing plug-in for JSON structure organization, and the preliminary complete JSON data of the original form 21 is obtained through the multi-page integration plug-in, and then the missing information positioning and completion plug-in and the hallucination field filtering plug-in are used to process the missing information of the JSON data and the hallucination field problem of the large model 3, and finally the execution result 23 is obtained.

[0033] like Figure 2 As shown, the task framework chain 4 includes a prompt design module 41, an analysis module 42, an intelligent correction module 43 and a final output module 44; Specifically, the OCR tool plug-in fills the recognition and extraction results of the original form 21 into the prompt design module 41, which includes background information 411, task description 412, structure perception rules 413 and precautions 414. The structure perception rules 413 establish a spatial mapping relationship between the recognition and extraction results of the original form 21 and the standard JSON structure template, so that the large model 3 obtains the row and column organization structure of the data in the original form 21, thereby the prompt design module 41 generates a prompt based on the recognition and extraction results of the original form 21.

[0034] Based on the accurate understanding of the structure of the original form 21, the large model 3 can more accurately map the original form 21 data to the standard JSON structure template in the subsequent parsing process, thereby improving the accuracy of structured parsing.

[0035] Furthermore, in the prompt input parsing module 42 generated by the prompt design module 41, the structured parsing plug-in organizes the recognition and extraction results of the original form 21 into JSON structure page by page, and recursively merges them through the multi-page integration plug-in to obtain the JSON data of the original form 21. Each field of the JSON data contains both text and corresponding location coordinate information.

[0036] The structured parsing plugin organizes the extracted results of the original form 21 into a JSON structure page by page, ensuring that each page's data is organized according to a unified JSON format, facilitating subsequent data processing, storage, and analysis. The multi-page integration plugin recursively merges the JSON data from each page into the complete JSON data of the original form 21, making the data structure clearer and easier to understand.

[0037] Furthermore, the intelligent correction module 43 adopts a two-level verification mechanism. The missing location and completion plug-in locates the missing fields of the JSON data processed by the structured parsing plug-in, and uses local image cropping and visual large model three-recognition technology to complete the missing text content caused by the OCR tool plug-in recognition error. The hallucination field filtering plug-in eliminates the content in the JSON data that is semantically inconsistent with the original form 21 through coordinate backtracking, thereby obtaining the final execution result 23 which is input into the final output module 44.

[0038] The missing location and completion plug-in can accurately locate the missing fields in the JSON data processed by the structured parsing plug-in. For example, when processing a form containing multiple pages of complex tables, the OCR tool plug-in may be affected by factors such as image quality and font style during the recognition process, resulting in some text content not being correctly recognized, thereby forming a missing field in the JSON data. At this time, the missing location and completion plug-in can complete the missing field through local image cropping and visual large model 3-recognition technology. In the process of form information extraction, the large model 3 may produce content that is semantically inconsistent with the original form 21 for various reasons, that is, hallucination fields. At this time, the hallucination field filtering plug-in compares each field in the JSON data with the original form 21 through coordinate backtracking. Once semantically inconsistent content is found, it is immediately removed, thereby ensuring that the final output JSON data is highly consistent with the original form 21, thereby improving the accuracy of the data.

[0039] like Figure 2 As shown, a missing field B- and an illusion field D+ appear in the JSON data AB-CD+ generated by the parsing module 42. At this time, the intelligent correction module 43 updates the missing field B- through the missing location and completion plug-in, and deletes the illusion field D+ through the illusion field deletion plug-in, and finally outputs the correct JSON data ABCD.

[0040] Furthermore, in this embodiment, large model 3 is LLM model 31, which can accurately understand the semantics of the form and generate high-quality output, but is not limited to this. In other embodiments, other large models 3 may be used according to actual needs. Large model 3 mentioned in this embodiment is artificial intelligence large model 3, which refers to a type of artificial intelligence model with a large number of parameters constructed by an artificial neural network.

[0041] This embodiment also provides a multi-agent collaborative form information extraction system, the system comprising: The task planning module is configured to build the task planning layer 1. Based on the TTA dynamic adjustment concept, the large model 3 dynamically generates a task framework chain 4 that adapts to the target form through an iterative optimization mechanism; The execution reflection module is configured to construct the execution reflection layer 2. The large model 3 schedules the corresponding agent plug-in in the agent plug-in library 14 to execute the task framework chain 4 according to the task framework chain 4 output by the task planning module, and generates the execution result 23 and feedback reflection 22; The update feedback module is configured to update the execution result 23 and feedback reflection 22 output by the execution reflection module to the task planning module for iterative optimization of the next round of task framework chain 4.

[0042] This embodiment also provides a terminal device, including a memory and a processor, wherein the memory stores a computer program running on the processor, and when the processor executes the computer program, it implements the above-mentioned multi-agent collaborative method for extracting form information.

[0043] The above description is only a preferred embodiment of the present invention and is not intended to limit the design of this case. Any equivalent changes made based on the key design of this case shall fall within the scope of protection of this case.

Claims

1. A multi-agent collaborative form information extraction method, characterized by: The following steps are involved: S1: Build the task planning layer. Based on the TTA dynamic adjustment concept, the large model dynamically generates a task framework chain that adapts to the target form through an iterative optimization mechanism; S2: Build the execution reflection layer. The large model dispatches the corresponding agent plug-in in the agent plug-in library to execute the task framework chain based on the task framework chain output by the task planning layer, and generates execution results and feedback reflections. S3: Update the execution results and feedback reflections output by the execution reflection layer to the task planning layer for iterative optimization of the next round of task framework chain.

2. The multi-agent collaborative form information extraction method according to claim 1, characterized in that: The task planning layer includes task background, task requirements, sample form result set, agent plug-in library and current framework chain; The task planning layer uses a large model to select an intelligent plug-in from the intelligent plug-in library according to the task background, task requirements and example form result set to optimize the current framework chain, generate a task framework chain that adapts to the target form and transmit it to the execution reflection layer, and the task framework chain is synchronously updated to the current framework chain.

3. The multi-agent collaborative form information extraction method according to claim 2, characterized in that: The execution reflection layer executes the task framework chain according to the corresponding intelligent agent plug-in in the task framework chain scheduling intelligent agent plug-in library using the original form to obtain the execution result. The execution result and the original form are input into the large model together to generate feedback reflection of the execution result. The execution result and feedback reflection are updated to the example form result set of the task planning layer for the next round of iterative optimization of the current framework chain.

4. The multi-agent collaborative form information extraction method according to claim 3, characterized in that: The intelligent agent plug-in library includes an OCR tool plug-in, a structured parsing plug-in, a multi-page integration plug-in, a missing location and completion plug-in, and a hallucination field filtering plug-in; When the intelligent agent plug-in executes the task framework chain, the OCR tool plug-in performs text recognition and position coordinate extraction on the original form to obtain recognition and extraction results, and the recognition and extraction results are input into the structured parsing plug-in for JSON structure organization, and the preliminary complete JSON data of the original form is obtained through the multi-page integration plug-in. Then, the missing information positioning and completion plug-in and the hallucination field filtering plug-in are used to process the missing information and large model hallucination field problems of the JSON data, and finally the execution result is obtained.

5. The multi-agent collaborative form information extraction method according to claim 4, characterized in that: The task framework chain includes a prompt design module, an analysis module, an intelligent correction module and a final output module; The OCR tool plug-in fills the recognition and extraction results of the original form into the prompt design module, which includes background information, task description, structure perception rules and precautions. The structure perception rules establish a spatial mapping relationship between the recognition and extraction results of the original form and the standard JSON structure template, so that the large model obtains the row and column organization structure of the data in the original form, thereby allowing the prompt design module to generate prompts based on the recognition and extraction results of the original form.

6. The multi-agent collaborative form information extraction method according to claim 5, characterized in that: In the prompt input parsing module generated by the prompt design module, the structured parsing plug-in organizes the recognition and extraction results of the original form into JSON structure page by page, and recursively merges them through the multi-page integration plug-in to obtain the JSON data of the original form. Each field of the JSON data contains both text and corresponding location coordinate information.

7. The multi-agent collaborative form information extraction method according to claim 6, characterized in that: The intelligent correction module adopts a two-level verification mechanism. The missing location and completion plug-in locates the empty fields of the JSON data processed by the structured parsing plug-in, and uses local image cropping and visual large model re-recognition technology to complete the missing text content caused by the OCR tool plug-in recognition error. The hallucination field filtering plug-in removes the content in the JSON data that is inconsistent with the original form semantics through coordinate backtracking, thereby obtaining the final execution result and inputting it into the final output module.

8. The multi-agent collaborative form information extraction method according to claim 1, characterized in that: The large model is the LLM model.

9. A multi-agent collaborative form information extraction system, characterized by: The system comprises: The task planning module is configured to build the task planning layer. Based on the TTA dynamic adjustment concept, the large model dynamically generates a task framework chain that adapts to the target form through an iterative optimization mechanism; The execution reflection module is configured to build the execution reflection layer. The large model dispatches the corresponding agent plug-in in the agent plug-in library to execute the task framework chain according to the task framework chain output by the task planning module, and generates execution results and feedback reflections; The update feedback module is configured to update the execution results and feedback reflections output by the execution reflection module to the task planning module for iterative optimization of the next round of task framework chain.

10. A terminal device, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program running on the processor, and when the processor executes the computer program, it implements a multi-agent collaborative form information extraction method described in any one of claims 1-8.

Images