Multi-agent collaborative file large model fine tuning data set construction method and system

Abstract

The invention discloses a multi-agent collaborative file large model fine tuning data set construction method and system, and relates to the technical field of file processing. In order to solve the problems that images in archive documents are blurred, handwritten forms are difficult to recognize, OCR errors are high and the like, the adopted scheme comprises the steps that text information of different types of archive documents is obtained and output after being cleaned and verified, then natural paragraph division is completed based on a semantic structure, paragraphs which do not exceed a word number threshold value serve as word blocks, and the word blocks are stored in a database; cutting the paragraphs exceeding the word number threshold value into a plurality of sub-blocks according to a punctuation rule and a word number threshold value strategy; for each sub-block, generating a plurality of questions covering knowledge points through a question generation agent, and matching answers with consistent semantics for the generated questions by an answer search agent based on original text information to form question and answer pairs; performing matching verification and quality evaluation on the question-answer pairs, outputting and storing the question-answer pairs to a data set after passing, and arranging the question-answer pairs into a standard format to be used as a large model fine-tuning data set.

Description

Technical Field

[0001] This invention relates to the field of archival processing technology, specifically to a method and system for constructing a fine-tuning dataset for a large archival model through multi-agent collaboration. Background Technology

[0002] With the widespread application of Large Language Models (LLMs) in general domains, adapting them to vertical industries such as archival management and historical document processing has become a current research hotspot. However, archival data has unique characteristics, which poses a significant challenge to the construction of fine-tuning data for large models, specifically as follows:

[0003] 1. The complexity and recognition challenges of archival data. Archival data typically contains a large amount of handwriting, blurry scans, and unstructured layouts (such as tables, seals, and annotations). Traditional OCR technology has low accuracy in such scenarios, resulting in typos, missing fields, or formatting issues in the text. Furthermore, archival documents often involve old documents and special symbols (such as page numbers, superscripts, and corrections), further increasing the difficulty of machine recognition. If these errors are directly used for fine-tuning large models, the models will learn noisy data, severely affecting the accuracy and professionalism of the generated text.

[0004] 2. Limitations of Existing Data Construction Methods. Currently, the construction of large-scale model fine-tuning datasets mainly relies on rule-based processing, which has the following shortcomings: ① High labor costs: The scale of archival data is enormous, and manual proofreading and annotation are inefficient, making it difficult to meet the needs of large-scale training. ② Low degree of automation: Existing OCR post-processing technologies (such as regular expression matching and rule templates) are unable to cope with the diversity of archival data and cannot achieve semantic-level error correction and structural restoration. ③ Lack of end-to-end solutions: Existing methods typically handle recognition, cleaning, and structuring independently, leading to error accumulation and ultimately making it difficult to guarantee data quality.

[0005] 3. Development Opportunities for Multimodal and Large-Scale Model Technologies. In recent years, multimodal large-scale models (such as GPT-4V) have demonstrated powerful capabilities in image and text understanding and cross-modal reasoning, while large language models (such as Deepseek) have performed excellently in text generation and semantic analysis. This provides new ideas for the intelligent processing of archival data, such as multimodal collaboration: combining visual models (image parsing) and language models (semantic understanding) can improve the recognition accuracy of handwritten and fuzzy text; agent automation: through the division of labor and cooperation among multiple agents (such as recognition, verification, reconstruction, and evaluation), end-to-end automated data processing workflows can be built, reducing human intervention.

[0006] 4. Industry Demand Drives Technological Innovation. The digitization and intelligentization of archives are crucial directions for cultural preservation and knowledge management, and various fields urgently need high-precision text structuring technologies. Traditional methods struggle to meet the requirements of high efficiency and low error rates, while automated data construction methods based on multi-agent collaboration can significantly improve the accuracy of archival data processing and promote the intelligent upgrading of the industry.

[0007] In summary, existing technologies face problems such as large recognition errors, high dependence on manual labor, and fragmented processes when processing archival data. The development of multimodal large model and intelligent agent collaboration technology provides a new solution for this field. Summary of the Invention

[0008] This invention addresses the problems of blurred images, difficulty in recognizing handwriting, high OCR errors, and severe structural damage in archival documents. It provides a method and system for constructing a large-scale archival model fine-tuning dataset through multi-agent collaboration. With a multi-agent collaboration mechanism as the core, it realizes automated preprocessing of archival data and construction of high-quality question-answer pairs.

[0009] In a first aspect, the present invention provides a method for constructing a fine-tuning dataset for a large-scale archive model through multi-agent collaboration, and the technical solution adopted to solve the above-mentioned technical problems is as follows:

[0010] A method for constructing a fine-tuning dataset for a large-scale archive model through multi-agent collaboration includes the following steps:

[0011] S1. For three types of document files—Word, parsable PDF, and scanned PDF—text information is obtained through direct reading, tool extraction, and multimodal recognition, respectively, and then output after cleaning and verification.

[0012] S2. Based on the semantic structure, the text information output in step S1 is divided into natural paragraphs. Paragraphs that do not exceed the word count threshold are treated as one word block, while paragraphs that exceed the word count threshold are cut into multiple sub-blocks according to punctuation rules and word count threshold strategy, and each sub-block is ensured to be within the context window limit of the large model.

[0013] S3. For each sub-block, firstly, the question generation agent generates multiple questions covering knowledge points, and then the answer search agent matches semantically consistent answers to the generated questions based on the text information output in step S1, forming question-answer pairs.

[0014] S4. Perform matching verification on the question-answer pairs output in step S3, evaluate the quality of the question-answer pairs that pass the matching verification, and save the output of the evaluated question-answer pairs to the structured question-answer pair dataset.

[0015] S5. Organize the question-answer pairs in the question-answer pair dataset into a standard format for use as a dataset for fine-tuning large models.

[0016] Optionally, step S1 specifically includes:

[0017] (a) For Word format archive documents, directly read the text content, first delete the header, footer, table of contents and page numbers, and then clean up typos and semantic errors through a semantic-level document verification agent;

[0018] (b) For parsable PDF format archives, use Python tools to extract text content, complete the cleaning of table of contents, page numbers, headers and footers, and perform semantic verification.

[0019] (c) For scanned PDF documents, the document is first cut into pages and converted into images. The text in the images is then identified by a multimodal parsing agent. The text content is then compared and verified with the images by a multimodal cross-validation agent. Finally, the text is reconstructed and the format is restored.

[0020] Preferably, the semantic-level verification agent involved is based on a locally deployed large language model, which has semantic understanding and automatic error correction capabilities, and can automatically clean page numbers, annotations, punctuation marks and seals;

[0021] The multimodal parsing agent uses a large-scale multimodal model deployed locally, which supports processing image blur, handwriting, and document images with complex layouts;

[0022] The multimodal cross-validation agent relies on a locally deployed cross-modal alignment model, which has the ability to accurately match text and image features. It can automatically verify the consistency between the recognized text and the original image information, correct character omissions, misidentifications and misalignments, and ensure the integrity and accuracy of the archive text.

[0023] Optionally, step S4 is performed, using a quality assessment agent to assess the quality of question-answer pairs that have passed the matching verification from three dimensions: authenticity, relevance, and trainability. The output of the question-answer pairs that have passed the assessment is then saved to a structured question-answer pair dataset.

[0024] Preferably, the question-generating agent, answer-searching agent, and quality assessment agent are all deployed in a local server environment to ensure that the data processing does not depend on external networks and to meet the confidentiality requirements of the archives field.

[0025] Secondly, the present invention provides a multi-agent collaborative system for constructing a large-scale archive model fine-tuning dataset, and the technical solution adopted to solve the above-mentioned technical problems is as follows:

[0026] A multi-agent collaborative system for constructing a fine-tuning dataset for a large archival model, comprising:

[0027] The document processing module is used to obtain text information for three types of document files: Word, parsable PDF, and scanned PDF, through direct reading, tool extraction, and multimodal recognition, respectively, and then output the text after cleaning and verification.

[0028] The text segmentation module is used to divide the text information output by the document processing module into natural paragraphs based on semantic structure. Paragraphs that do not exceed the word count threshold are treated as a single word block, while paragraphs that exceed the word count threshold are trimmed into multiple sub-blocks according to punctuation rules and word count threshold strategies, ensuring that each sub-block is within the context window limit of the large model.

[0029] The question-answer generation module is used to generate multiple questions covering knowledge points for each sub-block through the question generation agent. Then, the answer search agent matches semantically consistent answers to the generated questions based on the text information output by the document processing module, forming question-answer pairs.

[0030] The verification and evaluation module is used to match and verify the question-answer pairs output by the question-answer generation module, evaluate the quality of the question-answer pairs that pass the matching and verification, and save the output of the question-answer pairs that pass the evaluation to a structured question-answer pair dataset.

[0031] The dataset output module is used to organize the question-answer pairs in the question-answer pair dataset into a standard format for use as a dataset for fine-tuning large models.

[0032] Optionally, the document processing modules involved specifically include:

[0033] The Word document processing unit is used to directly read the text content of Word format archive documents, first deleting headers, footers, table of contents and page numbers, and then cleaning up typos and semantic errors through a semantic-level document verification agent;

[0034] The PDF document processing unit is used to extract the text content of parsable PDF format archive documents with the help of Python tools, complete the cleaning operations of table of contents, page numbers, headers and footers, and perform semantic verification processing.

[0035] The scanned document processing unit is used to process scanned PDF documents by first cutting them into pages and converting them into images, then using a multimodal parsing agent to identify the text in the images, and then using a multimodal cross-validation agent to compare and verify the text content with the images, and finally performing text reconstruction and format restoration.

[0036] Preferably, the semantic-level verification agent involved is based on a locally deployed large language model, which has semantic understanding and automatic error correction capabilities, and can automatically clean page numbers, annotations, punctuation marks and seals;

[0037] The multimodal parsing agent uses a large-scale multimodal model deployed locally, which supports processing image blur, handwriting, and document images with complex layouts;

[0038] The multimodal cross-validation agent relies on a locally deployed cross-modal alignment model, which has the ability to accurately match text and image features. It can automatically verify the consistency between the recognized text and the original image information, correct character omissions, misidentifications and misalignments, and ensure the integrity and accuracy of the archive text.

[0039] Optionally, the verification and evaluation module uses a quality evaluation agent to evaluate the quality of question-answer pairs that pass the matching verification from three dimensions: authenticity, relevance, and trainability. The output of the evaluated question-answer pairs is then saved to a structured question-answer pair dataset.

[0040] Preferably, the question-generating agent, answer-searching agent, and quality assessment agent are all deployed in a local server environment to ensure that the data processing does not depend on external networks and to meet the confidentiality requirements of the archives field.

[0041] The present invention provides a method and system for constructing a fine-tuning dataset for a large-scale archive model through multi-agent collaboration. Compared with existing technologies, the advantages of this method and system are as follows:

[0042] 1. This invention realizes the fully automated generation of structured fine-tuning data from original archives, significantly improving the efficiency and quality of data construction, providing highly reliable data support for the targeted fine-tuning of large models in the field of archives, and solving the problems of high dependence on manual labor in constructing datasets in the field of archives and large recognition errors of PDF scans;

[0043] 2. The intelligent agent used in this invention is deployed locally, which meets the security and confidentiality requirements of archival data, and is particularly suitable for application scenarios with high data sensitivity such as government affairs, education, and medical care. Attached Figure Description

[0044] Appendix Figure 1 This is a flowchart illustrating the implementation of the method of the present invention;

[0045] Appendix Figure 2 This is a system module connection diagram of the present invention. Detailed Implementation

[0046] To make the technical solution, the technical problem solved, and the technical effect of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments.

[0047] Example 1:

[0048] Combined with appendix Figure 1This embodiment proposes a method for constructing a fine-tuning dataset for a large-scale archive model through multi-agent collaboration, which includes the following steps:

[0049] S1. For three types of document files—Word, parsable PDF, and scanned PDF—text information is obtained through direct reading, tool extraction, and multimodal recognition, respectively, and then output after cleaning and verification.

[0050] This step specifically includes the following operations:

[0051] (a) For Word format archive documents, directly read the text content, first delete the header, footer, table of contents and page numbers, and then clean up typos and semantic errors through a semantic-level document verification agent;

[0052] (b) For parsable PDF format archives, use Python tools to extract text content, complete the cleaning of table of contents, page numbers, headers and footers, and perform semantic verification.

[0053] (c) For scanned PDF documents, the document is first cut into pages and converted into images. The text in the images is then identified by a multimodal parsing agent. The text content is then compared and verified with the images by a multimodal cross-validation agent. Finally, the text is reconstructed and the format is restored.

[0054] The semantic-level verification agent mentioned in this embodiment is based on a locally deployed large language model, possessing semantic understanding and automatic error correction capabilities, and can automatically clean page numbers, annotations, punctuation, and seals; the multimodal parsing agent adopts a locally deployed large multimodal model, supporting the processing of blurred images, handwritten text, and document images with complex layouts; the multimodal cross-validation agent relies on a locally deployed cross-modal alignment large model, possessing the ability to accurately match text and image features, and can automatically verify the consistency between the recognized text and the original image information, correcting character omissions, misrecognitions, and layout misalignments, ensuring the integrity and accuracy of the archive text.

[0055] S2. Based on the semantic structure, the text information output in step S1 is divided into natural paragraphs. Paragraphs that do not exceed the word count threshold are treated as one word block, while paragraphs that exceed the word count threshold are cut into multiple sub-blocks according to punctuation rules and word count threshold strategy, and each sub-block is ensured to be within the context window limit of the large model.

[0056] S3. For each sub-block, firstly, the question generation agent generates multiple questions covering knowledge points, and then the answer search agent matches semantically consistent answers to the generated questions based on the text information output in step S1, forming question-answer pairs.

[0057] S4. Perform matching and verification on the question-answer pairs output in step S3;

[0058] A quality assessment agent is used to evaluate the quality of question-answer pairs that pass the matching verification from three dimensions: authenticity, relevance, and trainability. The output of the evaluated question-answer pairs is then saved to a structured question-answer pair dataset.

[0059] The question-generating agent, answer-searching agent, and quality assessment agent mentioned in this embodiment are all deployed in a local server environment to ensure that the data processing process does not depend on external networks and meets the confidentiality requirements of the archives field.

[0060] S5. Organize the question-answer pairs in the question-answer pair dataset into a standard format for use as a dataset for fine-tuning large models.

[0061] Example 2:

[0062] Combined with appendix Figure 2 This embodiment proposes a multi-agent collaborative system for constructing a fine-tuning dataset for a large archival model, which includes:

[0063] The document processing module is used to obtain text information for three types of document files: Word, parsable PDF, and scanned PDF, through direct reading, tool extraction, and multimodal recognition, respectively, and then output the text after cleaning and verification.

[0064] The text segmentation module is used to divide the text information output by the document processing module into natural paragraphs based on semantic structure. Paragraphs that do not exceed the word count threshold are treated as a single word block, while paragraphs that exceed the word count threshold are trimmed into multiple sub-blocks according to punctuation rules and word count threshold strategies, ensuring that each sub-block is within the context window limit of the large model.

[0065] The question-answer generation module is used to generate multiple questions covering knowledge points for each sub-block through the question generation agent. Then, the answer search agent matches semantically consistent answers to the generated questions based on the text information output by the document processing module, forming question-answer pairs.

[0066] The verification and evaluation module is used to match and verify the question-answer pairs output by the question-answer generation module, evaluate the quality of the question-answer pairs that pass the matching and verification, and save the output of the question-answer pairs that pass the evaluation to a structured question-answer pair dataset.

[0067] The dataset output module is used to organize the question-answer pairs in the question-answer pair dataset into a standard format for use as a dataset for fine-tuning large models.

[0068] In this embodiment, the document processing module specifically includes:

[0069] The Word document processing unit is used to directly read the text content of Word format archive documents, first deleting headers, footers, table of contents and page numbers, and then cleaning up typos and semantic errors through a semantic-level document verification agent;

[0070] The PDF document processing unit is used to extract the text content of parsable PDF format archive documents with the help of Python tools, complete the cleaning operations of table of contents, page numbers, headers and footers, and perform semantic verification processing.

[0071] The scanned document processing unit is used to process scanned PDF documents by first cutting them into pages and converting them into images, then using a multimodal parsing agent to identify the text in the images, and then using a multimodal cross-validation agent to compare and verify the text content with the images, and finally performing text reconstruction and format restoration.

[0072] It should be added that the semantic-level verification agent involved is based on a locally deployed large language model, which has semantic understanding and automatic error correction capabilities, and can automatically clean page numbers, annotations, punctuation, and seals; the multimodal parsing agent adopts a locally deployed large multimodal model, which supports processing document images with blurred images, handwriting, and complex layouts; the multimodal cross-validation agent relies on a locally deployed cross-modal alignment large model, which has the ability to accurately match text and image features, and can automatically verify the consistency between the recognized text and the original image information, correct character omissions, misrecognitions, and layout misalignments, and ensure the integrity and accuracy of the archival text.

[0073] In this embodiment, the verification and evaluation module uses a quality evaluation agent to evaluate the quality of question-answer pairs that pass the matching verification from three dimensions: authenticity, relevance, and trainability. The output of the evaluated question-answer pairs is then saved to a structured question-answer pair dataset.

[0074] It should be added that the question-generating agent, answer-searching agent, and quality assessment agent involved are all deployed in a local server environment to ensure that the data processing process does not rely on external networks and meets the confidentiality requirements of the archival field.

[0075] In summary, the multi-agent collaborative method and system for constructing a large-scale archival model fine-tuning dataset of the present invention can realize the intelligent and unmanned operation of the entire dataset construction process, significantly reduce labor costs and time consumption, and solve the problems of high dependence on manual labor and large errors in PDF scan recognition in the construction of archival datasets. It can be widely applied to various practical application scenarios such as digital libraries and archival digitization, providing technical support for intelligent archival processing and knowledge extraction.

[0076] The above specific examples illustrate the principles and implementation methods of the present invention in detail. These embodiments are merely for the purpose of helping to understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made to the present invention by those skilled in the art without departing from the principles of the present invention should fall within the patent protection scope of the present invention.

Claims

1. A method for constructing a fine-tuning dataset for a large-scale archive model through multi-agent collaboration, characterized in that, Includes the following steps: S1. For three types of document files—Word, parsable PDF, and scanned PDF—text information is obtained through direct reading, tool extraction, and multimodal recognition, respectively, and then output after cleaning and verification. S2. Based on the semantic structure, the text information output in step S1 is divided into natural paragraphs. Paragraphs that do not exceed the word count threshold are treated as one word block, while paragraphs that exceed the word count threshold are cut into multiple sub-blocks according to punctuation rules and word count threshold strategy, and each sub-block is ensured to be within the context window limit of the large model. S3. For each sub-block, firstly, the question generation agent generates multiple questions covering knowledge points, and then the answer search agent matches semantically consistent answers to the generated questions based on the text information output in step S1, forming question-answer pairs. S4. Perform matching verification on the question-answer pairs output in step S3, evaluate the quality of the question-answer pairs that pass the matching verification, and save the output of the evaluated question-answer pairs to the structured question-answer pair dataset. S5. Organize the question-answer pairs in the question-answer pair dataset into a standard format for use as a dataset for fine-tuning large models.

2. The method for constructing a fine-tuning dataset for a large-scale archive model through multi-agent collaboration according to claim 1, characterized in that, Step S1 specifically includes: (a) For Word format archive documents, directly read the text content, first delete the header, footer, table of contents and page numbers, and then clean up typos and semantic errors through a semantic-level document verification agent; (b) For parsable PDF format archives, use Python tools to extract text content, complete the cleaning of table of contents, page numbers, headers and footers, and perform semantic verification. (c) For scanned PDF documents, the document is first cut into pages and converted into images. The text in the images is then identified by a multimodal parsing agent. The text content is then compared and verified with the images by a multimodal cross-validation agent. Finally, the text is reconstructed and the format is restored.

3. The method for constructing a fine-tuning dataset for a large-scale archive model based on multi-agent collaboration as described in claim 2, characterized in that, The semantic-level verification agent is based on a locally deployed large language model and has semantic understanding and automatic error correction capabilities. It can automatically clean page numbers, annotations, punctuation marks and seals. The multimodal parsing agent adopts a large-scale multimodal model deployed locally, which supports processing image blur, handwriting, and document images with complex layouts; The multimodal cross-validation agent relies on a locally deployed cross-modal alignment model, which has the ability to accurately match text and image features. It can automatically verify the consistency between the recognized text and the original image information, correct character omissions, misidentifications, and layout misalignments, and ensure the integrity and accuracy of the archive text.

4. The method for constructing a fine-tuning dataset for a large-scale archive model through multi-agent collaboration according to claim 1, characterized in that, In step S4, a quality assessment agent is used to evaluate the quality of the question-answer pairs that have passed the matching verification from three dimensions: authenticity, relevance, and trainability. The output of the evaluated question-answer pairs is then saved to a structured question-answer pair dataset.

5. The method for constructing a multi-agent collaborative archive large model fine-tuning dataset according to claim 4, characterized in that, The question-generating agent, answer-searching agent, and quality assessment agent are all deployed in a local server environment, ensuring that the data processing process does not rely on external networks and meets the confidentiality requirements of the archival field.

6. A multi-agent collaborative system for constructing a fine-tuning dataset for a large archival model, characterized in that, It includes: The document processing module is used to obtain text information for three types of document files: Word, parsable PDF, and scanned PDF, through direct reading, tool extraction, and multimodal recognition, respectively, and then output the text after cleaning and verification. The text segmentation module is used to divide the text information output by the document processing module into natural paragraphs based on semantic structure. Paragraphs that do not exceed the word count threshold are treated as a single word block, while paragraphs that exceed the word count threshold are trimmed into multiple sub-blocks according to punctuation rules and word count threshold strategies, ensuring that each sub-block is within the context window limit of the large model. The question-answer generation module is used to generate multiple questions covering knowledge points for each sub-block through the question generation agent. Then, the answer search agent matches semantically consistent answers to the generated questions based on the text information output by the document processing module, forming question-answer pairs. The verification and evaluation module is used to match and verify the question-answer pairs output by the question-answer generation module, evaluate the quality of the question-answer pairs that pass the matching and verification, and save the output of the question-answer pairs that pass the evaluation to a structured question-answer pair dataset. The dataset output module is used to organize the question-answer pairs in the question-answer pair dataset into a standard format for use as a dataset for fine-tuning large models.

7. The system for constructing a multi-agent collaborative archive large model fine-tuning dataset according to claim 6, characterized in that, The document processing module specifically includes: The Word document processing unit is used to directly read the text content of Word format archive documents, first deleting headers, footers, table of contents and page numbers, and then cleaning up typos and semantic errors through a semantic-level document verification agent; The PDF document processing unit is used to extract the text content of parsable PDF format archive documents with the help of Python tools, complete the cleaning operations of table of contents, page numbers, headers and footers, and perform semantic verification processing. The scanned document processing unit is used to process scanned PDF documents by first cutting them into pages and converting them into images, then using a multimodal parsing agent to identify the text in the images, and then using a multimodal cross-validation agent to compare and verify the text content with the images, and finally performing text reconstruction and format restoration.

8. The system for constructing a multi-agent collaborative archive large model fine-tuning dataset according to claim 7, characterized in that, The semantic-level verification agent is based on a locally deployed large language model and has semantic understanding and automatic error correction capabilities. It can automatically clean page numbers, annotations, punctuation marks and seals. The multimodal parsing agent adopts a large-scale multimodal model deployed locally, which supports processing image blur, handwriting, and document images with complex layouts; The multimodal cross-validation agent relies on a locally deployed cross-modal alignment model, which has the ability to accurately match text and image features. It can automatically verify the consistency between the recognized text and the original image information, correct character omissions, misidentifications, and layout misalignments, and ensure the integrity and accuracy of the archive text.

9. The system for constructing a multi-agent collaborative archive large model fine-tuning dataset according to claim 6, characterized in that, The verification and evaluation module uses a quality evaluation agent to evaluate the quality of question-answer pairs that pass the matching verification from three dimensions: authenticity, relevance, and trainability. The output of the evaluated question-answer pairs is then saved to a structured question-answer pair dataset.

10. The system for constructing a multi-agent collaborative archive large model fine-tuning dataset according to claim 9, characterized in that, The question-generating agent, answer-searching agent, and quality assessment agent are all deployed in a local server environment, ensuring that the data processing process does not rely on external networks and meets the confidentiality requirements of the archival field.

Images