Archive structured information extraction method and system based on multi-modal large model, and medium

Abstract

The invention relates to the technical field of natural language processing, in particular to an archive structured information extraction method and system based on a multi-modal large model and a medium, and the method comprises the following steps: S1, establishing a mapping table of fields to be extracted; s2, data annotation; s3, constructing a layout analysis model, an archive structured extraction model and an archive structured integration model; s4, screening key information pages based on the layout analysis model; s5, extracting single-page structured information based on an archive structured extraction model; and S6, based on the archive structured integration model, integrating single-page structured information results. Through application of the multi-modal large model, accurate layout analysis, strong structured extraction capability and efficient information integration are realized, automatic extraction and integration from archive image data to structured information are realized, manual intervention is reduced, and processing efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method, system, and medium for extracting structured information from multimodal large-scale archives. Background Technology

[0002] Structured information extraction from archives is the process of converting digitized image data into structured data, and it is an important means of utilizing digitized archives. Traditionally, structured information extraction from digitized images often involves manual anchor box recognition. This involves manually anchoring selection boxes beforehand, recognizing the text within those boxes, and then binding the recognized content to pre-defined key fields. Building upon manual anchor box recognition, there is also object detection recognition, which replaces the manual anchoring process by first detecting potentially key information text blocks using detection algorithms, and then recognizing the content of those text blocks. Key Information Extraction (KIE) algorithms, based on data annotation and small pre-trained models, are used for fine-tuning or training on documents from specific vertical domains. Universal Information Extraction (UIE) algorithms, based on large-scale pre-trained models, are also used for fine-tuning data from specific vertical domains.

[0003] While manual anchor box recognition is simple and easy to implement, it relies heavily on manual selection and lacks any adaptive capabilities. Object detection and selection recognition algorithms, along with key information extraction algorithms, replace the manual selection process and perform well in scenarios with relatively fixed document layouts. However, their generalization ability is weak, limiting their capacity when handling complex document layouts. General information extraction methods based on large models can be categorized into those based on large language models and those based on multimodal models. Both demonstrate strong generalization capabilities in the field of structured information extraction. However, large language models based on pure text rely on the accuracy of preceding text recognition algorithms and do not utilize any text location information. While multimodal model-based extraction methods do not rely on text recognition results and possess implicit spatial location awareness mechanisms, their perception capabilities are limited, making it difficult to accurately obtain text location information.

[0004] In summary, this paper proposes a structured information extraction method based on a multimodal large model archive. While preserving the generalization ability of the multimodal large model, it explicitly utilizes the coordinate position information of document elements, reduces image-text ambiguity, lowers the probability of the model generating illusions, and helps to locate important information in long texts. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] The main objective of this invention is to provide a method for extracting structured information from multimodal large model archives, in order to solve the problems of weak generalization ability of traditional methods in the prior art, insufficient utilization of text location information by large models, and slow extraction efficiency of long texts.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, this invention provides a method for extracting structured information from multimodal large-scale archives, comprising the following steps:

[0009] S1, Establish a field mapping table to be extracted: Based on the different categories of files to be processed, define the keywords to be extracted for the different files and construct a field mapping table to be extracted, wherein the field mapping table to be extracted includes: the mapping relationship between file category and keyword name to be extracted;

[0010] S2, Data Labeling: Based on the different types of archive datasets to be processed, a set number of archives are sampled for each type. Keywords are extracted from the sampled data according to the field mapping table to be extracted and labeled to obtain the archive sample dataset. The labeling information includes keyword name, keyword text position and keyword text content.

[0011] S3, Construct a layout analysis model, a structured extraction model, and a structured integration model: A layout analysis model is constructed based on the YOLO detection model to obtain layout information from archival image data; a structured extraction model is constructed based on the Qwen series visual multimodal models to extract information from the mapping table definition of the fields to be extracted from archival image data; a structured integration model is constructed based on the Qwen series large language models to integrate the structured information obtained from the structured extraction model.

[0012] S4. Based on the layout analysis model, filter key information pages: Use the trained layout analysis model to perform layout analysis on the archive page image list L, filter out key information pages that contain keywords and have a confidence level greater than a set threshold, and store the key information pages in the key information page list G.

[0013] S5, based on the archive structure extraction model, extracts structured information from a single page; including:

[0014] S51, construct single-page structured information extraction keywords; wherein, the single-page structured information prompts include: role setting and task description: define the file structured extraction model as an image parser to extract keyword information, a list of keywords to be extracted, the format of the returned result, and an example of the returned format;

[0015] S52, based on the single-page structured information extraction words constructed in step S51 and the key information page list G obtained in step S4, the archive structured extraction model is used to sequentially traverse all images in the list G, perform reasoning on each image, and extract information according to the single-page structured information extraction words constructed in step S51 to obtain single-page structured information;

[0016] S53, After extracting all images using the archive structure extraction model, the extracted single-page structured information is combined into a result set R;

[0017] S6, based on the archival structured integration model, integrates the results of single-page structured information, including:

[0018] S61, constructing prompts for the structured integration model; wherein, the prompts for the structured integration model include: role setting and task description: defining the file structured integration model as a text analyzer to extract keyword information, text information sorted by page number, a list of keywords to be extracted, the format of the returned results, and an example of the returned format;

[0019] S62, based on the prompt words set in step S61, the structured information extraction result of the archives is obtained using the structured integration model of the archives.

[0020] Preferably, after step S6, the method further includes: step S7, vector encoding and storage of the extracted structured information of the archive: the structured information extracted in step S62 is encoded using the vector encoding model BAAIGeneralEmbedding to generate an encoded result vector, and the result vector is stored in a vector database.

[0021] Preferably, between step S3 and step S4, the following step is further included: step S31 model training and fine-tuning: based on the archive sample dataset in step S2, the layout analysis model, the archive structure extraction model, and the archive structure integration model are trained and fine-tuned.

[0022] Preferably, step S4 further includes: determining whether the data to be extracted is in image format: if the archive data is not image data, it is converted to image data before layout analysis is performed; if the archive data is image data, no conversion is required, and layout analysis is performed directly.

[0023] Preferably, the number set in step S2 is such that the minimum number of files per category is not less than the set number.

[0024] Preferably, step S31 includes:

[0025] S311, Page Layout Analysis Model Training: Based on the archive sample dataset in step S2, keyword names and keyword text location information are selected to construct an archive page layout analysis model training dataset. The page layout analysis model is trained using the archive page layout analysis model training dataset. The number of training iterations is set to 100 training rounds. The Adam optimizer is selected, and the initial learning rate is set to 0.01, which is then adjusted by decay during the training process.

[0026] S312, Fine-tuning of the archive structure extraction model: Based on the archive sample dataset in step S2, select keyword names and keyword text content information to construct a fine-tuning dataset for the archive structure extraction model. Use the fine-tuning dataset for the archive structure extraction model to fine-tune the archive structure extraction model using the quantized low-rank adaptation method. The fine-tuning is implemented based on the transformers third-party library, and the number of iterations for fine-tuning is set to 10,000 training steps.

[0027] S313, Fine-tuning of the structured integration model of archives: Based on the archive sample dataset in step S2, select keyword names, keyword text content information, and keyword information determined by the final archiving of the entire archive to construct a fine-tuning dataset for the structured integration model of archives. Using the fine-tuning dataset of the structured integration model of archives, fine-tune the structured integration model of archives through the quantized low-rank adaptation method. The fine-tuning is implemented based on the transformers third-party library, and the number of iterations for fine-tuning is set to 10,000 training steps.

[0028] Preferably, the keyword text location information is the position information of the keyword text box on the page; the keyword text content information is the text content information extracted from the corresponding keyword; and the keyword information finally determined for the entire archive is the core keyword name used for archiving and retrieval, which is finally confirmed manually after all processing steps have been completed.

[0029] Preferably, in step S1, the categories of different files to be processed include personnel files, financial files, and engineering files.

[0030] The present invention also provides a structured information extraction system based on multimodal large model archives, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a structured information extraction method based on multimodal large model archives as described above.

[0031] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a method for extracting structured information from a multimodal large model archive as described in any of the preceding claims.

[0032] (III) Beneficial Effects

[0033] This application proposes a method for extracting structured information from archives based on a multimodal large model. By establishing a mapping table of fields to be extracted, the method clarifies the keywords to be extracted for different archive categories, making data labeling more accurate. This avoids the problems of ambiguous field definitions and inaccurate extraction that may exist in existing technologies. Furthermore, the mapping table of fields to be extracted and model parameters can be flexibly adjusted according to different archive categories and needs, exhibiting strong scalability. By constructing prompt words, the method clarifies the model's role setting, task description, page-number-sorted text information, the list of keywords to be extracted, and the format and examples of the returned results. This helps the model better understand the task requirements and improves the accuracy of information extraction. Through the application of a multimodal large model, the method achieves accurate layout analysis, powerful structured extraction capabilities, and efficient information integration, realizing the automated extraction and integration of structured information from archive image data, reducing manual intervention and improving processing efficiency. Attached Figure Description

[0034] Figure 1 This embodiment provides a flowchart illustrating a method for extracting structured information from multimodal large model archives.

[0035] Figure 2 A schematic diagram of a method for extracting structured information from multimodal large model archives, provided for another embodiment;

[0036] Figure 3 This is a schematic diagram illustrating the annotation of a method for extracting structured information from multimodal large model archives provided in this embodiment;

[0037] Figure 4 This embodiment provides a schematic diagram of a layout analysis method for extracting structured information from multimodal large-scale archives.

[0038] Figure 5 This embodiment provides a schematic diagram of single-page structured extraction and structured integration of a method for extracting structured information from a multimodal large model archive.

[0039] Figure 6 This embodiment provides a schematic diagram of the structured extraction result vector encoding and storage process of a structured information extraction method based on multimodal large model archives.

[0040] Figure 7This embodiment provides a schematic diagram of the hardware structure of a structured information extraction system based on a multimodal large model archive. Detailed Implementation

[0041] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0043] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0044] In this invention, unless otherwise explicitly specified and limited, the terms "connection" and "fixed" should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; "connection" can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0045] like Figure 1 As shown, this embodiment provides a method for extracting structured information from a multimodal large model archive, which includes steps S1 to S6.

[0046] S1, Establish a field mapping table to be extracted: Based on the different categories of files to be processed, define the keywords to be extracted for the different files and construct a field mapping table to be extracted, wherein the field mapping table to be extracted includes: the mapping relationship between file category and keyword name to be extracted;

[0047] Specifically, the different types of archives to be processed are the result of a systematic classification by the processor based on the function (business area) of the archives. Their media formats mainly include paper archives and electronic archives: paper archives use paper as the medium and can be, for example, traditional documents or handwritten ledgers. Electronic archives are stored in digital form and can be, for example, PDF documents, digital scans, or document image data.

[0048] Optionally, the different types of archives can be selected according to the actual situation, such as personnel files, financial files, and engineering files. The keywords to be extracted from these different archives are defined as words that accurately reflect the core content or theme. The definition and extraction logic are based on thematic representativeness, business relevance, and distinguishability: Thematic representativeness means directly related to the core content of the archive, such as "name" and "resume" in personnel files. Business relevance means highly relevant to the needs of archive management or utilization, such as "voucher number" and "amount" in accounting files. Distinguishing ability means being able to distinguish different archive categories or individuals, such as "case number" and "judgment result" in litigation files.

[0049] Specifically, the keywords to be extracted from the personnel files include: name, gender, date of birth, ID number, contact information, education, graduating institution, major, start date, position, department, departure date, labor contract number, salary structure, social security contribution base, reward and punishment records, performance rating, and emergency contact person; the keywords to be extracted from the financial files include, for example, invoice code, invoice date, amount, payer / payee, contract number, contract amount, payment time, bank account number, transaction serial number, tax rate, tax amount, reason for reimbursement, approver, accounting subject, voucher number, settlement cycle, audit conclusion, and tax declaration date; and the keywords to be extracted from the engineering files include, for example, project name, project number, construction location, design unit, construction unit, supervision unit, start date, completion date, construction progress, quality acceptance conclusion, equipment model, material specifications, bill of quantities, change order number, as-built drawing number, settlement amount, and file archiving number.

[0050] S2, Data Labeling: Based on the different types of archive datasets to be processed, a set number of archives are sampled for each type. Keywords are extracted from the sampled data according to the field mapping table to be extracted and labeled to obtain the archive sample dataset. The labeling information includes keyword name, keyword text position and keyword text content.

[0051] Optionally, original archival data can be collected from multiple channels such as the archives of partner units, public datasets, and scanned copies of historical archives. The archival classification standards can be defined according to business needs, and the data can be subdivided according to function (business field). The data of different carriers can be formatted and unified into image data, and then cleaned and denoised. Each type of archive can be constructed with an independent data subset to obtain the different types of archival datasets that need to be processed.

[0052] The set quantity can be set according to the actual situation, such as 50 samples, 100 samples or even more than 100 samples for each type of file.

[0053] S3. Constructing a layout analysis model, a structured extraction model, and a structured integration model: A layout analysis model is constructed based on the YOLO (You Only Look Once) detection model to extract layout information from archival image data. The YOLO detection model is characterized by its speed and high accuracy, effectively extracting layout information quickly from archival image data. A structured extraction model is constructed based on the Qwen series visual multimodal models to extract information defined in the field mapping table from archival image data. The Qwen series visual multimodal models can process both image and text information simultaneously, significantly improving archival processing efficiency and data usability. A structured integration model is constructed based on the Qwen series large language models to integrate the structured information obtained from the structured extraction model. The Qwen series large language models have powerful text understanding and generation capabilities, automatically understanding implicit relationships in structured information and generating integrated results that meet business needs, significantly improving archival utilization efficiency and decision support capabilities.

[0054] S4. Based on the layout analysis model, key information pages are selected: The trained layout analysis model is used to perform layout analysis on the list of archival page images L (img1, img2, ..., imgN), selecting key information pages that contain keywords and have a confidence level greater than a set threshold, and storing the key information pages in the key information page list G; this achieves efficient selection from archival images to key information pages, providing a precise data entry point for subsequent structured extraction and integration, and significantly improving the automation level of the archival processing workflow.

[0055] Optionally, in this embodiment, step S4 includes:

[0056] S41, Deploy the layout analysis model: Load the trained layout analysis model, which adapts to the archive scenario through transfer learning and supports the detection of layout elements such as title area, body text area, table area, and stamp area.

[0057] S42, Input preprocessing: Scale the images in the file page image list L(img1,img2,...,imgN) to the model input size to ensure consistent input data distribution.

[0058] S43, Detect layout elements: The layout analysis model performs forward inference on each image in the list of images on the archive page L (img1,img2,...,imgN), and outputs the detected keyword name and its keyword text bounding box coordinates (x1,y1,x2,y2) and confidence score.

[0059] S44, Location Mapping: Combining the keyword text location information marked in step S2, the detected layout elements are associated with the field mapping table to be extracted.

[0060] S45, Filter confidence: Screen layout elements and retain layout elements with a confidence level greater than a set threshold, such as 0.85, to ensure the reliability of the detection results.

[0061] S46, Determine the key page: If a page in the file page image list L(img1,img2,...,imgN) contains at least one high-confidence layout element associated with the keyword, then the page is determined to be a key information page.

[0062] S47, Result Storage and Post-processing: Store the selected key information pages in list G, recording their file paths, detected layout elements, and confidence scores. Post-process list G to remove duplicate pages or pages with similar content.

[0063] S5, based on the archive structure extraction model, extracts structured information from a single page; including:

[0064] S51, construct single-page structured information extraction keywords; wherein, the single-page structured information prompts include: role setting and task description: defining the file structured extraction model as an image parser to extract keyword information, the list of keywords to be extracted, the format of the returned result, and an example of the returned format; in other embodiments, the single-page structured information prompts also include: setting a value of null if the keyword in the image does not contain text content.

[0065] Optionally, step S51 involves guiding the Qwen series visual multimodal model to perform a single-page information extraction task by constructing structured prompts. The single-page structured information prompts include: role settings, task description, keyword list, return format specifications, and additional format example instructions. In other embodiments, the single-page structured information prompts further include: a constraint confidence threshold (forcibly filtering fields with confidence < 0.85) and conflict resolution (selecting the region with the highest confidence when multiple regions match the same field).

[0066] Specifically, the role setting includes defining the model as a "high-precision archive image parser," requiring it to have fine-grained field positioning capabilities.

[0067] The task description includes a clear extraction objective of "extracting keyword information defined in the field mapping table from the current file page", requiring priority extraction of fields from the key information page selected in step S4, and secondary verification (e.g., combining contextual semantic inference) of low-confidence fields (such as fuzzy handwriting).

[0068] The keyword list dynamically loads the field mapping table to be extracted generated in step S1, which may be, for example: personnel file: ["name", "gender", "date of birth", "education", "professional title"] financial file: ["invoice number", "amount", "invoice date", "seller name"] engineering file: ["copy number", "title", "document number", "responsible person", "date of completion"].

[0069] The return format specification mandates the output of a JSON Schema format, for example:

[0070] {"$defs":{"ExtractResult":{"additionalProperties":true,"properties":{},"title":"ExtractResult","type":"object"}},"properties":{"extract_res":{"items":{"$ref":"# / $defs / ExtractResult"},"title":"Extract Res","type":"array"}},"required":["extract_res"],"title":"SingleExtractResult","type":"object"}

[0071] The additional formatting example is {"extract_res":[{"copy number":"","title":"","document number":"","responsible person":"","date of writing":""}]}.

[0072] S52, based on the single-page structured information extraction words constructed in step S51 and the key information page list G(img1,img3,...,imgN) obtained in step S4, the archive structured extraction model is used to sequentially traverse all images in the list G(img1,img3,...,imgN), and reasoning is performed on each image. Information is extracted based on the single-page structured information extraction words constructed in step S51 to obtain single-page structured information;

[0073] S53 employs an archival structured extraction model to extract information from all images. After this process, the extracted single-page structured information is combined into a result set R(img1,img3,...,imgN). This achieves an efficient conversion from archival images to structured data, providing a high-quality data foundation for subsequent integrated analysis.

[0074] S6, based on the archival structured integration model, integrates the results of single-page structured information, including:

[0075] S61, constructing prompts for the structured integration model; wherein, the prompts for the structured integration model include: role setting and task description: defining the file structured integration model as a text analyzer to extract keyword information, text information sorted by page number, a list of keywords to be extracted, the format of the returned results, and an example of the returned format;

[0076] Optionally, step S61 includes: guiding the Qwen series large language model to perform a cross-page information integration task by constructing structured prompt words, wherein the prompt words of the structured integration model include role settings, task description, text information sorted by page number, a list of keywords to be extracted, the format of the returned result, and an example of the returned format.

[0077] Specifically, the role is defined as an "intelligent file integration engine" for the structured integration model, and it is specified that it possesses: long text modeling capabilities and cross-page logical reasoning capabilities.

[0078] The task description clearly states that the integration goal is to "integrate single-page structured information into a complete archive entity in page number order", and sets the implementation to parse cross-page related fields.

[0079] The return format specification mandates the output of a JSON Schema format, for example:

[0080] {"$defs":{"ExtractResult":{"additionalProperties":true,"properties":{},"title":"ExtractResult","type":"object"}},"properties":{"extract_res":{"items":{"$ref":"# / $defs / ExtractResult"},"title":"Extract Res","type":"array"}},"required":["extract_res"],"title":"SingleExtractResult","type":"object"}

[0081] The additional formatting example is: {"extract_res":[{"copy number":"","title":"","document number":"","responsible person":"","date of writing":""}]}.

[0082] S62, based on the prompt words set in step S61, the structured information extraction result of the archives is obtained using the archive structured integration model. This transforms unstructured or semi-structured archive data into a structured format, not only improving the automation level of information processing but also providing solid support for subsequent data applications, demonstrating significant technological advancement and practical value.

[0083] Preferably, after step S6, the method further includes step S7: vector encoding and storage of the extracted structured information results from the archives. The structured information extracted in step S62 is encoded using the BAAI General Embedding vector encoding model to generate encoded result vectors, which are then stored in a vector database. This step not only achieves semantic-level modeling of the archive information but also provides high-quality vector data support for subsequent intelligent retrieval, knowledge discovery, and data analysis, significantly improving the information processing efficiency and intelligence level of the archive management system. The BAAI General Embedding (BGE) model is an open-source text embedding model launched by the Beijing Academy of Artificial Intelligence (BAAI), designed to provide high-quality vector representations for natural language processing tasks. In other embodiments, the vector encoding model can be set according to actual conditions to satisfy any vector encoding model that maps structured information into vector representations in the semantic space, enabling the system to understand and compare the semantic similarity of archive content.

[0084] Preferably, step S3 and step S4 further include: step S31, model training and fine-tuning: based on the archival sample dataset from step S2, the layout analysis model, the archival structure extraction model, and the archival structure integration model are trained and fine-tuned. Through effective training and fine-tuning of the models, the overall performance and usability of the system are greatly improved. This process ensures that the system can still provide efficient and accurate services when faced with diverse and complex archival materials.

[0085] Preferably, step S4 further includes: firstly, checking the input archival data to determine whether the data to be extracted is in image format; if the archival data is not image data, converting it to image data before performing layout analysis, i.e., for non-image format data (such as PDF, Word documents, etc.), it is necessary to convert it to image format first. If the archival data is image data, no conversion is required, and the conversion step can be skipped directly to perform layout analysis. This applies to archival materials that already exist in image form, such as scanned copies, photographs, etc.

[0086] Furthermore, the set quantity in step S2 is that the minimum number of files in each category is not less than a set quantity. Specifically, sampling is carried out according to the minimum number of files in each category being not less than 100.

[0087] Preferably, step S31 includes:

[0088] S311, Layout Analysis Model Training: Based on the archive sample dataset from step S2, keyword names and keyword text location information are selected to construct an archive layout analysis model training dataset. The layout analysis model is trained using this training dataset. The number of training iterations is set to 100 training rounds, the Adam optimizer is selected, and the initial learning rate is set to 0.01, which is then adjusted by decay during the training process. Through training, the archive layout analysis model can more accurately identify the positional relationships of different parts in the document.

[0089] S312, Fine-tuning of the archival structure extraction model: Based on the archival sample dataset from step S2, a fine-tuning dataset for the archival structure extraction model is constructed by selecting keyword names and keyword text content information. The archival structure extraction model is then fine-tuned using this dataset through a quantized low-rank adaptation method. This fine-tuning is implemented using the Transformers third-party library, and the number of iterations is set to 10,000 training steps. This enhances the accuracy and reliability of the archival structure extraction model in practical applications.

[0090] S313, Fine-tuning of the Archival Structured Integration Model: Based on the archival sample dataset from step S2, keyword names, keyword text content information, and keyword information determined by the final archiving of the entire document are selected to construct a fine-tuning dataset for the archival structured integration model. Using this fine-tuning dataset, the archival structured integration model is fine-tuned through a quantized low-rank adaptation method. The fine-tuning is implemented using the Transformers third-party library, and the number of iterations is set to 10,000 training steps. These multi-level and multi-faceted optimization measures aim to improve the model's understanding of different types of archives and the accuracy of information extraction. Through carefully designed dataset construction, selection of appropriate optimization algorithms, and reasonable setting of training parameters, not only is the model performance improved, but a solid foundation is also laid for subsequent intelligent applications.

[0091] Preferably, the keyword text location information is the position information of the keyword text box on the page; the keyword text content information is the text content information extracted from the corresponding keyword; and the keyword information finally determined for the entire archive is the core keyword name used for archiving and retrieval, which is finally confirmed manually after all processing steps have been completed.

[0092] Preferably, in step S1, the categories of different files to be processed include: personnel files, financial files, and engineering files; in other embodiments, the personnel files, financial files, and engineering files also include various documents to be processed. Specifically, for example, the various documents in the engineering files may include: state-owned land transfer contracts, construction land planning permits and their drawings, general construction contracts, supervision contracts, survey and design contracts, design and construction general contracting contracts, and administrative documents.

[0093] The following describes a method for extracting structured information from a multimodal large-scale archive by introducing the specific operation process:

[0094] The overall process flow of this embodiment is as follows: Figure 2 As shown, the training hardware used in this embodiment is Nvidia RTXA6000, the operating system is Ubuntu 22.04, and the programming language is Python 3.9.

[0095] Step 1: Classify the archive types. In actual engineering projects, archive types are classified into 35 major categories according to engineering construction standards, such as administrative documents, construction planning and land use permits, engineering planning permits, contract management, construction permits, engineering supervision, and completion acceptance filing. Each major category can be further divided into subcategories. For example, the construction planning and land use permit category can be further subdivided into subcategories such as state-owned land transfer contracts and land use planning permits. All major categories are divided into a total of 153 subcategories, with subcategories being the smallest unit for structured extraction.

[0096] Step 2: Establish a mapping table for fields to be extracted. Based on the subcategories defined in Step 1, define the keywords to be extracted for the corresponding file categories and construct the mapping table. The mapping table for the actual project is shown in Table 1.

[0097] Table 1. Mapping table of fields to be extracted for file types (partial)

[0098]

[0099]

[0100] Step 3: Vertical Industry Data Labeling: For the subcategories mentioned in Step 1, sample data with a minimum of 100 documents per category to ensure even distribution of the dataset. Label the sampled data according to the file type field mapping table defined in Step 2. However, in actual labeling, the above categories have similarities in the data. For example, the Chinese number in document archives is similar to the certificate number in certificate archives. Therefore, labeling can be done according to the standards for administrative documents, engineering certificates, contracts, and invoices to reduce the number of categories and improve the effect of subsequent layout analysis models. Taking the administrative document page as an example, the specific labeling content is as follows: Figure 3 As shown:

[0101] Step 4: Model training and fine-tuning. The layout analysis model uses the YOLOV8 detection model as the base model, and the document structure extraction and document structure integration models use the Qwen series models as the base models. The document structure extraction model is a visual multimodal model, and the document structure integration model is a large language model. The parameter size of both models is 7B. The base model in this step can be replaced according to the model effect and business scenario.

[0102] Among them, the page layout analysis model training

[0103] Using the vertical industry data annotated in step three, select the keyword text box position information and keyword names to build a training dataset for the document layout analysis model;

[0104] The training archive layout analysis model was performed over 100 epochs using the Adam optimizer with an initial learning rate of 0.01 that decayed during training.

[0105] Fine-tuning of the structured extraction model for archives

[0106] Using the industry data annotated in step three, select keywords to extract content information and keyword names, and establish a fine-tuning dataset for the archive structure extraction model;

[0107] Due to limitations in computing power hardware, the fine-tuning method in this embodiment is Quantized Low-Rank Adaptation (QLoRA). The fine-tuning directly uses the transformers third-party library API, and the number of fine-tuning iterations is 10,000 steps.

[0108] Fine-tuning of the structured integration model for archives;

[0109] Using the industry data annotated in step three, select keywords to extract content information and keyword names, as well as the keyword information determined in the final archiving of the entire document, and establish a fine-tuning dataset for the structured integration model of the archive;

[0110] Due to limitations in computing power hardware, the fine-tuning method in this embodiment is Quantized Low-Rank Adaptation (QLoRA). The fine-tuning directly uses the transformers third-party library API, and the number of fine-tuning iterations is 10,000 steps.

[0111] Step 5: Obtain the location information of basic elements in the archive image;

[0112] In the example of archival data format conversion, the archival data file is a digitized PDF file. First, the PyMuPDF third-party library is used to parse the data into images to obtain an archival data image list L(img1,img2,...,imgN).

[0113] Using the layout analysis model trained in step four, inference is performed on the images in the image list L sequentially. If the inference result contains keywords and the confidence level is greater than 0.6, it is determined that the image on that page contains key information, and the image on that page is stored in the key information page list G. The process is as follows. Figure 4 As shown, G(img1,img3,...,imgN) represents the (1,3,...,N)th page of the image list L containing key information. In actual project archives, there is a large amount of long text data, such as contract archives. The effective pages only account for 5% or even 1% of the total text pages. If all images are subjected to large model inference, the efficiency is extremely low and a lot of computing resources are wasted. Through this step, a detection algorithm with a smaller computational load is used to filter non-key information pages for subsequent large model extraction, which greatly shortens the time for the algorithm to extract the structure of multi-page archives.

[0114] Step Six: Extracting Structured Information from a Single Page;

[0115] Construct single-page structured information extraction prompts. These prompts must clearly describe the role of the large model in the task, the keywords to be extracted, the agreed-upon return format, and an example of the return format. In this embodiment, JSON Schema is used to define the data format returned by the large model, and an example of the returned JSON format is provided. Compared to directly using JSON to define the return data format, using JSON Schema offers higher stability. The single-page structured information extraction prompts in this embodiment are shown below. In the JSON Schema specification, ExtractResult and SingleExtractResult are predefined classes:

[0116] You are an image parser. Please extract the following keyword information from the input image. If the keyword does not exist in the image, the corresponding value is "" (empty value):

[0117] 1. Keywords to be extracted:

[0118] - Copy number

[0119] -Title

[0120] -Document Number

[0121] -Responsible person

[0122] -Date of writing

[0123] 2. The returned results must conform to the following JSON Schema specifications:

[0124] {"$defs":{"ExtractResult":{"additionalProperties":true,"properties":{},"title":"ExtractResult","type":"object"}},"properties":{"extract_res":{"items":{"$ref":"# / $defs / ExtractResult"},"title":"Extract Res","type":"array"}},"required":["extract_res"],"title":"SingleExtractResult","type":"object"}

[0125] 3. Example of specifying the returned JSON format:

[0126] {"extract_res":[{"Copy ID":"","Title":"","Document Number":"","Responsible Person":"","Date of Issuance":""}]}

[0127] The constructed model prompts and the list of key information images G(img1,img3,...,imgN) obtained in step five are used as inputs to the structured extraction model. All images in list G are traversed sequentially to extract key information from each page of images. After all images are extracted, a set of structured extraction results R(img1,img3,...,imgN) for each page is obtained.

[0128] Step 7: Integrate the extracted structured results from each page;

[0129] Based on the single-page structured extraction result set R(img1,img3,...,imgN) obtained in step six, the data in set R is first organized, and null values ​​are filtered out to extract the results. The page numbers and single-page extraction results are then used to construct a structured integration model prompt. The structured integration model prompt in this embodiment is shown below. In the JSON Schema specification, ExtractResult and SingleExtractResult are predefined classes.

[0130] You are a text analyzer. The following text information has been extracted from a PDF file and is arranged in page order. Please extract the following keywords from the input text. If a keyword does not exist, the value will be "".

[0131] The text information is as follows:

[0132] Page 1:

[0133] First page of extraction results:

[0134] Copy number:

[0135] Document Number: XXXX Document No.

[2025] 3

[0136] ...

[0137] Date of Issuance: April XX, 2025

[0138] Page 2:

[0139] Results extracted from page 2:

[0140] Copy number:

[0141] Document Number: XXXX

[0142] ...

[0143] Date of Issuance: March XX, 2024

[0144] ...

[0145] 1. Keywords to be extracted:

[0146] -File title

[0147] -Document Number

[0148] -Responsible person

[0149] -Date of writing

[0150] 2. The returned results must conform to the following JSON Schema specifications:

[0151] {"$defs":{"ExtractResult":{"additionalProperties":true,"properties":{},"title":"ExtractResult","type":"object"}},"properties":{"extract_res":{"items":{"$ref":"# / $defs / ExtractResult"},"title":"Extract Res","type":"array"}},"required":["extract_res"],"title":"SingleExtractResult","type":"object"}

[0152] 3. Example of specifying the returned JSON format:

[0153] {"extract_res":[{"Copy ID":"","Title":"","Document Number":"","Responsible Person":"","Date of Issuance":""}]}

[0154] The constructed prompts are input into the structured integration model to obtain the final extracted structured information from the archives. A flowchart illustrating the extraction and integration process is shown below. Figure 5 As shown;

[0155] Step 8: Store the integration results obtained in Step 7 into a local vector database for future business use. The encoding and storage process is as follows: Figure 6 As shown.

[0156] Vector encoding: The structured information extracted in step seven is encoded using the open-source vector encoding model BAAI GeneralEmbedding (BGE) to obtain the encoded result vector.

[0157] The obtained result vector is stored in a local vector database. In this embodiment, the vector database used is Milvus.

[0158] This application also provides a structured information extraction system based on multimodal large model archives, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a structured information extraction method based on multimodal large model archives as described in any of the above claims.

[0159] like Figure 7 As shown, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a method for extracting structured information from a multimodal large model archive as described in any of the preceding claims.

[0160] Figure 7 This is a schematic diagram of a hardware structure for running a method for extracting structured information from a multimodal large model archive, provided by an embodiment of the present invention. (See diagram for example.) Figure 7 As shown, this embodiment / computer 6 includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60, for example, a program for extracting structured information based on a multimodal large model archive. When the processor 60 executes the computer program 62, it implements the steps described in the embodiments of the multimodal large model archive structured information extraction method. Alternatively, when the processor 60 executes the computer program 62, it implements the functions of each module / unit in the aforementioned device embodiments.

[0161] For example, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 62 in the computer 6.

[0162] The computer 6 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. The computer 6 may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will understand that... Figure 7 This is merely an example of computer 6 and does not constitute a limitation on computer 6. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer 6 may also include input / output devices, network access devices, buses, etc.

[0163] The processor 60 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0164] The memory 61 can be an internal storage unit of the computer 6, such as a hard drive or memory. The memory 61 can also be an external storage device of the computer 6, such as a plug-in hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device. Furthermore, the memory 61 can include both internal storage units and external storage devices. The memory 61 is used to store the computer program and other programs and data required by the terminal device. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0165] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0166] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0167] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0168] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0169] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0170] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0171] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0172] The above are merely specific application examples of the present invention and do not constitute any limitation on the scope of protection of the present invention. In addition to the above embodiments, the present invention may have other implementations. All technical solutions formed by equivalent substitution or equivalent transformation fall within the scope of protection claimed by the present invention.

Claims

1. A method for extracting structured information from multimodal large-scale archives, characterized in that, Including the following steps: S1, Establish a field mapping table to be extracted: Based on the different categories of files to be processed, define the keywords to be extracted for the different files and construct a field mapping table to be extracted, wherein the field mapping table to be extracted includes: the mapping relationship between file category and keyword name to be extracted; S2, Data Labeling: Based on the different types of archive datasets to be processed, a set number of archives are sampled for each type. Keywords are extracted from the sampled data according to the field mapping table to be extracted and labeled to obtain the archive sample dataset. The labeling information includes keyword name, keyword text position and keyword text content. S3, Construct a layout analysis model, a structured extraction model, and a structured integration model: A layout analysis model is constructed based on the YOLO detection model to obtain layout information from archival image data; a structured extraction model is constructed based on the Qwen series visual multimodal models to extract information from the mapping table definition of the fields to be extracted from archival image data; a structured integration model is constructed based on the Qwen series large language models to integrate the structured information obtained from the structured extraction model. S4. Based on the layout analysis model, filter key information pages: Use the trained layout analysis model to perform layout analysis on the archive page image list L, filter out key information pages that contain keywords and have a confidence level greater than a set threshold, and store the key information pages in the key information page list G. S5, based on the archive structure extraction model, extracts structured information from a single page; including: S51, construct single-page structured information extraction keywords; wherein, the single-page structured information prompts include: role setting and task description: define the file structured extraction model as an image parser to extract keyword information, a list of keywords to be extracted, the format of the returned result, and an example of the returned format; S52, based on the single-page structured information extraction words constructed in step S51 and the key information page list G obtained in step S4, the archive structured extraction model is used to sequentially traverse all images in the list G, perform reasoning on each image, and extract information according to the single-page structured information extraction words constructed in step S51 to obtain single-page structured information; S53, After extracting all images using the archive structure extraction model, the extracted single-page structured information is combined into a result set R; S6, based on the archival structured integration model, integrates the results of single-page structured information, including: S61, constructing prompts for the structured integration model; wherein, the prompts for the structured integration model include: role setting and task description: defining the file structured integration model as a text analyzer to extract keyword information, text information sorted by page number, a list of keywords to be extracted, the format of the returned results, and an example of the returned format; S62, based on the prompt words set in step S61, the structured information extraction result of the archives is obtained using the structured integration model of the archives.

2. The method for extracting structured information from a multimodal large-scale archive according to claim 1, characterized in that, The process after step S6 also includes step S7, which involves vector encoding and storage of the extracted structured information from the archive: the structured information extracted in step S62 is encoded using the vector encoding model BAAI General Embedding to generate an encoded result vector, which is then stored in a vector database.

3. The method for extracting structured information from a multimodal large-scale archive according to claim 2, characterized in that, Between step S3 and step S4, the following steps are also included: step S31 model training and fine-tuning; based on the archive sample dataset in step S2, the layout analysis model, the archive structure extraction model, and the archive structure integration model are trained and fine-tuned.

4. The method for extracting structured information from a multimodal large-scale archive according to claim 3, characterized in that, Step S4 further includes: determining whether the data to be extracted is in image format: if the archive data is not image data, it is converted to image data before layout analysis is performed; if the archive data is image data, no conversion is required, and layout analysis is performed directly.

5. The method for extracting structured information from a multimodal large-scale archive according to claim 4, characterized in that, The set quantity in step S2 is that the minimum number of files in each category is not less than the set quantity.

6. The method for extracting structured information from a multimodal large-scale archive according to claim 5, characterized in that, Step S31 includes: S311, Page Layout Analysis Model Training: Based on the archive sample dataset in step S2, keyword names and keyword text location information are selected to construct an archive page layout analysis model training dataset. The page layout analysis model is trained using the archive page layout analysis model training dataset. The number of training iterations is set to 100 training rounds. The Adam optimizer is selected, and the initial learning rate is set to 0.01, which is then adjusted by decay during the training process. S312, Fine-tuning of the archive structure extraction model: Based on the archive sample dataset in step S2, select keyword names and keyword text content information to construct a fine-tuning dataset for the archive structure extraction model. Use the fine-tuning dataset for the archive structure extraction model to fine-tune the archive structure extraction model using the quantized low-rank adaptation method. The fine-tuning is implemented based on the transformers third-party library, and the number of iterations for fine-tuning is set to 10,000 training steps. S313, Fine-tuning of the structured integration model of archives: Based on the archive sample dataset in step S2, select keyword names, keyword text content information, and keyword information determined by the final archiving of the entire archive to construct a fine-tuning dataset for the structured integration model of archives. Using the fine-tuning dataset of the structured integration model of archives, fine-tune the structured integration model of archives through the quantized low-rank adaptation method. The fine-tuning is implemented based on the transformers third-party library, and the number of iterations for fine-tuning is set to 10,000 training steps.

7. The method for extracting structured information from a multimodal large-scale archive according to claim 6, characterized in that, The keyword text location information refers to the position of the keyword text box on the page; the keyword text content information refers to the text content extracted from the corresponding keyword; the keyword information for the final archiving of the entire document refers to the core keyword names used for archiving and retrieval, which are finally confirmed by humans after all processing steps have been completed.

8. The method for extracting structured information from a multimodal large-scale archive according to claim 1, characterized in that, In step S1, the categories of different files to be processed include personnel files, financial files, and engineering files.

9. A structured information extraction system based on a multimodal large-scale archive, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method for extracting structured information from a multimodal large model archive as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for extracting structured information from a multimodal large model archive as described in any one of claims 1 to 8.

Images