System and method for extracting project information based on multi-modal large model

Abstract

The invention discloses a system and a method for extracting project information based on a multi-modal large model, and the system comprises a file preprocessing module which is used for receiving and recognizing an input project document file, and extracting the metadata information of the file; the file type identification module is used for selecting different processing modes according to file types; the information extraction module is used for extracting text elements, labels, layer information and geometric elements from the document according to the file type and embedding the text elements, the labels, the layer information and the geometric elements into a vector space; the multi-modal model processing module is used for generating semantic description according to the text content, the picture description and the element metadata; the clustering module is used for clustering elements in the DWG drawing based on the coordinates; the knowledge graph construction module is used for constructing a knowledge graph based on the description, the text content and the element metadata; the vector database is used for storing embedded text content, description and related context information; the question analysis module is used for receiving user questions, extracting keywords, and retrieving in the vector database and the knowledge graph construction module to obtain related contexts; and the answer generation module is used for generating answers based on the retrieved context information in combination with a multi-modal large model and providing original text reference.

Description

Technical Field

[0001] The present invention relates to the technical field of engineering document processing, and in particular to a system and method for extracting project information based on a multimodal large model. Background Art

[0002] As the scale of engineering projects increases, various documents (such as DWG drawings, Word, PDF, etc.) and the graphics, text, annotations and other information contained therein become more and more complex.

[0003] Existing document information extraction methods are mostly manual processes, which are inefficient and prone to errors. Especially when dealing with different types of files, there are problems such as incomplete information extraction or inaccurate data.

[0004] With the advancement of technology, some people are beginning to use digitalization and large model-based processing to manage documents, such as:

[0005] Chinese patent application No. 202210155224.4 discloses a method and system for identifying key content of visa documents at power construction sites, wherein the identification method includes: obtaining visa documents and preprocessing the documents; identifying document text, form filling content and stamps in the image; classifying and arranging the identified document text, form filling content and stamps, and judging the integrity of the form filling content; and outputting the recognition results and the integrity judgment results.

[0006] Chinese patent application number CN202311786791.0 discloses an engineering project document management system based on digital twin technology, which is equipped with a standardization module, a collaborative design module, a results processing module and an analysis and display module. The data produced during the implementation of the engineering project is automatically archived and managed in a unified manner, and the results data can be published as maps and model services. The lossless display and multi-dimensional analysis in three-dimensional real scenes optimize the efficiency of engineering collaborative design, improve the quality of engineering project design, and also promote the accumulation of results data and professional field technology.

[0007] Chinese patent application number CN202410867247.7 discloses a document image restoration and multimodal large model training method and device, which involves artificial intelligence fields such as computer vision, deep learning and large models. The method may include: obtaining a target document image to be processed; using the target document image as the input of the multimodal large model to obtain an output first file, the first file is a hypertext markup language file that conforms to a predetermined representation format; generating a second file as a restoration result based on the first file, the second file is an editable file, and the second file includes the document content in the target document image. Summary of the invention

[0008] The purpose of the present invention is to provide a system and method for extracting project information based on a multimodal large model, which can automatically extract, analyze and output information from original engineering project documents based on the multimodal large model, and can automatically process various types of documents (such as DWG drawings, Word documents, PDF files, etc.), extract key information from the documents and generate a knowledge graph for users to query, and finally generate answers according to user needs and provide document source references, thereby helping users improve engineering project management efficiency.

[0009] To achieve the above object, the technical solution of the present invention is:

[0010] A system for extracting project information based on a multimodal large model, comprising:

[0011] The file preprocessing module receives and identifies the input engineering document files and extracts the metadata information of the files;

[0012] A file type identification module, which selects different processing methods according to the input file type; the file types include but are not limited to DWG drawings, Word documents, and PDF files;

[0013] The information extraction module extracts text elements, annotations, layer information, and geometric elements from documents according to the file type and embeds them into the vector space;

[0014] The multimodal model processing module generates semantic descriptions based on the extracted text content, image descriptions, metadata of engineering documents, and element attributes in DWG drawings;

[0015] Clustering module, clustering elements in DWG drawings based on coordinates;

[0016] The knowledge graph construction module builds the knowledge graph based on the generated description, text content and element metadata;

[0017] A vector database that stores embedded text content, descriptions, and their associated contextual information;

[0018] The question parsing module receives the user's question, extracts keywords, and retrieves relevant context in the vector database and knowledge graph construction module;

[0019] The answer generation module generates answers based on the retrieved context information and combines the multimodal large model, and provides original text citations.

[0020] Preferably, the information extraction module extracts the following information from the DWG drawing:

[0021] Layer information, including layer metadata;

[0022] Text elements and their coordinates, and embed the text content into vector space;

[0023] Annotation content and its coordinates;

[0024] The geometric elements are converted into pictures and are scaled and aligned to the model space of the drawing.

[0025] Preferably, the clustering module clusters text elements, annotations and geometric elements based on the coordinate positions of the elements in the DWG drawings, and generates descriptions using a multimodal model in combination with other attributes of the elements including layer, line type, user attributes and context, and embeds the descriptions into a vector space.

[0026] Preferably, the information extraction module processes the Word document and the PDF file as follows:

[0027] Text block processing, recording the position of each text block in the source document and embedding the text into vector space;

[0028] Generate descriptions of images in documents and combine them with text content to generate more detailed semantic descriptions, which are embedded in the vector space.

[0029] The present invention also provides a method for extracting project information based on a multimodal large model, which comprises the following steps:

[0030] a) File preprocessing: identifying the input file type and extracting metadata information;

[0031] b) Extract information based on file type, including extracting layer information, text content and coordinates, annotations and geometric elements from DWG drawings and embedding them into vector space; for Word or PDF documents, extract text in blocks and generate image descriptions;

[0032] c) Clustering based on coordinates and generating semantic descriptions for element sets in DWG drawings;

[0033] d) Build a knowledge graph based on text content, description and metadata;

[0034] e) Based on the questions input by the user, the user question is parsed, including intent recognition, summary generation, and analysis of the solution method steps. The reference context is retrieved based on the vector database, and the relevant node information is retrieved based on the knowledge graph.

[0035] f) Combine the multimodal large model to generate answers and provide original location references.

[0036] Preferably, the file preprocessing in step a) includes extracting metadata information of the input file, and the metadata information includes the creation time, modification time, and author of the file.

[0037] Preferably, the step e) question parsing includes extracting keywords from the user's question and searching the vector database and the knowledge graph simultaneously to obtain reference context and related node information.

[0038] Preferably, the step f) generating answers includes generating multiple questions according to the input template, processing them separately to obtain answers, and formatting and outputting the generated content through a multimodal large model to provide final information output and original text citations.

[0039] In the system and method of the present invention:

[0040] 1. File preprocessing and metadata information extraction

[0041] The system first pre-processes the input files, automatically identifies the file type (such as DWG drawings, Word, PDF, etc.), and extracts the file's metadata information (such as file creation time, modification time, author, etc.). These metadata provide necessary context information for subsequent processing and are stored in the system's metadata repository.

[0042] 2. Extract and store files according to input file type

[0043] The system uses different extraction methods depending on the type of input file:

[0044] DWG drawings: By parsing DWG files, extract layer information, text elements and their coordinates, annotation information and their positions, convert geometric elements into images, and scale and align them with the scale of the model space.

[0045] Word and PDF documents: The text in the document is processed in blocks, the location of the text in the source document is recorded, and the text is embedded in the vector space. At the same time, for the pictures in the document, the system generates a preliminary description through a multimodal model, aligns it with the context semantics to generate a more detailed description, and finally embeds the description into the vector space.

[0046] 3. For DWG drawing information extraction, the system specifically performs the following steps:

[0047] Extract layer information and record metadata of different layers;

[0048] Extract text elements and their coordinates, and embed the text content into vector space;

[0049] Extract annotation content and its coordinates, and cluster them together with text elements based on the coordinates;

[0050] Convert geometric elements into images and scale them according to the drawing scale to align them with the model space;

[0051] Cluster the elements in the drawings based on coordinate and text information, and generate descriptions for each clustered set of elements using a multimodal large model;

[0052] The metadata and annotation information of descriptions and text elements are embedded into the vector space, and a knowledge graph is constructed based on it.

[0053] 4. Information extraction and description generation of documents such as Word and PDF

[0054] For Word and PDF documents, the system divides the document text into blocks, records the position of each text block in the original document, and embeds the text blocks into the vector space;

[0055] For images in documents, the system first generates a preliminary description and aligns it with the contextual semantics of the document text, then generates a more accurate image description based on the semantics and embeds the description into the vector space.

[0056] Finally, based on the document text and description, the system builds a knowledge graph, representing different text blocks, image descriptions and their relationships in the graph.

[0057] 5. Retrieval and answer generation based on user questions

[0058] After the user enters a question, the system first searches the question based on the vector database to obtain the context x related to the question;

[0059] At the same time, the system extracts the relevant node information of the question from the knowledge graph and retrieves the context y related to the node information in the vector database;

[0060] Based on the multimodal large model, the context x and y are combined to generate the answer to the question, and the location of the answer content in the original document is provided as a reference.

[0061] 6. Multiple question generation and final information output

[0062] When the question entered by the user involves multiple pieces of information, the system automatically generates multiple sub-questions based on the input template, and processes each sub-question separately to generate the corresponding content and original text citation;

[0063] Finally, the system formats and outputs the generated content based on the Large Language Model (LLM), and gives the final output results and related original text citation information.

[0064] Beneficial effects of the present invention:

[0065] The present invention uses a multimodal large model to extract, analyze and output information from original engineering project documents, automatically process various types of documents (such as DWG drawings, Word documents, PDF files, etc.), extract key information and build a knowledge map for users to query; at the same time, the system can efficiently generate accurate answers based on user questions and provide document source references. This greatly improves the efficiency of engineering project document management, reduces manual operation errors, and has high practical value.

[0066] In particular, the present invention focuses on the extraction and application of engineering project design data information. Compared with other existing information extraction technologies based on large language models, the present invention not only uses text and images in file types such as word, text, and pdf as information extraction sources, but also further uses dwg files and their apis commonly used in engineering project design as information extraction sources. It can effectively extract engineering entity design information in engineering projects, thereby achieving the effect of obtaining more accurate, complete, and valuable engineering information of the project. The efficiency of the present invention in the application of information acquisition in the engineering project industry is incomparable with other technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is a flow chart of the method for extracting project information based on a multimodal large model described in the present invention. DETAILED DESCRIPTION

[0068] The present invention will be further described below in conjunction with the embodiments and drawings.

[0069] The system for extracting project information based on a multimodal large model according to the present invention comprises:

[0070] The file preprocessing module receives and identifies the input engineering document files and extracts the metadata information of the files;

[0071] A file type identification module, which selects different processing methods according to the input file type; the file types include but are not limited to DWG drawings, Word documents, and PDF files;

[0072] The information extraction module extracts text elements, annotations, layer information, and geometric elements from documents according to the file type and embeds them into the vector space;

[0073] The multimodal model processing module generates semantic descriptions based on the extracted text content, image descriptions, metadata of engineering documents, and element attributes in DWG drawings;

[0074] Clustering module, clustering elements in DWG drawings based on coordinates;

[0075] The knowledge graph construction module builds the knowledge graph based on the generated description, text content and element metadata;

[0076] A vector database that stores embedded text content, descriptions, and their associated contextual information;

[0077] The question parsing module receives the user's question, extracts keywords, and retrieves relevant context in the vector database and knowledge graph construction module;

[0078] The answer generation module generates answers based on the retrieved context information and combines the multimodal large model, and provides original text citations.

[0079] Preferably, the information extraction module extracts the following information from the DWG drawing:

[0080] Layer information, including layer metadata;

[0081] Text elements and their coordinates, and embed the text content into vector space;

[0082] Annotation content and its coordinates;

[0083] The geometric elements are converted into pictures and are scaled and aligned to the model space of the drawing.

[0084] Preferably, the clustering module clusters text elements, annotations and geometric elements based on the coordinate positions of the elements in the DWG drawings, and generates descriptions using a multimodal model in combination with other attributes of the elements including layer, line type, user attributes and context, and embeds the descriptions into a vector space.

[0085] Preferably, the information extraction module processes the Word document and the PDF file as follows:

[0086] Text block processing, recording the position of each text block in the source document and embedding the text into vector space;

[0087] Generate descriptions of images in documents and combine them with text content to generate more detailed semantic descriptions, which are embedded in the vector space.

[0088] See also Figure 1 The present invention also provides a method for extracting project information based on a multimodal large model, which comprises the following steps:

[0089] a) File preprocessing: identifying the input file type and extracting metadata information;

[0090] b) Extract information based on file type, including extracting layer information, text content and coordinates, annotations and geometric elements from DWG drawings and embedding them into vector space; for Word or PDF documents, extract text in blocks and generate image descriptions;

[0091] c) Clustering based on coordinates and generating semantic descriptions for element sets in DWG drawings;

[0092] d) Build a knowledge graph based on text content, description and metadata;

[0093] e) Based on the questions input by the user, the user question is parsed, including intent recognition, summary generation, and analysis of the solution method steps. The reference context is retrieved based on the vector database, and the relevant node information is retrieved based on the knowledge graph.

[0094] f) Combine the multimodal large model to generate answers and provide original location references.

[0095] Preferably, the file preprocessing in step a) includes extracting metadata information of the input file, and the metadata information includes the creation time, modification time, and author of the file.

[0096] Preferably, the step e) question parsing includes extracting keywords from the user's question and searching the vector database and the knowledge graph simultaneously to obtain reference context and related node information.

[0097] Preferably, the step f) generating answers includes generating multiple questions according to the input template, processing them separately to obtain answers, and formatting and outputting the generated content through a multimodal large model to provide final information output and original text citations.

[0098] Example 1

[0099] Processing of DWG drawings

[0100] The system receives a DWG drawing file, first extracts the layers, text and annotation information, and converts the geometric elements into images; then, the system generates a detailed description of each layer and annotation based on the multimodal model, builds a knowledge graph, and embeds the description and drawing related information into a vector database; users can enter query questions, and the system generates answers based on the knowledge graph and vector database, and gives the reference location of the drawing.

[0101] Example 2

[0102] Word document processing

[0103] For an input Word document, the system first divides the text into blocks and embeds it into a vector space, then extracts images and generates semantic descriptions; by building a knowledge graph, the system can process user questions and provide references to precise locations in the document.

[0104] Through the above examples, the present invention realizes the comprehensive extraction and intelligent processing of multimodal project document information, which can greatly improve the efficiency and accuracy of engineering document management.

Claims

1. A system for extracting project information based on a multimodal large model, characterized in that: include: The file preprocessing module receives and identifies the input engineering document files and extracts the metadata information of the files; A file type identification module, which selects different processing methods according to the input file type; the file types include but are not limited to DWG drawings, Word documents, and PDF files; The information extraction module extracts text elements, annotations, layer information, and geometric elements from documents according to the file type and embeds them into the vector space; The multimodal model processing module generates semantic descriptions based on the extracted text content, image descriptions, metadata of engineering documents, and element attributes in DWG drawings; Clustering module, clustering elements in DWG drawings based on coordinates; The knowledge graph construction module builds the knowledge graph based on the generated description, text content and element metadata; A vector database that stores embedded text content, descriptions, and their associated contextual information; The question parsing module receives the user's question, extracts keywords, and retrieves relevant context in the vector database and knowledge graph construction module; The answer generation module generates answers based on the retrieved context information and combines the multimodal large model, and provides original text citations.

2. The system according to claim 1, characterized in that The information extraction module extracts the following information from the DWG drawing: Layer information, including layer metadata; Text elements and their coordinates, and embed the text content into vector space; Annotation content and its coordinates; The geometric elements are converted into pictures and are scaled and aligned to the model space of the drawing.

3. The system according to claim 1, characterized in that The clustering module clusters text elements, annotations and geometric elements based on the coordinate positions of elements in the DWG drawings, and generates descriptions using a multimodal model in combination with other attributes of the elements including layer, line type, user attributes and context, and embeds the descriptions into a vector space.

4. The system according to claim 1 or 2, characterized in that: The information extraction module processes the Word document and PDF file as follows: Text block processing, recording the position of each text block in the source document and embedding the text into vector space; Generate descriptions of images in documents and combine them with text content to generate more detailed semantic descriptions, which are embedded in the vector space.

5. A method for using the system for extracting project information based on a multimodal large model as claimed in any one of claims 1 to 4, characterized in that: The following steps are involved: a) File preprocessing: identifying the input file type and extracting metadata information; b) Extract information based on file type, including extracting layer information, text content and coordinates, annotations and geometric elements from DWG drawings and embedding them into vector space; for Word or PDF documents, extract text in blocks and generate image descriptions; c) Clustering based on coordinates and generating semantic descriptions for element sets in DWG drawings; d) Build a knowledge graph based on text content, description and metadata; e) Based on the questions input by the user, the user question is parsed, including intent recognition, summary generation, and analysis of the solution method steps. The reference context is retrieved based on the vector database, and the relevant node information is retrieved based on the knowledge graph. f) Combine the multimodal large model to generate answers and provide original location references.

6. The method according to claim 5, characterized in that: The file preprocessing in step a) includes extracting metadata information of the input file, and the metadata information includes the creation time, modification time, and author of the file.

7. The method according to claim 5, characterized in that: The step e) question parsing includes extracting keywords of the user's question and searching the vector database and the knowledge graph simultaneously to obtain reference context and related node information.

8. The method according to claim 5, characterized in that: The step f) generating answers includes generating multiple questions according to the input template, processing them separately to obtain answers, and formatting and outputting the generated content through a multimodal large model to provide final information output and original text citations.

Images