Document metadata marking and auxiliary retrieval method and system based on large language model

Abstract

The invention provides a document metadata marking and auxiliary retrieval method and system based on a large language model, aiming at the problems of multi-source isomerism, low retrieval precision, complicated maintenance, high deployment cost and the like of documents in the field of tunnel engineering. According to the method, through multivariate data analysis and content extraction, format specificity processing is carried out on documents of various formats, text content or non-text element description information is extracted, and a content abstract is formed through combination; intelligent metadata generation: calling a locally deployed tunnel field vertical large model, and generating structured metadata containing professional fields, document types, core keywords and content abstracts; metadata embedding and index construction: writing metadata into a file extension attribute or an affiliated file in a JSON format, and constructing a centralized metadatabase; and semantic retrieval is matched with user query and metadata, and a related document list is returned. According to the method, the professional document retrieval precision and robustness are improved, the maintenance and deployment cost is reduced, and efficient management of multi-element engineering documents is realized.

Description

Technical Field

[0001] This invention belongs to the field of knowledge graphs and information retrieval, and relates to a document metadata tagging and retrieval method based on a large language model. Background Technology

[0002] In the field of tunnel and underground engineering, projects generate massive amounts of multi-source, heterogeneous engineering data, including construction logs, various reports, CAD drawings, meeting minutes, research reports, technical guidelines, and documents in dozens of formats. This data is highly unstructured and specialized, making effective management and retrieval a long-standing pain point in the industry.

[0003] Existing technologies generally employ vector embedding-based retrieval schemes (such as those based on BERT models), converting document content into high-dimensional vectors for similarity matching. However, these methods have significant drawbacks: (1) The tagging information is not intuitive and the management and maintenance are complicated: the vectors are not readable by humans and must rely on an independent vector database. The correspondence between the database and the original file is extremely fragile. Once the file is moved or renamed, the entire index becomes invalid and the maintenance cost is high.

[0004] (2) Insufficient domain expertise: The general model has a superficial understanding of professional terms and concepts such as "shield segment assembly" and "surrounding rock grade determination", and the generated vector representation cannot accurately reflect the engineering semantics, resulting in low retrieval accuracy.

[0005] (3) Weak support for diverse data: Traditional solutions are unable to effectively process non-text or non-standard format files such as CAD drawings and complex engineering reports, and often can only ignore or process them simply, resulting in the loss of a large amount of key engineering information.

[0006] (4) High computational resource requirements: Generating high-quality vectors requires large-scale models, which are costly to compute and difficult to deploy on ordinary computers at the project site. Summary of the Invention

[0007] To address the problems existing in the background technology, this invention proposes a document metadata tagging and auxiliary retrieval method and system based on a large language model.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: On the one hand, this invention provides a document metadata tagging and assisted retrieval method based on a large language model, including: Multivariate data parsing and content extraction: Format-specific processing is performed on each document format of the original data to extract text content and descriptive information of non-text elements, and combine them into a content summary. Intelligent metadata generation involves calling a large vertical model in the tunnel field to analyze the content summary and generate standardized structured metadata. Metadata embedding and index building involves writing the generated metadata into extended attributes or supplementary files of the original file and building a centralized metadata database. Semantic retrieval matches the similarity between the user's query and the metadata in the metadata database, and returns a list of relevant documents.

[0009] Furthermore, the multi-source data parsing and content extraction steps include: Extract all text content and table data from text-based documents; Extract file path, file name, and meta-attributes from non-text files, and generate descriptive text based on the file name.

[0010] Furthermore, in the intelligent metadata generation step, the structured metadata includes at least: The professional field to which the document belongs; Document type; For two or more core keywords, priority should be given to selecting them from the predefined tunnel engineering keyword library; A summary of no more than 150 words.

[0011] Furthermore, in the metadata embedding and index building step, the metadata is in JSON format and is stored by writing it into the extended attributes of the original file and / or storing it in an auxiliary .meta file with the same name as the original file.

[0012] Furthermore, the semantic retrieval step employs the BM25 algorithm to perform similarity matching between the query statement and the summary and keyword fields in the metadata database.

[0013] Furthermore, the vertical large model in the tunnel domain is a model obtained by filtering out unnecessary datasets from existing large models and correcting them on a tunnel engineering corpus, and then deploying and inferring it locally on a personal computer.

[0014] Furthermore, the method supports input document formats including at least three of the following: Word, PDF, PPT, Excel, image, CAD file, and text file.

[0015] On the other hand, the present invention also provides a document metadata tagging and auxiliary retrieval system based on a large language model for implementing the above method, comprising: The multi-source data parsing and content extraction module is configured to perform format-specific processing on engineering documents of various formats. The intelligent metadata generation module is configured to call a large vertical model in the tunnel domain to generate structured metadata; The metadata embedding and indexing building module is configured to bind metadata to documents and build a metadata database; The semantic retrieval module is configured to perform similarity matching based on user queries and return retrieval results.

[0016] Compared with the prior art, the present invention has the following beneficial effects: (1) Strong domain specialization and high labeling accuracy: The vertical large model has a deep understanding of professional terms and engineering scenarios, and the generated metadata (such as keywords and abstracts) has an accuracy rate of over 95%, which greatly improves the retrieval accuracy.

[0017] (2) Powerful support for diverse data: Through the pragmatic strategy of "path preservation and description generation", the system can generate valuable metadata for non-text files such as CAD drawings and engineering pictures, and realize "full coverage" management of engineering data.

[0018] (3) The system is robust and has extremely low maintenance costs: metadata is physically bound to the file. After the file is moved or copied, the metadata is not lost. The system only needs to rescan to rebuild the index, reducing maintenance difficulty and cost by more than 80%.

[0019] (4) The deployment threshold and usage cost are extremely low: small-scale vertical models can be deployed on personal computers without expensive hardware or cloud APIs, and the overall cost is reduced by an order of magnitude compared with traditional solutions.

[0020] (5) The search results are intuitive and reliable: the returned metadata is human-readable rather than incomprehensible vectors, which users can quickly verify and judge, making decision-making efficient. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram of multivariate data parsing in this invention; Figure 3 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] On the one hand, such as Figure 1 As shown, this invention provides a document metadata tagging and assisted retrieval method based on a large language model, including: Multivariate data parsing and content extraction: Format-specific processing is performed on each document format of the original data to extract text content and descriptive information of non-text elements, and combine them into a content summary. Intelligent metadata generation involves calling a large vertical model in the tunnel field to analyze the content summary and generate standardized structured metadata. Metadata embedding and index building involves writing the generated metadata into extended attributes or supplementary files of the original file and building a centralized metadata database. Semantic retrieval matches the similarity between the user's query and the metadata in the metadata database, and returns a list of relevant documents.

[0024] This embodiment is based on a personal workstation equipped with an Intel i7-13700k processor, 64GB of memory, and an NVIDIA RTX 4060 graphics card.

[0025] Five hundred documents from a tunnel project, including construction logs (Word), design drawings (CAD), monitoring reports (Excel), technical briefing PPTs, on-site photos (JPG), and comprehensive documents (pdf), were placed in the monitoring directory.

[0026] like Figure 2 As shown, each format of the original data is processed specifically to extract text content and descriptive information of non-text elements, and then combined to form a content summary. Extract all text content and tabular data from text documents including construction logs, monitoring reports, and technical briefing presentations. Extract file paths, file names, and meta-attributes from non-text documents including design drawings and site photos, and generate descriptive text based on file names.

[0027] The system invokes a large-scale vertical model for the tunnel domain to automatically process all files, analyze the content summary, and generate standardized structured metadata. This large-scale vertical model for the tunnel domain is a model obtained by filtering out unnecessary datasets from existing large-scale models and refining it on a tunnel engineering corpus. It is then deployed and inferred locally on a personal computer.

[0028] For CAD files, the "filename.dwg" part and creation date are extracted and combined to form the descriptive text "Tunnel Support Design CAD Drawings". For Word and PDF files, all text is extracted. Then, a large vertical model of the tunnel domain is invoked to generate metadata for each file. For example, for a construction log, the following is generated: {"domain": "Tunnel Construction", "doc_type": "Construction Log", "keywords": ["Shield Tunneling", "Segment Assembly", "Synchronous Grouting"], "summary": "This log records the shield tunneling of section XX...}". The metadata is stored by writing it to the extended attributes of the original file and / or storing it in an attached .meta file with the same name as the original file. All metadata is written to the extended attributes of the corresponding files, and the processed files are archived. The system traverses the archive directory to build a metadata database.

[0029] The system performs similarity matching between the user's query and the metadata in the metadata database, returning a list of relevant documents. For example, a user queries "grouting pressure control standards during tunnel boring machine (TBM) advancement." The system performs BM25 matching on the summary and keyword fields of the metadata database, successfully returning relevant construction logs, technical specifications, and meeting minutes, whose metadata clearly shows a high degree of relevance to the query.

[0030] Compared with traditional vector retrieval systems, this invention improves the accuracy of retrieving professional questions by more than 30%, and after simulating batch file movement, the index recovery time is shortened from several hours (traditional methods require regenerating vectors) to a few minutes, demonstrating excellent practicality and maintainability.

[0031] On the other hand, such as Figure 3 As shown, the present invention also provides a document metadata tagging and auxiliary retrieval system based on a large language model to implement the above method, comprising: The multi-source data parsing and content extraction module is configured to perform format-specific processing on engineering documents of various formats. The intelligent metadata generation module is configured to call a large vertical model in the tunnel domain to generate structured metadata; The metadata embedding and indexing building module is configured to bind metadata to documents and build a metadata database; The semantic retrieval module is configured to perform similarity matching based on user queries and return retrieval results.

[0032] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A document metadata tagging and assisted retrieval method based on a large language model, characterized in that, include: Multivariate data parsing and content extraction: Format-specific processing is performed on each document format of the original data to extract text content and descriptive information of non-text elements, and combine them into a content summary. Intelligent metadata generation involves calling a large vertical model in the tunnel field to analyze the content summary and generate standardized structured metadata. Metadata embedding and index building involves writing the generated metadata into extended attributes or supplementary files of the original file and building a centralized metadata database. Semantic retrieval matches the similarity between the user's query and the metadata in the metadata database, and returns a list of relevant documents.

2. The method according to claim 1, characterized in that, The multi-source data parsing and content extraction steps include: Extract all text content and table data from text-based documents; Extract file path, file name, and meta-attributes from non-text files, and generate descriptive text based on the file name.

3. The method according to claim 1, characterized in that, In the intelligent metadata generation step, the structured metadata includes at least: The professional field to which the document belongs; Document type; For two or more core keywords, priority should be given to selecting them from the predefined tunnel engineering keyword library; A summary of no more than 150 words.

4. The method according to claim 1, characterized in that, In the metadata embedding and index building step, the metadata is in JSON format and is stored by writing it into the extended attributes of the original file and / or storing it in an auxiliary .meta file with the same name as the original file.

5. The method according to claim 1, characterized in that, The semantic retrieval step uses the BM25 algorithm to match the query statement with the summary and keyword fields in the metadata database based on similarity.

6. The method according to claim 1, characterized in that, The vertical large model in the tunnel domain is a model obtained by filtering out unnecessary datasets from existing large models and correcting them on a tunnel engineering corpus. It is then deployed and inferred locally on a personal computer.

7. The method according to claim 1, characterized in that, The method supports input document formats including at least three of the following: Word, PDF, PPT, Excel, image, CAD file, and text file.

8. A document metadata tagging and auxiliary retrieval system based on a large language model, implementing the method of any one of claims 1-7, characterized in that, include: The multi-source data parsing and content extraction module is configured to perform format-specific processing on engineering documents of various formats. The intelligent metadata generation module is configured to call a large vertical model in the tunnel domain to generate structured metadata; The metadata embedding and indexing building module is configured to bind metadata to documents and build a metadata database; The semantic retrieval module is configured to perform similarity matching based on user queries and return retrieval results.

Images