Enterprise multi-level multi-modal document question answering method based on large language model

Abstract

This application provides a multi-level, multimodal document question answering method for enterprises based on a large language model, relating to the fields of artificial intelligence and knowledge graph technology. It solves the technical problems of existing document question answering methods, such as document semantic fragmentation, missing multimodal information, weak query recall capabilities, and low answer credibility. The method includes: constructing a tree-shaped document graph and an enterprise knowledge graph based on multiple source documents within the enterprise; performing intent recognition and structured parsing on received user queries, and performing multi-path recall based on the constructed semantic vector index, keyword inverted index, and graph index to obtain candidate leaf nodes; sorting the candidate leaf nodes using a re-ranking model, selecting the top K leaf nodes with the highest relevance to obtain the selected node; and constructing a prompt message by combining the user query, the text content corresponding to the selected node, and the multimodal content bound to the selected node, inputting it into the large language model to generate the question answering result.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and knowledge graph technology, specifically a multi-level, multimodal document question answering method for enterprises based on a large language model. Background Technology

[0002] As enterprises deepen their informatization and digitalization processes, a large number of documents, including technical manuals, operating procedures, troubleshooting guides, and standards, are accumulated in electronic form, constituting the core knowledge assets of enterprises. These documents are typically characterized by complex hierarchical structures, containing a large amount of multimodal information such as images and tables, and are numerous and frequently updated. To efficiently utilize this knowledge, existing technologies usually employ question-answering schemes based on Retrieval-Augmented Generation (RAG). The general process of this type of scheme is as follows: first, the enterprise documents are segmented into fixed-length blocks or chunks based on simple rules; then, semantic similarity retrieval is performed using a vector database, or combined with keyword retrieval to recall text fragments that may be related to the user's question; finally, these retrieved text fragments are used as context and input into a large language model to generate the final answer.

[0003] However, the existing methods mentioned above have significant shortcomings when dealing with complex enterprise-level documents, resulting in poor question-answering performance. Specifically, during the document processing stage, fixed or simple rule-based block segmentation crudely fragments the original logic and hierarchical structure of the document. For example, a complete business semantic unit of "fault phenomenon - cause analysis - handling steps" may be split into different text blocks. This leads to the retrieved fragments often being incomplete and fragmented in context, failing to provide a complete and coherent decision-making basis for the large language model. Second, existing solutions are mainly geared towards plain text and lack effective perception and utilization of multimodal content such as images, tables, and diagrams in documents. In technical documents, key information is often presented through charts and graphs, but traditional methods cannot automatically associate and return this visual information during the question-answering process, seriously affecting the comprehensibility and practical operational guidance value of the generated answers. In addition, during the retrieval stage, relying solely on semantic vector similarity or keyword matching has weak recall capabilities for complex questions containing low-frequency technical terms, niche device models, or requiring multi-hop reasoning. For example, querying "the alarm handling process of a certain component of a certain model of equipment under specific operating conditions" involves multiple constraints of entities, attributes, and states. A single search method is prone to omissions or introducing a lot of noise. Furthermore, in the answer generation stage, the general-purpose large language model, without explicit guidance on the enterprise's unique knowledge structure and business rules, generates answers based solely on potentially incomplete search fragments. This can easily lead to "illusions," i.e., fabricated factual information, and the answers cannot be traced back to specific document sources. It is difficult to meet the stringent requirements for accuracy, credibility, and traceability of answers in enterprise operation and maintenance, fault handling, and other scenarios.

[0004] Therefore, existing technical solutions face challenges in areas such as structured document understanding, multimodal information fusion, accurate retrieval of complex queries, and assurance of factual accuracy of answers. There is an urgent need for an enterprise-level intelligent question-answering method that can systematically solve these problems. Summary of the Invention

[0005] This application provides a multi-level, multimodal document question-answering method for enterprises based on a large language model. It solves the technical problems in the prior art, such as the fragmentation of the original semantic units of the document due to fixed-length blocks, the difficulty in effectively retrieving and utilizing the multimodal content in the document, the insufficient recall ability for low-frequency professional terms and complex multi-hop relationship questions, and the tendency to produce factual errors or untraceable answers due to the lack of explicit modeling of the enterprise's internal knowledge structure.

[0006] To achieve the above objectives, this application adopts the following technical solution: Firstly, a multi-level, multimodal document question-answering method based on a large language model is provided, including: Collect and clean multi-source documents from within the enterprise, perform hierarchical structure parsing on the cleaned documents, and construct a tree-structured document graph based on the parsed title hierarchy and the multimodal content extracted from the documents; where each leaf node of the tree-structured document graph corresponds to a document fragment; Information is extracted from each leaf node in the tree-shaped document graph, and an enterprise knowledge graph is constructed based on the extracted entities and the relationships between them. The received user queries are subjected to intent recognition and structured parsing, and multiple candidate leaf nodes are obtained by multi-way recall based on the semantic vector index, keyword inverted index and graph index constructed based on the tree document graph and the enterprise knowledge graph. The multiple candidate leaf nodes are sorted using a reordering model, and the top K leaf nodes with the highest relevance are selected to obtain the selected node. The user query, the text content corresponding to the selected node, and the multimodal content bound to the selected node are combined to construct a prompt message, which is then input into the large language model to generate a question-and-answer result that includes the source of the reference and multimodal display information.

[0007] Based on the above technical solutions, this application provides a method for enterprise multi-level multimodal document question answering based on a large language model. By constructing a tree-shaped document graph, the inherent hierarchical structure and semantic units of enterprise documents are effectively preserved, providing a foundation for solving the semantic fragmentation problem caused by traditional fixed-length block segmentation. By binding multimodal content with tree nodes, unified processing of text and related diagrams is ensured in the retrieval and generation stages, creating conditions for outputting actionable, combined text and image answers. Furthermore, this method integrates three types of indexes—semantic vectors, keywords, and graphs—for multi-path retrieval, enabling the capture of user query intent from different dimensions. It takes into account semantic similarity, precise term matching, and deep logical connections, thereby improving the recall and coverage of complex and professional queries. Finally, by integrating multi-path retrieval results and optimizing them using a re-ranking model, and then combining selected content with a large language model to generate cited answers, a complete closed loop is formed, from accurate positioning and intelligent synthesis in massive heterogeneous documents to traceable presentation. Overall, this method systematically addresses a series of key challenges in enterprise-level document question answering, such as structured understanding, multimodal fusion, accurate retrieval, and reliable generation, thereby improving the accuracy, usability, and interpretability of question answering systems.

[0008] Furthermore, the process of collecting and cleaning multi-source documents within the enterprise, and parsing the hierarchical structure of the cleaned documents, includes: Collect multi-source heterogeneous documents from different systems within the enterprise; the multi-source heterogeneous documents include, but are not limited to, technical manuals, operating procedures, troubleshooting tools, and standard specifications, and the formats of the multi-source heterogeneous documents include, but are not limited to, PDF, Word, PPT, or scanned copies. The multi-source heterogeneous documents are subjected to unified format conversion, encoding standardization, layout cleanup, and duplicate content detection to obtain multiple cleaned documents. The encoding standardization refers to converting the character encoding of all documents into a unified format. The layout cleanup refers to removing page headers, footers, watermarks, and annotations from the documents using a rule-based repetitive expression matching method. The duplicate content detection refers to marking segments with similarity scores higher than a preset threshold as duplicate content by calculating the text similarity or semantic vector similarity between document segments. The cleaned documents are subjected to directory parsing, title detection, and paragraph segmentation to identify the directory structure, font size, numbering style, and spacing features before and after paragraphs for each document. The directory parsing is achieved by parsing the document's bookmarks or style structure. The title detection is based on a heuristic rule method, which includes identifying font size, font style, layout position, and numbering style. The paragraph segmentation is achieved by analyzing the document's line breaks, indentation, and line spacing features. Using the directory structure, font size, numbering style, and spacing features before and after paragraphs, a multi-feature fusion hierarchical partitioning algorithm is used to determine the hierarchical structure of the cleaned documents, thereby obtaining the title level of each document. The multi-feature fusion hierarchical partitioning algorithm is a rule-based decision tree or scoring model, which determines the title level of each text paragraph based on its font size, numbering depth, level in the directory, and spacing with context.

[0009] Furthermore, the workflow of the multi-feature fusion hierarchical partitioning algorithm is as follows: For each text paragraph in the cleaned document, font style features, sequence number depth features, directory structure anchoring features, and layout spacing features are extracted, and these features are quantified into numerical representations. Specifically, the font style features are quantified based on the font size and whether the text is bold; the sequence number depth features are determined by matching the numbering pattern at the beginning of the paragraph using regular expressions; the directory structure anchoring features are obtained by matching the paragraph text with parsed directory entries to obtain a preset level; and the layout spacing features are extracted by extracting the physical unit values ​​of the spacing before and after the paragraph. The quantized feature values ​​of each text paragraph are input into a predefined hierarchical determination model to calculate the comprehensive hierarchical score of the paragraph. Based on the comprehensive hierarchical score, a preset hierarchical threshold range is matched to preliminarily determine the title level to which the text paragraph belongs, thus obtaining the preliminary determination level. The hierarchical determination model adopts a weighted scoring model or a rule-based branching decision tree model, wherein the weights assigned to the directory structure anchoring feature and the sequence number depth feature are higher than the weights assigned to the font style feature and the layout spacing feature. Based on the physical order of the document, a global consistency check is performed on the initial determination level of all text paragraphs. When a level jump that does not conform to the document logic is detected, the determination level of the current paragraph is adjusted according to the level characteristics of the adjacent paragraphs, and finally the document title level with the correct structure is output.

[0010] Furthermore, the step of constructing a tree-structured document graph based on the parsed heading levels and the multimodal content extracted from the document includes: Based on the title hierarchy, numbering style, and directory structure, the titles at each level and their subordinate paragraphs are organized into a tree-like document diagram structure containing chapters, sections, subsections, and entries, and the tree-like document diagram is stored as a hierarchical JSON document; The document page is parsed using computer vision and object detection technologies to identify multimodal objects in the document. The type, page number, and bounding box coordinates of each modal object within the page number are extracted to obtain multimodal content. The multimodal objects include at least: text blocks, images, tables, and diagrams. Based on the page number of each modal object, determine all sections that are on the same page or adjacent to the modal object as candidate sections; Based on the multimodal content, a comprehensive binding score is calculated for each modal object and the candidate section. This comprehensive binding score is obtained by weighted summation of page number consistency score, vertical proximity score, and reference marker matching score. Specifically, page number consistency is assigned a score based on whether the modal object and the candidate node are on the same page number; vertical proximity score is obtained by calculating the distance between the bounding box coordinates of the modal object and the bounding box coordinates of the candidate node in the vertical direction of the page, and then normalizing this distance using a Gaussian kernel function; and reference marker matching score is assigned a score by detecting whether the candidate node's text contains references to the modal object. Each modal object is bound to the section with the highest overall binding score, and the identifier and storage path of each modal object are recorded in the attributes of the corresponding leaf node of the tree document graph.

[0011] Furthermore, the step of extracting information from each leaf node in the tree-shaped document graph and constructing an enterprise knowledge graph based on the extracted entities and the relationships between entities includes: Based on a rule engine composed of a predefined domain dictionary and regular expression templates, the first entity information is obtained by extracting structured information with a fixed format from the text content of the leaf nodes. Based on the named entity recognition model that is further pre-trained on the domain corpus, entities and relationships are extracted from the text content of the leaf nodes to obtain the second entity information and the relationship between the second entities. Using a large language model, the text content of the leaf nodes is extracted for semantic reasoning. Causal and conditional relationships in the text content are extracted to obtain the third entity relationship. Based on the first entity information and the second entity information, disambiguation and normalization are performed on entities that refer to the same object extracted from different leaf nodes to obtain the processed entities. The processed entities are used as nodes, and the corresponding second or third entity relationships are used as edges to construct an enterprise knowledge graph.

[0012] Furthermore, the process of performing intent recognition and structured parsing on the received user query includes: The user query text is input into a fine-tuned large language model to obtain the intent recognition result; the fine-tuned large language model is trained to classify the query into predefined intent categories, which at least include fault diagnosis, operation query, concept explanation, and multi-hop relation reasoning. Based on intent recognition, the fine-tuned large language model is used to extract key elements from user queries and organize them into a structured representation to obtain structured parsing results; the key elements include at least: entities, attributes, relationships, and operating conditions.

[0013] Furthermore, the construction process of the semantic vector index, keyword inverted index, and graph index includes: The semantic encoding model is used to encode the text content of each leaf node in the tree document graph to obtain the vector representation of each node. The vector representations of all leaf nodes and their corresponding node representations are stored in a vector database to obtain a semantic vector index. The semantic encoding model is a pre-trained language model, including but not limited to the BERT model and the BGE model. The text content of each leaf node is segmented and stop words are removed to obtain several keyword terms. A keyword inverted index is constructed based on the keyword terms. The keyword inverted index establishes a mapping relationship from the keyword terms to the list of leaf node identifiers containing the keyword terms. Based on the enterprise knowledge graph, a graph database is used to store entity nodes, relationships between entities, node attributes, and edge attributes, and an index is built from entity name to graph node ID to obtain the graph index.

[0014] Furthermore, the multi-path recall refers to the process of searching in parallel using the semantic vector index, keyword inverted index, and graph index based on the structured representation of the user query, specifically including: Semantic Vector Recall: The semantic encoding model is used to encode the user query to obtain the user query vector, and the cosine similarity between the user query vector and the vector representation of all nodes in the vector database is calculated to obtain the semantic score. The top-N nodes with the highest semantic scores are then returned. Keyword Recall: Based on the BM25 algorithm, calculate the keyword matching score between user query q and each leaf node i, and return the Top-N nodes with the highest keyword matching scores; wherein, the keyword matching score The calculation formula is: w represents the intersection of the user query q and the leaf node i, i.e., the keywords that appear together. This indicates the frequency of the keyword term w in leaf node i. `b` is a hyperparameter used by the user to adjust word frequency saturation, and `b` is a hyperparameter used to adjust the degree of document length normalization. The length of the text content of leaf node i. Indicates the inverse document frequency of term w; Graph Recall: Entities identified from user queries Starting with the aforementioned graph index, a traversal query is performed within the enterprise knowledge graph to find entities. Entities connected by a path The associated leaf nodes are identified, and a graph correlation score is calculated based on the path weights. The top-N nodes with the highest graph correlation scores are then returned. The graph correlation score... The calculation formula is: , This indicates that in a knowledge graph, starting from the initial entity node... The entity node associated with leaf node i The set of all paths between them Let p represent a path in set P. This represents the entity nodes u and v connected by the edge in path p. Representing an edge The weight.

[0015] Furthermore, the process of sorting the multiple candidate leaf nodes using a reordering model includes: All candidate child nodes obtained from the multi-path recall are deduplicated to form the initial candidate set; For each node i in the candidate set, obtain the semantic score, keyword matching score, and graph relevance score of the node, and calculate the final relevance score of each candidate node by weighted summation. All candidate nodes are sorted in descending order based on the final relevance score, and the top K nodes are selected as the selected nodes; where K < N.

[0016] Furthermore, the enterprise knowledge graph supports incremental updates, and the incremental update process includes: When a new enterprise document is available, repeat the steps of building the tree document graph and extracting information, and extract new entity and relation triples from the leaf nodes of the new enterprise document. The similarity between the new entity and existing entities in the enterprise knowledge graph is calculated. If the similarity is greater than a preset similarity threshold, the entity is marked as the same entity and merged. Otherwise, the new entity is added as a new node in the enterprise knowledge graph, resulting in an updated enterprise knowledge graph. The newly extracted relationships are added to the updated enterprise knowledge graph to complete the incremental update of the knowledge graph.

[0017] Furthermore, the prompt information includes at least: (1) user query, (2) text content of the selected node, the source document number of the text content of the selected node and its hierarchical path in the tree document diagram; and (3) multimodal content bound to the selected node.

[0018] Compared with the prior art, the beneficial effects of this application are: First, at the knowledge organization and understanding level, this application fundamentally improves the machine's ability to understand unstructured enterprise knowledge through refined document parsing and multimodal fusion. Specifically, unified cleaning of multi-source heterogeneous documents and hierarchical parsing based on multiple features can accurately restore the inherent complex structure of documents, ensuring that each retrieval unit is a semantically complete business fragment, which lays the foundation for subsequent accurate recall. Furthermore, by combining visual analysis, location proximity, and text reference detection, multimodal content such as images and tables is accurately anchored to relevant text sections, realizing the native association between text and visual information, enabling the system to "understand" and utilize charts. In the information extraction stage, the integration of rule engines, domain pre-trained models, and large language models balances rule clarity, domain specialization, and deep reasoning capabilities, enabling the systematic extraction of structured entities and complex relationships from text, constructing a high-quality, computable enterprise knowledge graph, and connecting scattered knowledge points into a network.

[0019] Secondly, at the knowledge retrieval and utilization level, this application significantly enhances the responsiveness to complex and specialized queries by constructing a multi-dimensional index and performing collaborative retrieval. Simultaneously constructing a semantic vector index, a keyword inverted index, and a knowledge graph index is equivalent to equipping the same knowledge base with three retrieval engines: "semantic understanding," "literal matching," and "logical reasoning." Faced with user queries, the system performs semantic vector recall, keyword recall, and knowledge graph recall in parallel. This multi-pronged strategy effectively addresses challenges common in enterprise scenarios, such as low-frequency terms and complex combinations of conditions, improving the recall rate and coverage of relevant information. Subsequently, through a re-ranking model, the multi-path results are intelligently fused and sorted to filter out the most relevant and comprehensive information nodes, providing high-quality raw materials for generating high-quality answers.

[0020] Finally, regarding system sustainability and answer generation quality, the design of this application ensures the practicality and reliability of the method. The knowledge graph supports incremental updates, enabling the system to continuously evolve with updates to enterprise documents, avoiding the cost of repeated construction. In the answer generation stage, the system does not allow the large language model to create answers out of thin air, but rather carefully organizes prompts, explicitly providing sorted and filtered node text, its precise document source, hierarchical position, and bound multimodal content identifiers. This evidence-based approach effectively constrains the large language model, enabling it to generate traceable answers based on given materials, and naturally integrates multimodal display guidance, greatly enhancing the factual accuracy, operability, and user trust of the answers. In summary, this application systematically solves the problems of knowledge structuring, retrieval accuracy, and generation credibility in enterprise multimodal document question answering through a series of interconnected and meticulous steps. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating an enterprise multi-level multimodal document question-answering method based on a large language model, provided for an embodiment of this application; Figure 2 A flowchart illustrating another enterprise multi-level multimodal document question-answering method based on a large language model provided in this application embodiment; Figure 3 This is a flowchart illustrating another enterprise multi-level multimodal document question-answering method based on a large language model, provided as an embodiment of this application. Detailed Implementation

[0023] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0024] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0025] To address the technical problems in existing technologies, such as the fragmentation of original semantic units in documents due to fixed-length block segmentation, difficulty in effectively retrieving and utilizing multimodal content in documents, insufficient recall capability for low-frequency technical terms and complex multi-hop relationships, and the susceptibility to factual errors or untraceable answers due to the lack of explicit modeling of the enterprise's internal knowledge structure, this application provides an enterprise multi-level multimodal document question-answering method based on a large language model. This method includes: Collect and clean multi-source documents from within the enterprise, perform hierarchical structure parsing on the cleaned documents, and construct a tree-structured document graph based on the parsed title hierarchy and the multimodal content extracted from the documents; where each leaf node of the tree-structured document graph corresponds to a document fragment; Information is extracted from each leaf node in the tree-structured document graph, and an enterprise knowledge graph is constructed based on the extracted entities and the relationships between them. The system performs intent recognition and structured parsing on received user queries, and performs multi-path recall based on semantic vector index, keyword inverted index and graph index built on tree document graph and enterprise knowledge graph to obtain multiple candidate leaf nodes; The reordering model is used to sort multiple candidate leaf nodes, and the top K leaf nodes with the highest relevance are selected to obtain the selected node. The user query, the text content corresponding to the selected node, and the multimodal content bound to the selected node are combined to construct a prompt message, which is then input into the large language model to generate question-and-answer results that include citation sources and multimodal display information.

[0026] Based on this, this method systematically improves the accuracy, traceability, and practicality of enterprise-level document question answering by preserving the document hierarchy, integrating multimodal content, and constructing a multi-dimensional retrieval index, effectively solving the core problems of knowledge fragmentation, incomplete recall, and unreliable answers in existing technologies.

[0027] like Figure 1 As shown in the embodiment of this application, an enterprise multi-level multimodal document question answering method based on a large language model is provided, including: S1. Collect and clean multi-source documents within the enterprise, perform hierarchical structure parsing on the cleaned documents, and construct a tree-shaped document diagram based on the parsed title hierarchy and the multimodal content extracted from the documents.

[0028] Hierarchical structure parsing refers to the identification and organization of logical units such as chapters, sections, and paragraphs in a document, restoring it from a flat text flow into a hierarchical structure. Each leaf node in a tree-structured document graph corresponds to a document fragment. The purpose of constructing a tree-structured document graph is to preserve the original semantic integrity and logical connections of the document, ensuring that each leaf node corresponds to a semantically complete business unit and avoiding contextual breaks caused by forced segmentation.

[0029] In some implementations, document hierarchy can be identified based on the document's directory structure, heading styles such as font size and bolding, and numbering patterns such as "1.1.1". Specifically, by parsing the underlying tags or bookmarks of portable document formats such as PDF or Microsoft Office documents, the hierarchical relationship between headings and body text can be extracted. For documents lacking structured markup, such as scanned documents, optical character recognition (OCR) technology can be used to convert the image into text, and then heuristic rules such as regular expressions can be used to match heading numbers and analyze paragraph indentation and line spacing to infer hierarchical relationships. After hierarchical identification is completed, the document content is divided into nodes according to the hierarchy and organized into a tree structure based on parent-child relationships to obtain a tree-structured document graph.

[0030] It should be noted that the hierarchical structure of a document may be ambiguous due to differences in formatting or layout. For example, some documents do not use explicit numbering, but only distinguish headings by font size. Therefore, multiple features can be combined for comprehensive judgment during the parsing process, such as analyzing font size, line spacing, indentation, and paragraph semantics simultaneously, to improve the robustness of hierarchy recognition.

[0031] For example, for a technical manual containing multi-level headings, firstly, its bookmark structure is parsed to obtain the framework of first-level and second-level headings. Then, for paragraphs in the main text, by detecting whether their first line contains keywords such as "Step 1" or "Note", they are classified as child nodes under the corresponding second-level headings, ultimately forming a tree-shaped document diagram containing chapters, sections, paragraphs, and attached charts.

[0032] S2. Extract information from each leaf node in the tree-structured document graph and construct an enterprise knowledge graph based on the extracted entities and the relationships between them.

[0033] Information extraction refers to extracting structured information units, such as entities, attributes, and semantic relationships between entities, from unstructured text. Entities can be concrete objects such as devices, parameters, and fault phenomena, while relationships can be logical connections such as "cause," "contain," and "located in." The purpose of building an enterprise knowledge graph is to connect knowledge scattered across different document nodes to form a computable knowledge network, supporting complex logical reasoning and cross-segment relational queries.

[0034] In some implementations, named entity recognition (NAME) techniques from the field of natural language processing can be used to identify entities in text. For example, a language model can be pre-trained using a transformer-based bidirectional encoder representation technique and fine-tuned using annotated corpus from within the enterprise to accurately identify domain-specific entities such as "the controller of model X". For extracting relationships between entities, text classification or sequence labeling methods based on the pre-trained language model can be used, or rule-based pattern matching can be employed to identify fixed phrases such as "A is composed of B". The extracted entities and relationships are stored in the form of resource description framework triples, i.e., "entity-relationship-entity", and stored in a graph database, such as Neo4j, to construct a knowledge graph.

[0035] It should be noted that different leaf nodes may contain different descriptions pointing to the same real-world object, such as "fire control panel" and "fire alarm controller". When constructing a knowledge graph, entity alignment is required. This involves merging entities pointing to the same object into the same graph node by calculating the semantic similarity of entity names or using a pre-defined thesaurus, in order to ensure accurate connectivity of knowledge.

[0036] For example, "sensor A" and "controller B" and the relationship "connection" are extracted from one leaf node, and "controller B" and "alarm signal" and the relationship "send" are extracted from another node. Entity alignment identifies "controller B" as the same entity, thus forming a knowledge chain of "sensor A - connection - controller B - send - alarm signal" in the graph.

[0037] S3. Perform intent recognition and structured parsing on the received user queries, and perform multi-way recall based on the semantic vector index, keyword inverted index and graph index built on the tree document graph and enterprise knowledge graph to obtain multiple candidate leaf nodes.

[0038] Intent recognition refers to determining the purpose behind a user's query, such as seeking operational steps, fault causes, or concept definitions. Structured parsing involves extracting key elements from natural language queries, such as entity names, attribute conditions, and relational constraints, and converting them into a machine-processable structured form. Semantic vector indexing is an index structure that represents text semantics using vectors and supports similarity retrieval; keyword inverted indexing records the mapping relationship between words and documents containing those words; and graph indexing is based on the structure of nodes and edges in a knowledge graph, supporting retrieval by relational path. Multi-path recall refers to simultaneously utilizing multiple indexes to retrieve candidate nodes relevant to the query from different perspectives, thereby improving the comprehensiveness and accuracy of the recall results.

[0039] In some implementations, intent recognition can be achieved by training a classifier. This classifier takes the user's query text as input and outputs a predefined intent category. The classifier can employ models such as Support Vector Machines (SVMs) or Transformer-based bidirectional encoder representations. Structured parsing can utilize semantic parsing techniques to convert the query into a structured query language or a similar logical form. For semantic vector indexing, a pre-trained language model can be used to encode document fragments and queries into fixed-dimensional vectors, and then relevant fragments can be retrieved by calculating the cosine similarity between vectors. Keyword inverted indexes can be constructed using traditional text analysis techniques. After word segmentation, a mapping table from terms to document identifiers is established, and relevance scores are calculated using TF-IDF (Term Frequency-Inverse Document Frequency) or the BM25 algorithm during retrieval. Graph indexing utilizes the query language of graph databases, such as ciphertext query language, to find other entities connected by relational paths and their associated document nodes in the graph based on the entities identified in the query.

[0040] It's important to note that different indexes have different focuses: semantic vector indexes excel at capturing queries that are semantically similar but use different terms, keyword inverted indexes are more accurate for precise term matching, and graph indexes can handle complex relational queries that require multi-hop reasoning. These three types of indexes complement each other and can effectively address the retrieval needs of enterprise documents with diverse terminology and complex relationships.

[0041] For example, for the query "processing flow of model X device reporting error code Y in high temperature environment", intent identification classifies it as "fault diagnosis". Structured parsing extracts the entities "model X device", "error code Y", and the condition "high temperature environment". In multi-path recall, semantic vector index finds paragraphs with similar content semantics; keyword inverted index precisely matches paragraphs containing "model X" and "error code Y"; graph index starts from the "model X device" node in the graph, finds "error code Y" through the "has fault" relationship, and then locates the relevant document node through the "processing method" relationship.

[0042] S4. Use the reordering model to sort multiple candidate leaf nodes, and select the top K leaf nodes with the highest relevance to obtain the selected node.

[0043] Among them, the re-ranking model is used to refine the ranking of the mixed candidate set generated by multi-way recall, eliminate the problem of inconsistent scoring scales caused by different retrieval methods, and rank the nodes most likely to contain the answer first, thereby improving the quality of the final answer.

[0044] In some implementations, the re-ranking model can employ a neural network model based on a cross-encoder. This model concatenates the user query with the text content of candidate nodes as input, and uses an internal attention mechanism to deeply calculate the degree of matching between the two, outputting a relevance score. Compared to relying solely on vector similarity or keyword matching, this deep interaction model based on a cross-encoder can more accurately capture the semantic relationship between the query and document fragments. When training this model, manually labeled query-document relevance data can be used as a supervision signal.

[0045] It should be noted that re-ranking models typically incur significant computational overhead. Therefore, it is generally advisable to first use multi-path recall to quickly select a candidate set of moderate size, and then apply a re-ranking model to finely sort this candidate set, achieving a balance between efficiency and effectiveness. The top K nodes selected after sorting ensure both the richness of information and the removal of a large amount of noise.

[0046] For example, the multi-path recall returned 100 candidate nodes. The re-ranking model calculated the deep matching score between each of these 100 nodes and the query, then sorted them in descending order of score, and finally selected the top 5 nodes with the highest scores as input for subsequent steps.

[0047] S5. Combine the user query, the text content corresponding to the selected node, and the multimodal content bound to the selected node to construct a prompt message, input it into the large language model, and generate a question-and-answer result containing the source of the reference and multimodal display information.

[0048] The prompts are structured inputs that guide the large language model to generate the expected output. By providing the model with the original query, highly relevant textual evidence filtered through retrieval, and the sources of that evidence, the model's generation space can be effectively constrained, reducing the risk of factual errors, or "illusions." Citation sources refer to information such as the specific document name and chapter path on which the answer is based, ensuring the traceability of the answer. Multimodal presentation information refers to identifiers or paths pointing to visual content such as images and tables, enabling the final answer to be presented in a visually appealing way.

[0049] In some implementations, specific prompt templates can be designed. For example, the template might include sections like "Reference Documents," "User Questions," and "Answer Requirements." The "Reference Documents" section lists the text content and its source paths for selected nodes in a fixed format. The "Answer Requirements" section explicitly instructs the model to answer based on the provided reference information and specifies that when the answer involves a diagram, the corresponding diagram reference should be output. The filled-in prompt information is then sent to a large language model, such as a generative pre-trained transformer. The model generates a structured text based on the instructions, naturally incorporating explanations of the cited sources and instructions to invoke multimodal content.

[0050] It should be noted that the output format of a large language model can be designed to be easy for the front end to parse, such as a markup language format or a lightweight data exchange format, so that the question answering system can clearly display the answer content, the corresponding source document, and render the bound images or tables when needed.

[0051] For example, the constructed prompt message is: "Please answer the question based on the following reference. Reference 1: From Chapter 3, Section 2 of the Operation Manual, which states 'Pressing the reset button will clear the alarm.' User question: How to clear the alarm? Please provide an answer and cite the source." The answer generated by the large language model is: "According to Chapter 3, Section 2 of the Operation Manual, the method to clear the alarm is to press the reset button." Based on the above technical solutions, this application provides a multi-level, multimodal document question-answering method for enterprises based on a large language model. This method preserves the semantic integrity of documents by constructing a tree-structured document graph, achieves deep knowledge association by constructing a knowledge graph, improves retrieval accuracy and coverage through a combination of multi-path recall and re-ranking, and ensures the credibility and traceability of answers through a mechanism that generates prompts with prior evidence. This method systematically solves a series of problems existing in enterprise multimodal document question-answering technologies, such as knowledge fragmentation, lack of multimodal information, difficulty in recalling complex queries, and the tendency for answers to produce illusions. It achieves a complete technical closed loop from massive heterogeneous documents to accurate and reliable answers.

[0052] In one possible implementation of this application embodiment, the above-mentioned S1 can be specifically implemented by the following S101 and S102, which are described in detail below: S101. Clean and parse the hierarchical structure of the collected internal enterprise documents to obtain the title level and paragraph structure of each document.

[0053] Hierarchical structure parsing refers to automatically identifying the hierarchical relationships between headings, subheadings, and body text in an unstructured document, restoring the document from a linear text flow into a tree-like skeleton with hierarchical logic.

[0054] In some implementations, hierarchical structure resolution is achieved in the following ways: First, collect heterogeneous documents from different systems within the enterprise, including technical manuals, operating procedures, troubleshooting guides and standard specifications. The document formats include portable document formats, Microsoft Office documents, presentation documents or scanned copies. The collected documents are converted to a uniform format and have their encoding standardized, converting all documents into a uniform text format and unifying the character encoding to the universal character set conversion format-8; Next, the headers, footers, watermarks, and annotations in the document are removed by using a rule-based regular expression matching method. By calculating the text similarity or semantic vector similarity between document fragments, fragments with similarity higher than a preset threshold are marked as duplicate content and removed, thus completing the document cleaning process. Then, the cleaned document undergoes table of contents parsing, title detection, and paragraph segmentation. Table of contents parsing is achieved by parsing the document's bookmarks or style structure; title detection is based on heuristic rules, including identifying font size, font style, layout position, and numbering style; and paragraph segmentation is achieved by analyzing the document's line breaks, indentation, and line spacing features. Finally, by utilizing the directory structure, font size, numbering style, and spacing features before and after paragraphs, a multi-feature fusion hierarchical partitioning algorithm is used to determine the hierarchical structure of the cleaned documents and obtain the title level of each document.

[0055] It should be noted that due to the diverse sources and inconsistent formats of internal enterprise documents, some scanned documents lack structured information. For such documents, it is necessary to first convert the images into editable text using optical character recognition (OCR) technology before performing the aforementioned analysis. Furthermore, multi-feature fusion-based hierarchical segmentation algorithms can integrate information from multiple dimensions such as font, numbering, and spacing, effectively addressing differences in document layout styles and improving the accuracy of hierarchical recognition.

[0056] In some implementations, the hierarchical partitioning algorithm is based on rule-based decision trees or weighted scoring models. It extracts multi-dimensional features from each text paragraph in the document and makes a comprehensive judgment, ultimately outputting the heading level to which each paragraph belongs. Its specific workflow is as follows: Step 1: Feature Extraction and Quantization. For each text paragraph in the cleaned document, extract the following four types of features and quantize them into numerical representations: Font style features: Quantified based on font size and whether the text is bolded. The larger the font size and the more pronounced the bolding, the higher the font style feature value, indicating that it is more likely to be a high-level heading. The specific quantification method is: mapping font size to numerical values, with bolded portions receiving additional weight; Sequence numbering depth feature: This feature determines the numbering depth of the paragraph's beginning by matching numbering patterns at the start of the paragraph using regular expressions, such as "Chapter 1", "1.1", "1.1.1", "(I)", etc. For example, the numbering depth of "1.1.1" is 3, and the numbering depth of "Chapter 1" is 1. Paragraphs without a matching number are recorded as having a feature value of 0. Directory structure anchoring feature: Matches paragraph text with parsed directory entries. If a paragraph text is completely identical or highly similar to a certain entry in the directory, the preset level of that entry in the directory tree is obtained as the directory structure anchoring feature value for that paragraph. This feature can directly use document directory information to correct the level determination result; Spacing characteristics: Extract the physical unit values ​​of the spacing before and after the paragraph, usually in points. Paragraphs with larger spacing before the paragraph often mark the beginning of a new chapter, while paragraphs with larger spacing after the paragraph are often used to separate headings from body text.

[0057] Step 2: Comprehensive Hierarchy Score Calculation and Preliminary Judgment. The quantized feature values ​​of each text paragraph are input into a predefined hierarchy determination model to calculate the paragraph's comprehensive hierarchy score. The hierarchy determination model uses a weighted scoring model, and its calculation formula is as follows: ; in, , , , These are the quantified values ​​for font style features, sequence number depth features, directory structure anchoring features, and layout spacing features, respectively. , , , For the corresponding weight coefficients, satisfying In the weighting settings, directory structure anchoring features and sequence number depth features are given higher weights because these two types of features have the most direct and reliable correlation with the actual hierarchical structure of the document; font style features and layout spacing features have relatively lower weights and are used as auxiliary judgment criteria. Based on the calculated comprehensive level score, a preset level threshold range is matched. For example, paragraphs with a score between 0.8 and 1.0 are classified as level 1 headings, those between 0.6 and 0.8 as level 2 headings, those between 0.4 and 0.6 as level 3 headings, and paragraphs with a score below 0.4 are classified as body text. This yields the initial level classification for each text paragraph.

[0058] Step 3: Global consistency verification and adjustment.

[0059] Based on the physical order of the document, a global consistency check is performed on the initial hierarchical classification of all text paragraphs. The check process detects any hierarchical jumps that do not conform to the document's logic, such as jumping directly from a level 1 heading to a level 3 heading without a level 2 heading in between; or inverted heading levels, such as a later heading being at a higher level than the previous one. When such anomalies are detected, the classification level of the current paragraph is adjusted based on the hierarchical characteristics of adjacent paragraphs. The adjustment strategy includes: automatically completing missing intermediate levels based on contextual semantics and layout features for hierarchical transitions; and re-evaluating the overall hierarchical score of affected paragraphs for inverted hierarchical situations, and correcting them by referring to the hierarchical levels of their preceding paragraphs if necessary. The final output is a validated and adjusted document title hierarchy with correct structure, ensuring that the document's hierarchical tree structure conforms to actual reading logic.

[0060] It should be noted that this multi-feature fusion hierarchical partitioning algorithm integrates the visual, structural, and semantic features of documents, effectively handling differences arising from various document formats and layout styles. For documents lacking a table of contents, the algorithm primarily relies on sequence number depth features, font style features, and spacing features for judgment; for documents containing a complete table of contents, the table of contents anchoring features can provide high-confidence hierarchical information, significantly improving the accuracy of judgment. A global consistency verification mechanism further ensures the logical coherence of the output hierarchical structure, avoiding hierarchical errors caused by misjudgment due to a single feature.

[0061] For example, consider the paragraph "1.1.2 Power Module Installation" in a technical manual. Its sequence number depth feature is 3, the font style is bold (small four), the paragraph spacing is 12 points, and there is a completely matching entry in the table of contents. After feature extraction, the table of contents anchoring feature and sequence number depth feature both indicate a level of 3. The font style and spacing features also support that it is a heading rather than body text. The weighted calculation and overall level score fall within the threshold range for a level 3 heading, initially classifying it as a level 3 heading. Subsequent verification revealed that the preceding paragraph, "1.1.1 Chassis Fixing," is also a level 3 heading, while the following paragraph is body text. The hierarchical order is correct and requires no adjustment; therefore, the paragraph is ultimately output as a level 3 heading.

[0062] S102. Construct a tree-structured document diagram based on the parsed heading levels and the multimodal content extracted from the document.

[0063] A tree-structured document graph (BSG) is a logical model that organizes document content in a tree-like structure. Each node represents a document unit, with the root node corresponding to the entire document, intermediate nodes corresponding to headings at various levels, and leaf nodes corresponding to indivisible document segments, typically paragraphs or groups of closely related sentences. The purpose of constructing a BSG is to organically integrate the hierarchical structure of a document with multimodal content, ensuring that each leaf node not only contains text content but also links to related visual information such as images and tables, thus providing complete semantic units for subsequent retrieval.

[0064] In some implementations, the tree document graph is constructed in the following ways: First, based on the heading hierarchy, numbering style and directory structure obtained from S101 parsing, the headings at each level and their subordinate paragraphs are organized into a tree-shaped document diagram structure containing chapters, sections, subsections and entries, and this tree-shaped document diagram is stored as a hierarchical lightweight data exchange format document; Then, based on computer vision and object detection technologies, the document page is parsed to identify multimodal objects in the document, including text blocks, images, tables, and diagrams. The type, page number, and bounding box coordinates within the page number of each modal object are extracted to obtain the multimodal content. Next, based on the page number of each modal object, all sections located on the same or adjacent pages as the modal object are identified as candidate sections; Finally, based on the multimodal content, a comprehensive binding score is calculated for each modal object and its candidate subsections. Each modal object is then bound to the subsection with the highest comprehensive binding score, and the identifier and storage path of each modal object are recorded in the attributes of the corresponding leaf node in the tree document graph. The formula for calculating the comprehensive binding score is: ; Where o represents the modal object and s represents the candidate section; P(o,s) is the page number consistency score, which is 1 when the modal object and the candidate section are on the same page, and 0 otherwise; V(o,s) is the vertical proximity score, which is obtained by calculating the distance between the bounding box coordinates of the modal object and the bounding box coordinates of the candidate section in the vertical direction of the page, and normalizing the distance using a Gaussian kernel function. The closer the distance, the higher the score; R(o,s) is the reference marker matching score, which is assigned a score by detecting whether the candidate section text contains reference keywords pointing to the modal object, such as "as shown in Figure X" or "see Table X". If it contains such keywords, the score is 1, and otherwise it is 0. , , These are the weights of the three components, used to balance the impact of different factors on the binding result.

[0065] It's important to note that the comprehensive binding score design fully considers the association between multimodal content and text content. Page number consistency ensures that images and text are physically adjacent, vertical proximity further refines the spatial correspondence between images and text within the same page, and reference marker matching semantically confirms explicit textual references to charts and graphs. The combination of these three elements effectively and accurately anchors images, tables, and other content to the most relevant text paragraphs, avoiding binding errors caused by simply dividing content by location.

[0066] For example, page 5 of a technical document contains a wiring diagram and a section describing the wiring steps. The bounding box of the wiring diagram is located in the center of the page, while the text of the section describing the wiring steps is located in the upper half of the page, and the section text contains the phrase "as shown in Figure 5". When calculating the overall binding score, the page number consistency score is 1, the vertical proximity score is a high value calculated based on the vertical distance between the bounding box and the text area, and the reference marker matching score is also 1. The overall binding score of this section is significantly higher than that of other sections on the same page, therefore the wiring diagram is correctly bound to the section describing the wiring steps.

[0067] Based on the above technical solution, this embodiment of the application realizes the transformation from original multi-source heterogeneous documents to a structured knowledge system through S101 and S102. S101 restores the complex structure of the document through hierarchical parsing with multi-feature fusion; S102 realizes the accurate association between text and multimodal content through a comprehensive binding scoring mechanism, and constructs a tree-shaped document graph that integrates text and image information, laying a data foundation for subsequent accurate retrieval and reliable generation.

[0068] In one possible implementation of this application embodiment, the above-mentioned S2 can be specifically implemented by the following S201, S202 and S203, which are described in detail below: S201. Extract multi-source information from each leaf node in the tree-structured document graph to obtain entities and relationships.

[0069] Information extraction refers to the automated extraction of structured information units from unstructured text content. An entity is an object with specific semantic meaning in a document, such as equipment model, fault code, operating parameters, or job role; a relation is the semantic connection between entities, such as "cause," "contain," "located in," or "control." Multi-source information extraction refers to the comprehensive use of various techniques, such as rule engines, domain pre-trained models, and large language models, to parse text from different dimensions, overcoming the limitations of single extraction methods in terms of coverage and accuracy.

[0070] In some implementations, multi-source information extraction is specifically achieved in the following ways: First, a rule engine based on a predefined domain dictionary and regular expression templates extracts structured information with a fixed format from the text content of the leaf nodes to obtain the first entity information. The rule engine is suitable for extracting entities with clearly defined formats, such as device numbers, standard codes, and threshold parameters. For example, the regular expression "\b[AZ]{2}\d{4}\b" can match model codes consisting of two uppercase letters and four digits.

[0071] Next, based on the named entity recognition model that is further pre-trained on the domain corpus, entities and relationships are extracted from the text content of the leaf nodes to obtain the second entity information and the relationship between the second entities. This model is based on a transformer-based bidirectional encoder representation technology and is fine-tuned on the enterprise's internal document corpus to enable it to recognize entity types in specific domains, such as equipment, fault phenomena, and operation steps, and to extract sequence relationships such as "execution of A results in B".

[0072] Then, a large language model is used to perform semantic reasoning on the text content of the leaf nodes, extracting deep semantic relationships such as causal relationships, conditional relationships, and temporal relationships from the text content to obtain the third entity relationships. For example, conditional relationships are extracted from "If the temperature exceeds 80 degrees, the cooling system will be activated".

[0073] It should be noted that rule engines have the advantage of high accuracy and are suitable for information with fixed formats; named entity recognition models have good generalization ability and can recognize entities with non-fixed expressions; large language models have powerful semantic understanding and reasoning capabilities and can capture implicit logical relationships. These three extraction methods complement each other and can comprehensively mine structured knowledge in text from different levels, providing rich raw materials for building high-quality knowledge graphs.

[0074] For example, from the text "The detector with model FA-102 will trigger an alarm signal and notify the monitoring center when the smoke concentration reaches 0.5%" in a leaf node, the rule engine uses regular expressions to match "FA-102" as the first entity information; the named entity recognition model identifies "detector" as the device entity and "alarm signal" as the signal entity, and extracts the relationship "trigger"; the large language model further infers "smoke concentration reaches 0.5%" as the trigger condition, extracting the conditional relationship between "smoke concentration reaches 0.5%" and "triggers alarm signal". The three extraction results together constitute the information extraction output of this leaf node.

[0075] S202. The extracted entities and relations are merged and normalized to obtain standardized entities and relations.

[0076] Among them, fusion and normalization processing refers to integrating entities and relationships obtained from different leaf nodes and different extraction methods, eliminating ambiguity, merging synonymous expressions, and forming a unified and standardized knowledge representation. This ensures that each entity node in the enterprise knowledge graph is unique, avoids redundant nodes or broken knowledge chains caused by differences in expression, and thus ensures the connectivity and query accuracy of the knowledge graph.

[0077] In some implementations, the fusion and normalization processes are specifically achieved in the following ways: First, all entities extracted in S201 are normalized. For each entity, its name, type, and contextual features in the document are extracted to construct a feature vector. Contextual features refer to the semantic environment information of the text segment containing the entity, including keywords within a certain range before and after the entity, the sentence structure of the entity, other entities co-occurring with the entity, and the entity's positional distribution in the document. Contextual features are extracted as follows: For each extracted entity, N words before and after the entity are extracted as a context window, with N typically ranging from 50 to 100. Then, a pre-trained language model, such as a transformer-based bidirectional encoder representation, is used to encode the context window, mapping the semantic information surrounding the entity into a fixed-dimensional context vector. Simultaneously, the syntactic dependencies of the sentence containing the entity are extracted, recording the entity's grammatical roles as subject, object, or modifier, as discrete features. Furthermore, other entities with high co-occurrence frequencies with the entity are statistically analyzed to construct an entity co-occurrence matrix, incorporating co-occurrence relationships as part of the contextual features.

[0078] Next, the similarity between different entities is calculated. The similarity calculation comprehensively considers three aspects: string similarity of entity names, calculated using algorithms such as edit distance or longest common substring; semantic similarity of entity names, calculated by encoding entity names into vectors using a pre-trained language model and then calculating cosine similarity; and the similarity of entity context, obtained by calculating the cosine similarity between the context vectors of the two entities. These three similarities are then weighted and summed to obtain the overall similarity of the entities. When the overall similarity between two entities is greater than a preset threshold, they are marked as candidate entity pairs referring to the same object.

[0079] Then, entity linking technology is used to link candidate entity pairs to a unified entity identifier. For conflicting cases, such as two entities with the same name but different types, or high overall similarity but conflicting contexts, disambiguation is performed by incorporating contextual features. Specifically, a density-based clustering algorithm such as DBSCAN is used, taking the entity's feature vector as input to divide entities into different clusters. During clustering, entities with high similarity are grouped into the same cluster, while entities with low similarity are separated into different clusters. Entities with the same name but different types are clustered into different clusters due to significant differences in their contextual features. Entities within the same cluster are considered to refer to the same object, are uniformly normalized into a standard entity, and all aliases and their locations for that entity are recorded.

[0080] For relations, normalization is also required, mapping semantically similar relations to standard relation types. For example, "cause," "lead to," and "cause" are uniformly mapped to "causal relation"; "contain," "include," and "composed of" are uniformly mapped to "composition relation." For relations that cannot be matched by the dictionary, a large language model is used to determine the semantic similarity between two relation types. If the similarity is higher than a threshold, they are merged.

[0081] It is important to note that entity normalization is a crucial step in knowledge graph construction. In internal enterprise documents, the same device may be referred to by its full name, abbreviation, model code, or internal designation. If normalization is not performed correctly, multiple isolated nodes will appear in the knowledge graph, preventing the formation of a complete knowledge network. By using a normalization method that integrates semantic similarity and contextual features, these synonymous entities can be effectively identified and merged.

[0082] For example, the entities "fire control panel", "fire alarm controller", and "FA-102 controller" are extracted from different leaf nodes. By calculating similarity, the semantic vector cosine similarity between "fire control panel" and "fire alarm controller" reaches 0.92, and since both contexts involve fire alarm scenarios, they are determined to be the same entity and uniformly normalized to the standard entity "fire alarm controller". The string similarity between "FA-102 controller" and "fire alarm controller" is low, but through entity linking, it is found that its product description clearly states that "FA-102 controller belongs to the fire alarm controller series". Therefore, it is also linked to the same standard entity, and "FA-102 controller" is recorded as an alias for this entity.

[0083] S203. Construct an enterprise knowledge graph based on the normalized entities and relationships, and configure an incremental update mechanism.

[0084] Enterprise knowledge graphs are knowledge bases organized in a graph structure, where nodes represent entities and edges represent relationships between entities. The purpose of building an enterprise knowledge graph is to systematically organize structured knowledge scattered across different documents and nodes, forming a computable and reasonable knowledge network. This provides the foundation for building graph indexes in subsequent multi-path retrieval and responding to complex multi-hop queries. The incremental update mechanism means that when new enterprise documents are added, only the newly added parts are extracted and the graph is updated, without needing to rebuild the entire graph, thus ensuring the system's scalability and timeliness.

[0085] In some implementations, the construction and incremental updates of enterprise knowledge graphs are achieved through the following methods: First, the standardized entities obtained in S202 are used as nodes, and the standardized relationships are used as edges, and stored in a graph database. Each entity node stores its standardized name, all aliases, and the identifier of the document node to which it belongs; each relationship edge stores the relationship type, confidence level, and the identifier of the source document node.

[0086] Next, an index is constructed from entity names to graph node identifiers to support fast retrieval. When a new enterprise document is added, steps S101 to S202 are repeated to extract new entity and relation triples from the leaf nodes of the new document. For newly extracted entities, their similarity is calculated with existing entities in the knowledge graph, using the same method as entity normalization in S202. If the similarity is greater than a preset similarity threshold, it is marked as the same entity, and the attributes of the new entity are merged into the existing entity node, such as adding a new alias or updating the frequency of occurrence; otherwise, the new entity is added as a new node in the enterprise knowledge graph. For newly extracted relations, regardless of whether the connected entity is an existing node or a new node, the relation edge is added to the graph, and the source and confidence of the relation are recorded.

[0087] Finally, update the index from entity name to graph node identifier to complete the incremental update of the knowledge graph.

[0088] It is important to note that the incremental update mechanism of an enterprise knowledge graph is crucial. Enterprise documents are typically numerous and continuously updated. If all historical documents need to be reprocessed every time a new document is added, it will lead to enormous consumption of computing resources and processing delays. Through incremental updates, the system can maintain the timeliness of the knowledge graph at a lower cost, ensuring it reflects the latest document content promptly.

[0089] For example, the existing knowledge graph contains entity nodes "Sensor A" and "Controller B", and the relation "Sensor A - Send Signal - Controller B". The newly added document contains a leaf node from which the entities "Type A Sensor" and "Control Unit B" are extracted, along with the relation "Type A Sensor - Data Reporting - Control Unit B". Based on entity similarity calculation, "Type A Sensor" has a similarity of 0.95 with the existing node "Sensor A", exceeding a preset threshold, and is therefore determined to be the same entity. "Type A Sensor" is added as a new alias for "Sensor A". Similarly, "Control Unit B" has a similarity of 0.88 with "Controller B", and is also determined to be the same entity. The relation "Data Reporting" is semantically similar to the existing relation "Send Signal" and is normalized to the same standard relation type. Therefore, this update only adds a new relation edge between existing nodes and updates the alias attributes of the nodes, without requiring the reconstruction of the entire knowledge graph.

[0090] Based on the above technical solution, this application embodiment realizes the construction and maintenance of an enterprise knowledge graph from tree-shaped document graph nodes through three steps: S201, S202, and S203. S201 comprehensively mines entities and relationships in the text by integrating a rule engine, a domain pre-trained model, and a large language model for multi-source extraction. S202 eliminates ambiguity and redundancy caused by synonymous entities through entity normalization based on semantic similarity and contextual features, ensuring the consistency of knowledge representation. S203 constructs a continuously evolving knowledge graph through graph database storage and incremental update mechanisms, providing a high-quality knowledge foundation for subsequent accurate retrieval and complex reasoning, completing the transformation from unstructured documents to a structured knowledge network, and providing knowledge support for the entire question-answering system.

[0091] In one possible implementation of the embodiments of this application, combined with Figure 1 ,like Figure 2 As shown, the above S3 can be implemented through the following S301, S302 and S303, which are explained in detail below: S301. Perform intent recognition and structured parsing on the received user query to obtain the intent category and structured representation of the user query.

[0092] Intent recognition refers to determining the type of purpose behind a user's query, such as seeking troubleshooting steps, querying operational procedures, explaining technical concepts, or performing multi-condition association reasoning. Structured parsing refers to converting user queries in natural language form into key-value pairs or logical expressions that are easy for machines to process, and extracting key elements such as entities, attributes, relationships, and operating conditions.

[0093] In some implementations, intent recognition and structured parsing are specifically implemented in the following ways: User query text is input into a fine-tuned large language model to obtain intent recognition results. This fine-tuned large language model is trained to classify queries into predefined intent categories, which at least include fault diagnosis, action queries, concept explanation, and multi-hop relation reasoning. The fine-tuning process uses an internally labeled query dataset to supervise the fine-tuning of the base large language model, such as a generative pre-trained transformer, to ensure it can accurately identify query intents in specific enterprise scenarios.

[0094] Based on intent recognition, key elements are extracted from user queries using the same fine-tuned large language model and organized into a structured representation to obtain structured parsing results. The key elements of the parsing results include at least entities such as equipment model and fault code, attributes such as temperature threshold and operating status, relationships such as cause and location, and operating conditions such as high-temperature environment and specific operating conditions. The output of the structured parsing can be a lightweight data exchange format object, such as "{"intent": "fault diagnosis", "entities": [{"name": "model X equipment", "type": "equipment"}], "conditions": [{"name": "high-temperature environment", "type": "operating condition"}], "relations": []}".

[0095] It should be noted that the results of intent recognition can be used to guide the weight allocation of subsequent retrieval strategies. For example, for queries of the "concept explanation" type, the focus can be on semantic vector recall, while for queries of the "fault diagnosis" type, the weight of graph recall can be enhanced.

[0096] S302. Construct semantic vector indexes and keyword inverted indexes based on tree-structured document graphs, and construct graph indexes based on enterprise knowledge graphs.

[0097] Among them, the semantic vector index is an index structure that represents text semantics through vectors and supports similarity retrieval, capable of capturing the semantic similarity between a query and a document fragment. The keyword inverted index is an index structure that records the mapping relationship between words and documents containing those words, supporting precise literal matching retrieval. The graph index, based on the structure of nodes and edges in a knowledge graph, supports retrieval by relational path, enabling multi-hop reasoning queries. The purpose of constructing these three types of indexes is to provide diverse retrieval channels for multi-path recall, comprehensively covering user query needs from three dimensions: semantic similarity, keyword matching, and logical association.

[0098] In some implementations, the construction of the three types of indexes is specifically achieved in the following ways: Construction of the semantic vector index: A semantic encoding model is used to encode the text content of each leaf node in the tree-structured document graph, obtaining a vector representation of each node. The semantic encoding model is a pre-trained language model, including but not limited to a transformer-based bidirectional encoder representation model and a general text embedding model. The text input is then converted into a dense vector of fixed dimensions. The vector representations of all leaf nodes and their corresponding node identifiers are stored in a vector database to obtain the semantic vector index. The vector database supports efficient approximate nearest neighbor retrieval, quickly returning the node most similar to the query vector.

[0099] In a preferred implementation, for leaf nodes bound with multiple pictures or tables, a multi-modal node representation can also be adopted to enhance the indexing effect. Let the node text vector be , the set of pictures or tables bound to it be I(i), and the vector representation of the j-th picture or table be . Then the multi-modal node representation can be obtained through attention-weighted fusion: ; where the attention weight is calculated and normalized from the correlation between the text vector and the picture vector: ; is the vector inner product or a bilinear scoring function with parameters.

[0100] The fused multi-modal vector representation can be stored in the vector database instead of the pure text vector, enabling the retrieval process to perceive both text and image semantics simultaneously.

[0101] Construction of the keyword inverted index: The text content of each leaf node is tokenized and stop words are removed to obtain a number of keyword terms. Tokenization can use the maximum matching algorithm based on a dictionary or the hidden Markov model based on statistics. Stop words include common words with no practical meaning such as "de", "le", "zai", etc. Then, a keyword inverted index is constructed based on the keyword terms. This index establishes a mapping relationship from the keyword terms to the list of leaf node identifiers containing the keyword terms. The inverted index is stored using a hash table or a B-tree structure to support fast keyword queries.

[0102] Construction of the graph index: Based on the enterprise knowledge graph constructed by S2, a graph database such as Neo4j is used to store entity nodes, relationship edges between entities, as well as node attributes and edge attributes. The graph database natively supports traversal queries based on the graph structure. At the same time, an index from entity names to graph node identifiers is established. This index uses a hash table or a B-tree structure and is used to quickly locate the corresponding nodes in the graph when receiving entity elements in the query, providing an entry for graph traversal. In subsequent retrievals, the calculation of the graph association degree score can adopt the maximum value of the product of path weights: ; where represents all path sets between the starting entity node and the entity node associated with leaf node i, represents the weight of the edge (u, v).

[0103] It's important to note that the three types of indexes serve different retrieval needs. Semantic vector indexes can associate semantically similar but word-specific queries with documents; for example, a query for "how to turn off the alarm" can match nodes containing the content "deactivate the alarm sound." Keyword inverted indexes offer the advantage of exact matching for queries containing technical terms or model codes. Graph indexes can handle complex queries requiring multi-step reasoning, such as finding all fault types and solutions related to a specific device. Multimodal node representations further enhance the index's ability to perceive mixed text and image content, enabling the system to utilize both text and image semantics for retrieval. The three indexes work together to effectively improve the comprehensiveness and accuracy of retrieval.

[0104] For example, in a tree-structured document graph, the leaf node "Pressing the reset button will clear the alarm" contains text content and a diagram showing the location of the reset button. A semantic encoding model encodes the text into vectors, while the diagram is converted into vectors using an image encoding model. These vectors are then fused using attention-weighted fusion and stored in a vector database. The text of this node is segmented to obtain terms such as "press," "reset button," "clear," and "alarm," and an inverted index is established from these terms to the node's identifier. In the enterprise knowledge graph, the entity "reset button" is stored as a node, and an index is created from the name "reset button" to the node's identifier.

[0105] S303. Based on the structured representation of the user query, use semantic vector index, keyword inverted index and graph index to perform multi-way recall and obtain multiple candidate leaf nodes.

[0106] Multi-path recall refers to parallel searching across semantic vector indexes, keyword inverted indexes, and graph indexes to retrieve candidate leaf nodes relevant to the user's query from different dimensions. The purpose of multi-path recall is to overcome the limitations of single-search methods. Semantic vector recall compensates for the insensitivity of literal matching to synonyms, keyword recall ensures accurate capture of specialized terms, and graph recall supports multi-hop relational reasoning. The combination of these three methods effectively improves the coverage and diversity of the recall results.

[0107] In some implementations, multi-path recall is achieved in the following ways: Semantic Vector Recall: The user query is encoded using the same semantic encoding model as in S302 to obtain the user query vector. The cosine similarity between the user query vector and the vector representations of all nodes in the vector database is calculated to obtain a semantic score, and the top N nodes with the highest semantic scores are returned.

[0108] Keyword Recall: Based on the BM25 algorithm, calculate the keyword matching score between user query q and each leaf node i, and return the top N nodes with the highest keyword matching scores. The BM25 algorithm's calculation formula is: ; where w represents the intersection of the user query q and the leaf node i, i.e., the keywords that appear together. This indicates the frequency of keyword w in leaf node i. `b` is a hyperparameter that adjusts the word frequency saturation, typically set to 1.2 to 2.0, used to control the degree of influence of word frequency on the score. `b` is a hyperparameter that adjusts the normalization of document length, typically set to 0.75, used to control the influence of document length on the score. The length of the text content of leaf node i. This represents the average text length of all leaf nodes. The inverse document frequency of term w is represented by the formula: Where N is the total number of leaf nodes. This represents the number of leaf nodes containing the term w.

[0109] Graph Recall: Entities identified from user queries Starting with the entity name index constructed using S302 to graph node identifiers, the initial entity node in the graph is quickly located. Then, the graph database is used to traverse and query the enterprise knowledge graph, searching for leaf nodes associated with entity e_i that are connected to entity e_q via a path. For each found leaf node, a graph relevance score is calculated based on the path weight, and the top N nodes with the highest graph relevance scores are returned. The formula for calculating the graph relevance score is: ;in, This indicates that in a knowledge graph, starting from the initial entity node... The entity node associated with leaf node i The set of all paths between. This represents a specific path in set P. This represents the entity nodes u and v connected by the edge in path p. This represents the weight of the edge (u,v). The weight can be preset based on the importance of the relationship type or the authority of the document source; for example, a "directly related" relationship has a higher weight than an "indirectly related" relationship. The formula means that the maximum product of the edge weights across all paths is selected as the graph relevance score. The larger the weight product, the stronger the association between the starting entity and the target entity. If multiple entities are identified in the user query, graph retrieval is performed starting from each entity, and the results are then merged.

[0110] The candidate leaf nodes obtained from semantic vector recall, keyword recall, and graph recall are merged and deduplicated to form an initial candidate set, which serves as the input for subsequent re-ranking steps.

[0111] It should be noted that the N value for the three recall methods in multi-path recall can be configured according to the actual application scenario. Setting the N value too small may lead to missed recalls, while setting it too large will increase the computational overhead of subsequent reordering. Typically, a value between 50 and 200 is suitable. In graph recall, the path traversal depth can be limited according to the query complexity, for example, limited to within 3 hops to avoid traversal explosion. Edge weight. The design can incorporate business knowledge, such as setting higher weights for high-value relationships like "fault phenomenon - handling method" to give related nodes higher graph correlation scores.

[0112] For example, a user query might be "How should I handle a device of model X reporting error code Y in a high-temperature environment?" Structured parsing yields the entities "device of model X" and "error code Y," along with the operating condition "high-temperature environment." Semantic vector recall encodes the query and calculates its similarity with all nodes in the vector database, returning the 50 nodes most semantically similar to the overall query. Keyword recall uses the BM25 algorithm to precisely match nodes containing terms such as "model X," "error code Y," and "high temperature," returning the 50 nodes with the highest matching degree. Graph recall starts with the "device of model X" node in the graph, traversing its connected entity nodes, searching for nodes connected to "error code Y" through the "has a fault" relationship, and then connected to related document nodes through the "handling method" relationship, returning the 50 nodes with the highest relevance. After merging and deduplicating the three types of recall results, a total of 120 candidate leaf nodes are obtained for subsequent re-ranking.

[0113] Based on the above technical solution, this embodiment of the application realizes the retrieval process from user query to candidate document nodes through three steps: S301, S302, and S303. S301 uses a fine-tuned large language model to perform intent recognition and structured parsing of the query, providing precise element guidance for retrieval; S302 constructs three types of indexes: semantic vectors, keyword inverted indexes, and graphs, forming a multi-dimensional retrieval infrastructure; S303 executes three recall strategies in parallel, comprehensively acquiring candidate nodes from three dimensions: semantic similarity, literal exact matching, and logical association reasoning, improving the recall capability of complex queries and ensuring the accuracy and robustness of the question-answering system.

[0114] In one possible implementation of the embodiments of this application, combined with Figure 1 ,like Figure 3 As shown, the above S4 specifically includes the following S401 to S403: S401. Deduplicatize and merge the candidate leaf nodes obtained from the multi-way recall to construct the initial candidate set.

[0115] Deduplication and merging refer to integrating the three candidate results from semantic vector recall, keyword recall, and graph recall, removing duplicate nodes, and forming a unified initial candidate set.

[0116] In some implementations, deduplication and merging are specifically achieved in the following ways: Let the set of candidate nodes obtained from semantic vector recall be denoted as . The set of candidate nodes obtained by keyword recall is denoted as The set of candidate nodes obtained by the graph recall is denoted as Perform a union operation on these three sets to obtain the initial candidate set. During the merging process, the unique identifier of the leaf node is used as the criterion for deduplication. When the same node identifier appears in multiple recall sets, only one node record is retained in the initial candidate set, along with the source information of that node's retrieval. The source information can be represented as a flag vector, for example, using a three-dimensional vector to mark whether the node comes from semantic vector recall, keyword recall, and graph recall, for reference during subsequent ranking.

[0117] It should be noted that the candidate sets returned by different recall methods may differ in size. For example, semantic vector recall may return a large number of semantically similar but not highly relevant nodes, while graph recall returns a relatively small number of nodes, but they are usually more strongly related. Therefore, the union method is used to ensure that all potentially relevant nodes are included in the subsequent ranking process, avoiding missed recalls due to overly strict threshold settings.

[0118] S402. Calculate the final relevance score for each leaf node in the initial candidate set using the re-ranking model.

[0119] The re-ranking model's role is to refine the ranking of nodes in the initial candidate set. The multi-path recall stage uses different scoring mechanisms based on vector similarity, BM25 score, and graph path weights, resulting in inconsistent scoring scales and making direct comparison impossible. The re-ranking model, through a unified scoring function, comprehensively considers the semantic relevance of nodes, keyword matching degree, graph association strength, and the quality factors of the nodes themselves, outputting a comparable final relevance score for each node, thereby selecting the nodes most likely to contain the answer.

[0120] In some implementations, the reordering model is implemented in the following ways: For each node i in the initial candidate set, obtain the semantic score generated by that node during the multi-path recall phase. Keyword matching score Graph correlation score If a node is not recalled using a particular recall method, its score is recorded as 0. Then, a weighted summation method is used to calculate the final relevance score for each candidate node: ;in, , , These are the normalized values ​​of semantic score, keyword matching score, and graph correlation score, respectively. Normalization can be achieved using min-max normalization or Z-score normalization, mapping scores at different scales to the same numerical range, such as [0,1]. , , These are weighting coefficients for the three scoring methods, used to adjust the importance of different recall methods in the final ranking. These weighting coefficients can be determined experimentally based on the actual application scenario; for example, in a technical document Q&A scenario, they can be set to... =0.3、 =0.3、 =0.4, to reflect the importance of graph recall in handling complex logical relationships. Q(i) is the node quality factor, which can be calculated based on prior information such as the authority of the document source, text length, and whether it contains multimodal content. For example, nodes from official technical manuals are given a higher quality factor. The weighting coefficients for the quality factor.

[0121] Besides the weighted summation method mentioned above, re-ranking models can also employ more complex machine learning models, such as those based on cross-encoders. This model concatenates the user query with the text content of candidate nodes, inputting it into a pre-trained semantic matching model, such as a transformer-based bidirectional encoder representation. The model then uses an internal attention mechanism to deeply calculate the matching degree between the query and the nodes, outputting a relevance score. This model requires pre-tuning using manually labeled query-node relevance data to adapt it to the specific ranking needs of the enterprise.

[0122] For example, the initial candidate set has three nodes with normalized scores as follows: Node A has a semantic score of 0.9, a keyword score of 0.2, and a graph score of 0.1; Node B has a semantic score of 0.3, a keyword score of 0.8, and a graph score of 0.2; Node C has a semantic score of 0.2, a keyword score of 0.1, and a graph score of 0.9. Let the weight coefficients be... =0.3、 =0.3、 =0.4, and all quality factors are 0. Therefore, the final score for node A is 0.3×0.9+0.3×0.2+0.4×0.1=0.27+0.06+0.04=0.37, the final score for node B is 0.3×0.3+0.3×0.8+0.4×0.2=0.09+0.24+0.08=0.41, and the final score for node C is 0.3×0.2+0.3×0.1+0.4×0.9=0.06+0.03+0.36=0.45. Node C receives the highest final score due to its higher graph score.

[0123] S403. Sort all candidate nodes in descending order according to the final relevance score, and select the top K nodes as selected nodes.

[0124] In this context, descending order sorting refers to arranging all nodes in the initial candidate set from highest to lowest according to the final relevance scores calculated by S402. The value of K can be set based on the maximum context length of the large language model and the average length of the text for each node.

[0125] In some implementations, sorting and selection are achieved in the following ways: Nodes in the initial candidate set are ranked according to their final relevance scores. Sort the data in descending order to obtain an ordered list. Based on a preset value K, select the first K nodes from the ordered list as the selected nodes. The value K can be set to a fixed value, such as K=5, or it can be dynamically adjusted according to the complexity of the user's query. For example, for queries identified as "multi-hop relationship reasoning," the value K can be appropriately increased to obtain more related information. In addition to the text content of the node itself, the selected node also retains information such as its hierarchical path in the tree document graph, its document number, and the multimodal content identifier bound to the node, for use in subsequent answer generation steps.

[0126] It should be noted that the K value can usually be set between 3 and 10, and the optimal value can be determined through experiments in specific application scenarios. In addition, when the score difference between the nodes ranked lower and higher is very large, an adaptive truncation strategy can be adopted, that is, only the nodes whose scores exceed a certain threshold are selected, instead of adhering to a fixed K value.

[0127] Based on the above technical solution, this embodiment of the application achieves refined screening of the multi-path recall candidate set through three steps, S401 to S403. S401 constructs a complete and non-redundant initial candidate set through deduplication and merging; S402 integrates information from three dimensions—semantics, keywords, and graph—through re-ranking models such as weighted summation or cross-encoders, calculating a unified final relevance score for each candidate node; S403 selects the top K most relevant nodes through descending sorting and truncation selection, effectively solving the problem of inconsistent score scales and difficulty in direct comparison among different retrieval methods in multi-path recall. This converges the diverse candidate set into a high-quality selected node set, providing precise evidence support for the subsequent generation of accurate and reliable answers by the large language model.

[0128] In one possible implementation of this application embodiment, the above-mentioned S5 specifically includes the following S501 to S503: S501. Construct prompt information by integrating the text content corresponding to the user query and selected node, its source information, and the multimodal content bound to the selected node into a structured input.

[0129] In some implementations, the prompt message is constructed in the following ways: Following a preset template, the user query, the text content of the selected node, the source information of the selected node, and the multimodal content bound to the selected node are assembled into a structured prompt message. The prompt message must contain at least the following three parts: The first part is the user query. The original user query text received in S301 is directly placed into the prompt message as the question that needs to be answered; The second part consists of the text content and source information of the selected nodes. For the first K nodes selected in S4, their text content, source document number, and hierarchical path in the tree document diagram are organized into evidence entries. The format of each evidence entry can be designed as: "Evidence X: From document 'Document Name', Chapter Y, Section Z, content: [Node Text]". The node text can be a complete document fragment corresponding to a leaf node, such as a complete operation step or a complete description of a fault phenomenon. The source information includes the document's unique identifier, document name, and the complete hierarchical path obtained from the tree document diagram, such as "Chapter 3 / Section 2 / Operation Steps"; The third part consists of the multimodal content bound to the selected node. For each selected node, the multimodal object identifier and storage path are obtained by accessing the node's attributes in the tree document diagram. Multimodal content includes images, tables, diagrams, etc. In the prompt message, this multimodal content is represented by placeholders or reference markers, such as "[Image: Wiring Diagram]" or "[Table: Parameter Reference Table]", accompanied by a brief text description. Simultaneously, the instruction section of the prompt message explicitly tells the model that when the answer involves a certain multimodal content, the corresponding reference marker should be output so that the front-end system can render and display it.

[0130] After assembling the above three parts according to the template, you will get a complete prompt message. Explicit instructions can be added to the template, such as: "Please answer the user's question based on the following evidence. Answers must be strictly based on the provided evidence; do not fabricate information. Please indicate the source of the evidence cited in your answer. If the answer involves images or tables, please output the corresponding citation tags." It should be noted that for multimodal content, since large language models usually do not have the ability to directly recognize image content, using reference tags to prompt the model to call multimodal content at appropriate locations, and then having the front-end system actually render the image or table based on the tags, is an effective solution that balances the limitations of model capabilities and user experience.

[0131] For example, two nodes are selected. Node 1 comes from the document "Troubleshooting Guide," with the hierarchical path "Chapter 2 / Section 1 / Fault Code E001," and the text content is "Error code E001 indicates a sensor communication failure; please check the sensor connection cable." Node 1 is bound to a sensor wiring diagram, stored at " / images / sensor_wiring.png." Node 2 comes from the same document, with the hierarchical path "Chapter 2 / Section 2 / Handling Steps," and the text content is "Handling Steps: 1. Power off; 2. Re-plug the connection cable; 3. Power on again." The user query is "How to resolve error code E001?" The constructed prompt message is: User question: How to resolve error code E001? Please answer the questions based on the evidence provided. Your answers must be strictly based on the evidence provided; do not fabricate information. Please cite the sources of the evidence in your answers. If your answers involve images or tables, please include the corresponding citations.

[0132] Evidence 1: From Chapter 2, Section 1 of the document "Troubleshooting Guide," which states: Error code E001 indicates a sensor communication failure; please check the sensor connection cable. (Binding multimodal content: [Image: Sensor wiring diagram, storage path: / images / sensor_wiring.png]) Evidence 2: From Chapter 2, Section 2 of the document "Troubleshooting Guide," which states: "Troubleshooting steps: 1. Power off; 2. Reconnect and unplug the cables; 3. Power on again." S502. Input the constructed prompt information into the large language model to generate a text answer that includes the cited sources.

[0133] Large language models refer to large neural network language models pre-trained on massive amounts of text data, such as generative pre-trained transformer models, which possess powerful text understanding and generation capabilities. Inputting prompts into a large language model allows it to leverage its language generation capabilities to synthesize coherent and accurate answer text based on the provided evidence fragments. Because the prompts contain explicit evidence and source information, the large language model can naturally incorporate source citations when generating answers, thus achieving traceability of the answer.

[0134] In some implementations, large language models generate answers in the following ways: The input is the prompt information constructed by S501. The large language model service, either through an application programming interface (API) call or locally deployed, is used to obtain the text output generated by the model. During the generation process, the large language model prioritizes the use of evidence content to organize the answer according to the instructions in the prompt information and inserts citations at appropriate locations. The model-generated answer is typically in natural language paragraph form. For content requiring step-by-step explanation, a numbered list can be automatically generated. For parts involving multimodal content, the model will output the preset citation markers in the prompt information, such as "[Image: Sensor Wiring Diagram]". In the generated answer text, citation sources can be presented in various forms, such as parentheses at the end of the sentence indicating "(Source: Document Name, Chapter Location)", or superscript numbers corresponding to the sequence number of the evidence item. To facilitate subsequent analysis, the model can be required to output according to a pre-defined format, such as using a markup language format to embed citation markers into specific tags.

[0135] For example, when the prompt message constructed in S501 is input into the large language model, the model generates the following answer text: "According to the 'Troubleshooting Guide,' error code E001 indicates a sensor communication fault. The troubleshooting steps are as follows: 1. Power off; 2. Re-plug the sensor connection cable; 3. Power on again. For specific wiring methods, please refer to the sensor wiring diagram [Image: Sensor Wiring Diagram]. (Source: Chapter 2, Sections 1 and 2 of the 'Troubleshooting Guide')." This answer explicitly cites the document source and outputs the corresponding citation mark when mentioning the wiring diagram.

[0136] S503. Associate the generated answer with the multimodal content and output it to obtain the question-and-answer results containing the source of the reference and the multimodal display information.

[0137] The question-and-answer result is the final, complete answer presented to the user. This step integrates the text answer generated by the large language model with the bound multimodal content, creating a visually appealing and interactive presentation.

[0138] In some implementations, the output and display of question-and-answer results are achieved in the following ways: First, the text answers generated by the large language model in S502 are parsed. The parsing process identifies embedded citation tags and multimodal content citation tags in the answers. Citation tags can use specific formats, such as "[Source:Document Name]" or "(Source:X)", while multimodal content citation tags use preset placeholder formats, such as "[Image:Image Name]" or "[Table:Table Name]". The parser extracts these tags and records their positions in the answer text.

[0139] Then, based on the multimodal content storage paths bound to the selected K nodes in S501, the actual image files, table data, etc., are retrieved. For tables, they can be converted into Hypertext Markup Language table format for front-end rendering; for images, a Uniform Resource Locator (URL) or file path is provided.

[0140] Finally, the parsed text answer and multimodal content are encapsulated together into a structured response format, such as a Lightweight Data Interchange Format (LTC) object. This response object contains three main fields: a plain text answer field, which retains citation details but removes rendering-related markers; a multimodal content list field, containing the type, citation markers, content data, or path for each multimodal content item; and a source list field, containing the document name and hierarchical path for each cited source. Upon receiving this response, the front-end system can render images or tables at the appropriate locations in the answer text based on the multimodal content list, while also providing a click-through function for the source information.

[0141] Based on the above technical solution, this embodiment systematically realizes a complete generation and display process from evidence to answer through three steps, S501 to S503. S501 integrates the user query, the text content of the selected node and its source information, and the bound multimodal content into high-quality prompt information using a structured template, providing a sufficient and orderly evidentiary basis for model generation. S502 utilizes the generation capabilities of a large language model to synthesize a coherent answer containing cited sources based on the prompt information. By pre-positioning evidence, the model's generation behavior is effectively constrained, reducing the risk of factual errors. S503 parses the tags in the answer and associates them with the actual multimodal content, outputting structured question-and-answer results, achieving a visually appealing display and traceable answer sources. These three steps are closely linked, completing a closed loop from retrieval results to final user delivery, ensuring the accuracy, credibility, and usability of the question-and-answer results.

[0142] In addition, the method of this application also includes: S6. Record user feedback on the question-and-answer results to update parameters in the retrieval and generation process.

[0143] The role of user feedback records is to enable the system to continuously learn from user interactions, dynamically adjust search weights, reordering parameters, or prompt templates based on users' adoption, modification, and error correction behaviors, thereby achieving self-optimization and continuous evolution of the system.

[0144] In some implementations, user feedback recording and system optimization are achieved in the following ways: The Q&A results display interface provides interactive buttons such as "Like," "Dislike," "Report Error," and "Edit." When a user approves of an answer, the system records the positive feedback relationship between the query and the selected node; when a user edits or corrects an answer, the system records the correction information provided by the user. The system periodically collects this feedback data for the following optimization scenarios: First, update the positive and negative samples of the re-ranking model by incrementally training the model using a pairwise loss function. The loss function can be expressed as: Where σ(x) is the Sigmoid function, S(q,d) + S(q,d) ) represent the fusion scores of query q with positive sample document nodes and negative sample document nodes, respectively.

[0145] Secondly, the weighting coefficients α, β, and γ in the multi-path recall fusion score were adjusted to make the system more biased towards search sources with high user acceptance. Thirdly, the prompt message template was optimized to incorporate user preferences when modifying answers into the template design.

[0146] It should be noted that the collection and use of user feedback should adhere to data privacy protection principles, user information should be anonymized, and the feedback should only be used for system optimization purposes. Once the accumulated feedback data reaches a certain scale, it can trigger offline retraining or online incremental updates of the model, ensuring continuous improvement in system performance.

[0147] Building upon S1 to S6, this method can further introduce a multi-agent cooperative scheduling mechanism to handle complex multi-turn question-answering scenarios. The multi-agent cooperative mechanism divides the system functionality into four cooperating agents: Intent Recognition Agent: Responsible for analyzing the type of user query intent, determining whether the current question is a "new question" or a follow-up to a historical question, and identifying the multi-turn dialogue context involved in the query. When a follow-up question is detected, the intent recognition agent passes the context information to the retrieval planning agent to ensure that the retrieval scope is consistent with the dialogue history; Retrieval Planning Agent: Dynamically plans retrieval strategies based on intent recognition results and the structured representation of the current query. For complex questions requiring deep reasoning, the retrieval planning agent can increase the path depth of graph retrieval or expand the recall of semantic retrieval; for simple queries, a lightweight retrieval strategy is adopted to improve response speed. Answer generation agent: Invokes the large language model and generates a preliminary answer based on the node evidence provided by the retrieval planning agent and the user query. The answer generation agent is responsible for constructing prompts and controlling the output format during the generation process to ensure that the answer includes citation sources and multimodal tags; Quality Validation Agent: Performs consistency, completeness, and source citation checks on the output of the answer generation agent. The agent verifies whether the facts in the answer are consistent with the retrieved evidence, checks for the existence of cited sources, and evaluates whether the answer fully addresses the user's question. When quality validation fails, it triggers a second search or supplementary explanation, resulting in multiple rounds of iterative optimization.

[0148] The multi-agent collaborative mechanism can flexibly adjust strategies at different stages through cooperation among agents. In critical operation and maintenance scenarios, more stringent quality checks and increased search depth can be enabled; in general consultation scenarios, search depth can be reduced to improve response speed, thereby achieving a balance between accuracy and performance.

[0149] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0150] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and modifications.

Claims

1. A large language model-based enterprise multi-level multi-modal document question and answer method, characterized in that, include: Collect and clean multi-source documents from within the enterprise, perform hierarchical structure parsing on the cleaned documents, and construct a tree-structured document graph based on the parsed title hierarchy and the multimodal content extracted from the documents; where each leaf node of the tree-structured document graph corresponds to a document fragment; Information is extracted from each leaf node in the tree-shaped document graph, and an enterprise knowledge graph is constructed based on the extracted entities and the relationships between them. The received user queries are subjected to intent recognition and structured parsing, and multiple candidate leaf nodes are obtained by multi-way recall based on the semantic vector index, keyword inverted index and graph index constructed based on the tree document graph and the enterprise knowledge graph. The multiple candidate leaf nodes are sorted using a reordering model, and the top K leaf nodes with the highest relevance are selected to obtain the selected node. The user query, the text content corresponding to the selected node, and the multimodal content bound to the selected node are combined to construct a prompt message, which is then input into the large language model to generate a question-and-answer result that includes the source of the reference and multimodal display information.

2. The enterprise multi-level multi-modal document question answering method based on a large language model according to claim 1, characterized in that, The process of collecting and cleaning multi-source documents from within the enterprise, and parsing the hierarchical structure of the cleaned documents, includes: Collect multi-source heterogeneous documents from different systems within the enterprise; the multi-source heterogeneous documents include technical manuals, operating procedures, fault handling capabilities, and standard specifications, and the formats of the multi-source heterogeneous documents include PDF, Word, PPT, or scanned copies; The multi-source heterogeneous documents are subjected to unified format conversion, encoding standardization, layout cleanup, and duplicate content detection to obtain multiple cleaned documents. The encoding standardization refers to converting the character encoding of all documents into a unified format. The layout cleanup refers to removing page headers, footers, watermarks, and annotations from the documents using a rule-based repetitive expression matching method. The duplicate content detection refers to marking segments with similarity scores higher than a preset threshold as duplicate content by calculating the text similarity or semantic vector similarity between document segments. The cleaned documents are subjected to directory parsing, title detection, and paragraph segmentation to identify the directory structure, font size, numbering style, and spacing features before and after paragraphs for each document. The directory parsing is achieved by parsing the document's bookmarks or style structure. The title detection is based on a heuristic rule method, which includes identifying font size, font style, layout position, and numbering style. The paragraph segmentation is achieved by analyzing the document's line breaks, indentation, and line spacing features. Using the directory structure, font size, numbering style, and spacing features before and after paragraphs, a multi-feature fusion hierarchical partitioning algorithm is used to determine the hierarchical structure of the cleaned documents, thereby obtaining the title level of each document. The multi-feature fusion hierarchical partitioning algorithm is a rule-based decision tree or scoring model, which determines the title level of each text paragraph based on its font size, numbering depth, level in the directory, and spacing with context.

3. The enterprise multi-level multimodal document question answering method based on a large language model according to claim 1, characterized in that, The step of constructing a tree-structured document graph based on the parsed title hierarchy and the multimodal content extracted from the document includes: Based on the title hierarchy, numbering style, and directory structure, the titles at each level and their subordinate paragraphs are organized into a tree-like document diagram structure containing chapters, sections, subsections, and entries, and the tree-like document diagram is stored as a hierarchical JSON document; The document page is parsed using computer vision and object detection technologies to identify multimodal objects in the document. The type, page number, and bounding box coordinates of each modal object within the page number are extracted to obtain multimodal content. The multimodal objects include at least: text blocks, images, tables, and diagrams. Based on the page number of each modal object, determine all sections that are on the same page or adjacent to the modal object as candidate sections; Based on the multimodal content, a comprehensive binding score is calculated for each modal object and the candidate section. This comprehensive binding score is obtained by weighted summation of page number consistency score, vertical proximity score, and reference marker matching score. Specifically, page number consistency is assigned a score based on whether the modal object and the candidate node are on the same page number; vertical proximity score is obtained by calculating the distance between the bounding box coordinates of the modal object and the bounding box coordinates of the candidate node in the vertical direction of the page, and then normalizing this distance using a Gaussian kernel function; and reference marker matching score is assigned a score by detecting whether the candidate node's text contains references to the modal object. Each modal object is bound to the section with the highest overall binding score, and the identifier and storage path of each modal object are recorded in the attributes of the corresponding leaf node of the tree document graph.

4. The enterprise multi-level multimodal document question answering method based on a large language model according to claim 1, characterized in that, The step of extracting information from each leaf node in the tree-like document graph and constructing an enterprise knowledge graph based on the extracted entities and the relationships between them includes: Based on a rule engine composed of a predefined domain dictionary and regular expression templates, the first entity information is obtained by extracting structured information with a fixed format from the text content of the leaf nodes. Based on the named entity recognition model that is further pre-trained on the domain corpus, entities and relationships are extracted from the text content of the leaf nodes to obtain the second entity information and the relationship between the second entities. Using a large language model, the text content of the leaf nodes is extracted for semantic reasoning. Causal and conditional relationships in the text content are extracted to obtain the third entity relationship. Based on the first entity information and the second entity information, disambiguation and normalization are performed on entities that refer to the same object extracted from different leaf nodes to obtain the processed entities. The processed entities are used as nodes, and the corresponding second or third entity relationships are used as edges to construct an enterprise knowledge graph.

5. The enterprise multi-level multimodal document question answering method based on a large language model according to claim 1, characterized in that, The process of performing intent recognition and structured parsing on the received user query includes: The user query text is input into a fine-tuned large language model to obtain the intent recognition result; the fine-tuned large language model is trained to classify the query into predefined intent categories, which at least include fault diagnosis, operation query, concept explanation, and multi-hop relation reasoning. Based on intent recognition, the fine-tuned large language model is used to extract key elements from user queries and organize them into a structured representation to obtain structured parsing results; the key elements include at least: entities, attributes, relationships, and operating conditions.

6. The enterprise multi-level multimodal document question answering method based on a large language model according to claim 1, characterized in that, The construction process of the semantic vector index, keyword inverted index, and graph index includes: The text content of each leaf node in the tree-shaped document graph is encoded using a semantic coding model to obtain a vector representation of each node. The vector representations of all leaf nodes and their corresponding node representations are stored in a vector database to obtain a semantic vector index. The semantic coding model is a pre-trained language model. The text content of each leaf node is segmented and stop words are removed to obtain several keyword terms. A keyword inverted index is constructed based on the keyword terms. The keyword inverted index establishes a mapping relationship from the keyword terms to the list of leaf node identifiers containing the keyword terms. Based on the enterprise knowledge graph, a graph database is used to store entity nodes, relationships between entities, node attributes, and edge attributes, and an index is built from entity name to graph node ID to obtain the graph index.

7. The enterprise multi-level multimodal document question answering method based on a large language model according to claim 6, characterized in that, The path recall refers to the process of searching in parallel using the semantic vector index, keyword inverted index, and graph index based on the structured representation of the user query, specifically including: Semantic Vector Recall: The semantic encoding model is used to encode the user query to obtain the user query vector, and the cosine similarity between the user query vector and the vector representation of all nodes in the vector database is calculated to obtain the semantic score. The top-N nodes with the highest semantic scores are then returned. Keyword Recall: Based on the BM25 algorithm, calculate the keyword matching score between user query q and each leaf node i, and return the Top-N nodes with the highest keyword matching scores; wherein, the keyword matching score The calculation formula is: w represents the intersection of the user query q and the leaf node i, i.e., the keywords that appear together. This indicates the frequency of the keyword term w in leaf node i. `b` is a hyperparameter used by the user to adjust word frequency saturation, and `b` is a hyperparameter used to adjust the degree of document length normalization. The length of the text content of leaf node i. Indicates the inverse document frequency of term w; Graph Recall: Entities identified from user queries Starting with the graph index, the enterprise knowledge graph is traversed and queried to find entities. Entities connected by a path The associated leaf nodes are identified, and a graph association score is calculated based on the path weights. The top-N nodes with the highest graph association scores are then returned. The graph association score... The calculation formula is: , This indicates that in a knowledge graph, starting from the initial entity node... The entity node associated with leaf node i The set of all paths between them Let p represent a path in set P. This represents the entity nodes u and v connected by the edge in path p. Representing an edge The weight.

8. The enterprise multi-level multimodal document question answering method based on a large language model according to claim 1, characterized in that, The process of sorting the multiple candidate leaf nodes using a reordering model includes: All candidate child nodes obtained from the multi-path recall are deduplicated to form the initial candidate set; For each node i in the candidate set, obtain the semantic score, keyword matching score, and graph relevance score of the node, and calculate the final relevance score of each candidate node by weighted summation. All candidate nodes are sorted in descending order based on the final relevance score, and the top K nodes are selected as the selected nodes; where K < N.

9. The enterprise multi-level multimodal document question answering method based on a large language model according to claim 1, characterized in that, The enterprise knowledge graph supports incremental updates, and the incremental update process includes: When a new enterprise document is available, repeat the steps of building the tree document graph and extracting information, and extract new entity and relation triples from the leaf nodes of the new enterprise document. The similarity between the new entity and existing entities in the enterprise knowledge graph is calculated. If the similarity is greater than a preset similarity threshold, the entity is marked as the same entity and merged. Otherwise, the new entity is added as a new node in the enterprise knowledge graph, resulting in an updated enterprise knowledge graph. The newly extracted relationships are added to the updated enterprise knowledge graph to complete the incremental update of the knowledge graph.

10. The enterprise multi-level multimodal document question answering method based on a large language model according to claim 1, characterized in that, The prompt information includes at least: (1) user query, (2) text content of the selected node, the source document number of the text content of the selected node and its hierarchical path in the tree document diagram; and (3) multimodal content bound to the selected node.

Images