Method for realizing semantic chain retrieval enhanced generation processing aiming at knowledge questions and answers of long document

Abstract

The invention relates to a method for realizing semantic chain retrieval enhanced generation processing for knowledge questions and answers of long documents. The method mainly comprises two parts of knowledge base construction and user knowledge questions and answers. User knowledge questions and answers are constructed based on a knowledge base, and knowledge base construction relates to long document preprocessing, semantic chain type storage, information compression and knowledge extraction and storage. When a user performs knowledge questions and answers, a retrieval vector is formed through fine-grained semantic coding, and then related document vectors and knowledge maps are matched in a knowledge base through vector similarity calculation. If the length of the document exceeds the context scale of the model, document information after static information compression is indexed through information compression, whether dynamic compression is carried out or not is evaluated, and document fragments related to tasks are returned through dynamic compression. Otherwise, directly returning the related original document as the background knowledge of the generator. And the model generator generates an answer based on the related document information / information after information compression, the knowledge graph information, the user question and a model prompt.

Description

Technical Field

[0001] This invention relates to the field of deep learning, and more particularly to the field of natural language processing and generation technology. Specifically, it relates to a method, apparatus, processor, and computer-readable storage medium for enhancing semantic chain retrieval generation processing for knowledge question answering of long documents. Background Technology

[0002] Retrieval Augmented Generation (RAG) is a technique that enhances the generative capabilities of large models by integrating relevant external data. In scenarios such as question answering, this technique can provide users with more accurate, reliable, and in-depth answers, thereby optimizing the user experience and meeting users' demand for high-quality information.

[0003] In the traditional RAG framework, the input to a large language model consists of relevant data retrieved by the RAG from external databases and user input. This supplementary knowledge can effectively improve the quality of large language models and alleviate the illusion of large models. However, its effectiveness depends on the relevance and accuracy of the data retrieved by the RAG. Unnecessary retrieval or information irrelevant to user input leads to inaccurate model-generated answers. Ideally, people expect the model to return the latest and most accurate information from external data. However, irrelevant results and incomplete retrievals inevitably occur, which affects users' ability to obtain accurate knowledge and reduces the model's effectiveness.

[0004] The main challenge in long document knowledge question answering is how to extract global information and factual details from the document. Traditional RAG methods are prone to destroying global information in chunking strategies, and low-quality information retrieved from long documents can interfere with the model's judgment. In addition, the limited context window of large models severely restricts the model's performance in long document question answering. The LongRAG framework enhances the understanding of complex long document knowledge through a dual perspective (global information and factual details). LongCite improves the fidelity and verifiability of long context question answering by generating answers with fine-grained sentence-level citations. However, the above methods are limited by the need to use long embedding models and large language models that support long inputs and are insensitive to positional biases.

[0005] Information compression can transform long retrieved contextual information into a compact representation, thereby reducing the burden on downstream models, improving the information richness integrated into large language models, and helping models process large amounts of retrieved information more efficiently and generate accurate answers more accurately. However, if the compression quality is poor, key information may be lost, affecting the accuracy of the final answer. For example, the quality of the summary in context compression directly impacts the accuracy of the final answer. Furthermore, it is necessary to ensure semantic consistency between the compressed and original information during compression. However, some current compression techniques may exhibit semantic inconsistencies when dealing with complex or semantically ambiguous problems.

[0006] Knowledge graphs are a structured information storage method that effectively addresses the challenges of question answering and understanding long document repositories, with significant advantages in entity-related question answering, especially for global problems spanning the entire document repository. Knowledge graphs extract entities from queries and retrieve related entities and relationships from graph databases. The retrieval results form a vast network of entity relationships, which, combined with the query, is submitted to a large language model to generate an answer. During the query process, not only are the most relevant nodes found, but the search also extends to other related nodes. The retrieved network structure is input into the large language model, guiding it to generate a more comprehensive answer. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, apparatus, processor and computer-readable storage medium for semantic chain retrieval enhancement generation processing for knowledge question answering of long documents.

[0008] To achieve the above objectives, the present invention provides a method, apparatus, processor, and computer-readable storage medium for enhancing semantic chain retrieval and generating knowledge-based question answering for long documents, as well as the following:

[0009] This method for enhancing semantic chain retrieval and generating knowledge-based questions for long documents is characterized by the following steps:

[0010] (1) Construct a document vector library, an information compression library, and a knowledge graph library;

[0011] (2) Encode the user's input question with fine-grained semantics to obtain a retrieval vector containing fine-grained semantics;

[0012] (3) Perform vector similarity calculation, retrieve the corresponding document with high similarity to the current user input question from the document vector library, and count the context length of the document. Perform adaptive information compression processing based on the document length.

[0013] (4) Evaluate the correlation between user questions and retrieved documents, remove irrelevant document fragments, and use this as input for the generative model;

[0014] (5) Identify entity information in the user input question, link it with the graph information corresponding to the entity in the knowledge graph library, and obtain the input information of the generation model;

[0015] (6) Summarize all the semantic information obtained above to obtain the generation instructions based on the user input question;

[0016] (7) The generative model generates the final answer to the question based on the generative instructions.

[0017] Preferably, the document vector library is constructed in the following manner:

[0018] (1.1.1) For content with a length of Document D i The document is divided into document fragments of different sizes according to the semantic sequence of the document. Ordinary documents are divided into document fragments according to chapters, while technical documents are divided into document fragments according to technical architecture.

[0019] (1.1.2) Then organize the segmented semantic fragments into semantic chains: Among them, c i The document is a fragment, divided into [1, n] segments, C Di To organize document fragments into semantic chains, c titlei c is the name of the i-th segment in the document. contenti This refers to the content of the i-th section;

[0020] (1.1.3) Encode the semantic chain into a document vector: in, The encoded document vector, and d is the dimension of the vector, R is the set of real numbers, and Embedding(·) is the text vector encoder;

[0021] (1.1.4) When importing document vectors into the database, follow the instructions... The document vector is stored as a retrieval unit in the document vector library.

[0022] Preferably, the information compression library specifically involves: converting the context information of the current document into a compact representation, and compressing the document fragments according to task requirements in the following manner:

[0023] Static compression: Performed when a document is encoded and stored in the database, it includes retaining key information and removing redundant information. The static compression semantic chain is as follows: Where, c′ i This is a statically compressed document fragment. It is a semantic chain organized after static compression, including [1, n] segments;

[0024] Dynamic compression: During the model generation process, when the size of the static compressed data retrieved from the knowledge base still exceeds the length of the model input, dynamic compression is adopted, and the process proceeds to step (4).

[0025] Preferably, the knowledge graph base is constructed by extracting entity e and relation r from the user's question in the current document and forming a triplet. <e1,r 12 ,e2>, where e1 is the first entity, e2 is the second entity, and r 12 To establish the relationship between e1 and e2, the entity includes its own attributes and document attributes. This triple is stored in the knowledge graph library to facilitate subsequent cross-document knowledge linking.

[0026] Preferably, step (2) specifically includes the following steps:

[0027] (2.1) The user question is converted into an input sequence X = [x1, ..., x] through word segmentation. i ,…,x n ], where x i For the i-th word;

[0028] (2.2) Named entity recognition technology is used to extract all entities E = {e1, e2, ...} from the sentence;

[0029] (2.3) The entity concatenation input method is adopted to concatenate the entity at the beginning of the sentence sequence, thereby enhancing the weight of entity features in the retrieval feature sequence;

[0030] (2.4) Encode the user question into the retrieved feature sequence t in the following manner. 检索 :

[0031] t 检索 =EmbeddingR2(NER(X))+Embedding(X)

[0032] t 检索 =Embedding(E) +Embedding(X)

[0033] NER(·) is the entity extractor.

[0034] Preferably, step (3) specifically includes the following steps:

[0035] (3.1) By calculating vector similarity, retrieve the top N documents with the highest similarity scores to the user's question from the document vector library: D 检索 ={D 检1 ,…,D 检N}, and record the index where the document is stored. The vector similarity is calculated in the following way:

[0036]

[0037] Where Top N(·) represents the top N documents ranked by similarity score, Sim(·) is the vector similarity calculation function, and D 检索 For the retrieved documents, This is the encoded document vector;

[0038] (3.2) By statistical analysis of D 检索 The document's context length is checked against the input length of the generative model. If it is greater, the context length is reduced through static compression, and the compressed document is retrieved using its index. Otherwise, the relevant document is returned directly. If the document C' is being retrieved... 检索 ={C′1,…,C′ N If the context length is greater than the input length of the generated model, then dynamic information compression is performed, and step (4) is executed.

[0039] Preferably, step (4) specifically involves: evaluating the relevance value γ between the user's question and the retrieved document, and removing irrelevant document fragments by comparing a threshold. And As input to the generative model:

[0040]

[0041] Where k is the criterion for whether to retain it.

[0042] Preferably, step (5) specifically involves: based on the obtained entity e i Information, query for information about entity e i Spectral information Y E and entity e i Related document information Y D Among them, the spectral information Y E It is a piece of natural language organized using templates, containing k related entities. Relationship between entities And document ID information; Y DThe document is retrieved by document ID; the k entities E = {e1,…,e...} after processing by the entity extractor NER(·) are... k Generate a graph query statement, set the query degree value for the graph database, and organize the retrieved knowledge graph results into natural language Y according to a preset template. E The aforementioned spectral information Y E And document information Y D As input to the generative model.

[0043] This apparatus for enhancing semantic chain retrieval and generating knowledge-based question answering for long documents includes:

[0044] A processor is configured to execute computer-executable instructions;

[0045] The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the method described above for enhancing semantic chain retrieval and generating knowledge-based question answering for long documents.

[0046] The processor for knowledge question answering of long documents implements semantic chain retrieval enhancement generation processing, wherein the processor is configured to execute computer-executable instructions, and when the computer-executable instructions are executed by the processor, the steps of the above-described method for knowledge question answering of long documents implement semantic chain retrieval enhancement generation processing are implemented.

[0047] The computer-readable storage medium contains a computer program that can be executed by a processor to implement the steps of the method for enhancing semantic chain retrieval generation processing for knowledge question answering of long documents as described above.

[0048] This invention employs a method, apparatus, processor, and computer-readable storage medium for semantic chain retrieval enhancement generation processing in long document knowledge question answering. Through a semantic chain-based RAG method, combined with fine-grained document segmentation, chained storage, information compression, knowledge graphs, and knowledge linking technologies, it achieves highly efficient optimization of semantic retrieval and context preservation in long document knowledge question answering. This method effectively solves the problems of semantic mismatch, context loss, and limited model context capacity in existing technologies for long document retrieval. It has the following technical advantages:

[0049] First, semantic chain storage optimizes context preservation. By fine-grainedly segmenting documents and constructing semantic chains, document fragments are organized into a chain structure according to semantic relationships, effectively maintaining the coherence of contextual information. Compared to traditional slice storage methods, semantic chains can significantly reduce context loss issues and improve the completeness and relevance of search results.

[0050] Second, information compression technology improves retrieval efficiency. By introducing feature extraction, dimensionality reduction, and relevance assessment techniques, information compression is applied to document fragments, significantly reducing storage and computational overhead. Through optimized retrieval efficiency, this method can quickly locate fragments highly relevant to the query semantics in large-scale long document databases, solving the semantic mismatch problem caused by insufficient vector similarity matching accuracy in existing methods.

[0051] Third, knowledge linking enhances semantic integrity. By associating semantic chains with knowledge graphs through knowledge linking technology, and utilizing entity recognition and relation extraction, key information from document fragments is mapped to knowledge graph nodes. This technology ensures the semantic integrity of retrieval results while enhancing the ability to handle complex semantic relationships.

[0052] Fourth, significant performance improvements and enhanced robustness. It significantly improves the ability to handle complex semantic relationships in long documents and enhances the model's robustness to data perturbations. Attached Figure Description

[0053] Figure 1 This is a flowchart of the method for enhancing semantic chain retrieval and generating knowledge-based question answering for long documents according to the present invention.

[0054] Figure 2 This is a flowchart of a specific embodiment of the method for enhancing semantic chain retrieval generation for knowledge question answering of long documents according to the present invention. Detailed Implementation

[0055] To more clearly describe the technical content of the present invention, the following description is provided in conjunction with specific embodiments.

[0056] Before describing the embodiments of the present invention in detail, it should be noted that, in the following, the terms “comprising,” “including,” or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0057] Please see Figure 1As shown, this method for enhancing semantic chain retrieval and generating knowledge-based question answering for long documents mainly includes two parts: knowledge base construction and user knowledge question answering. User knowledge question answering is based on knowledge base construction, which involves preprocessing long documents, semantic chain storage, information compression, knowledge extraction, and storage. When a user performs knowledge question answering, the document is first semantically encoded into a retrieval vector, and then the vector similarity is used to match relevant document vectors and knowledge graphs in the knowledge base. If the document length exceeds the model context size, statically compressed document information is indexed through information compression. Then, dynamic compression is evaluated, returning compressed document fragments relevant to the task. Otherwise, the relevant original document is directly returned as background knowledge for the generator. Finally, the model generator generates an answer based on relevant document information / compressed information, knowledge graph information, user questions, and model prompts. The specific processing steps are as follows:

[0058] Step 1: Knowledge Base Construction. Knowledge base construction includes building libraries such as a document vector library, an information compression library, and a knowledge graph. When building the document vector library, firstly, documents are segmented into different sizes according to the article organization order based on task requirements, and document segment names are given, such as "basic model information." Then, a key:value storage method is used to form the semantic chain of the documents. The document segment name is used as the field name, and the document segment is used as the field information. Each document is assigned an ID according to the article organization order and stored in rows as retrieval units. This facilitates knowledge vector embedding and tracing. The information compression library transforms the contextual information of documents into a compact representation, dynamically adjusting and retaining information from the most critical document segments according to specific task requirements. Information compression uses a large language model to identify the importance of document segments and condense information. It follows the same storage sequence as the document vector library, facilitating subsequent information indexing. The knowledge graph is built based on the extraction of entities and relations from documents, organizing entities and relations into triples and storing them in the knowledge graph library. During storage, information such as the document ID of the entity is recorded, facilitating cross-document knowledge links.

[0059] Step two: Encode the user question into a retrieval vector. First, identify and record the entities in the user question. Then, encode the user question into a retrieval vector containing fine-grained semantics using feature weighting.

[0060] Step 3: Retrieve relevant documents. Using vector similarity calculation, retrieve the Top N relevant documents from the document vector library and record their indexes. Calculate the context length of each document; if it exceeds the model input length, proceed to Step 4 (information compression); otherwise, if the static compression conditions are met, directly return the relevant documents.

[0061] Step four, information compression. Based on the index from step three, documents that have undergone information compression are indexed. The relevance between the user question and the retrieved documents is further evaluated, and irrelevant document fragments are removed as input to the generative model. Dynamic compression is an operation triggered when static compression exceeds the model's tokens. Similarity is dynamically calculated based on the user question; therefore, dynamic compression is performed after the user question is encoded. If the context length of the document after static compression is still greater than the input length of the generative model, dynamic compression is executed.

[0062] Step 5, Knowledge Linking. Using the entities recorded in Step 2, link the relevant graph information of that entity in the knowledge graph library, including associated entities, relationships between entities, document ID information, etc., as input to the generative model.

[0063] Step Six: Organize Relevant Knowledge. Summarize the information from Steps Three, Four, and Five, and based on the user's questions, generate a new prompt.

[0064] Step 7: Generate the answer. The generative model generates the final answer based on the prompt from Step 6.

[0065] In practical applications, such as Figure 1 As shown, for the acquired long document data source, this technical solution also includes data cleaning operations to remove noise data irrelevant to the task, facilitating subsequent model operations such as text content recognition and sequence segmentation. Examples include TXT text extraction (PDF2Txt / Word2Txt, etc.), header and footer removal, special strings, and chart formatting conversion to TXT.

[0066] In one specific embodiment of the present invention, such as Figure 2 As shown, taking the long text document library question-and-answer question "When did the pomegranate model take place?" in the field of network security modeling as an example, the processing steps of this semantic chain RAG method are further illustrated:

[0067] 1. Knowledge Base Construction. Building a document vector library includes semantic chain segmentation, semantic chain encoding, and semantic chain storage. First, for content of length M... Di The document is segmented into document fragments of different sizes according to its semantic sequence, that is, the document is divided into document blocks of different sizes based on the article chapters. For example, the technical document D of "Modeling Ideas for the Pomegranate Model" i The code is divided into n semantic segments, each in key:value format, where the key is the chapter name and the value is the chapter content. The segmented semantic segments are then organized into a semantic chain. Where c titleiIt is the chapter name of section i in the document, c contenti This refers to the content of section i. Then, the semantic chain is encoded into a document vector, and the encoded document vector is X. c ∈R 1×d Where d is the dimension of the vector. The encoding formula is:

[0068]

[0069] Here, `Embedding(·)` is a text vector encoder, such as `bge-m3`, `m3e-large`, etc. Finally, when storing document vectors in the database, according to... It is stored as a retrieval unit. Thus, the document vector library retains not only document content information, but also document attribute information and semantic sequence information.

[0070] When building an information compression library, document fragments are compressed according to task requirements. Compression is divided into static compression and dynamic compression. Static compression is performed when the document is encoded and entered into the library. It mainly retains the key information in the document and removes some redundant information, such as removing footers and headers, removing paragraphs with irrelevant technical knowledge, and condensing expert comments. The pre-encoded sequence during library entry is as follows: Dynamic compression, by assessing the relevance of user questions to document fragments, preserves relevant document fragments compared to static compression. For example, for the question "When was the pomegranate model modeled?", only the "Basic Model Information" section will be retained. For the question "What is the modeling approach for the pomegranate model?", sections such as "Next Steps" and "Expert Comments" will be removed.

[0071] When building a knowledge graph database, entity e and relation r are extracted from documents and formed into triples. <e1,r 12 For example, <"Pomegranate", "Builder", "Zhang San">, <"Hunter", "Builder", "Zhang San">, etc. "Document ID information", "modeling time", "model type" etc. are stored as attributes of the model entity. The triples are stored in the knowledge graph library to facilitate subsequent knowledge links between documents.

[0072] 2. Fine-grained semantic encoding. The user question is segmented into words and then transformed into an input sequence X = [x1, ..., x...]. n ] = ["Pomegranate", "Model", "Modeling", "of", "Time"], where x i Representing the i-th word, the named entity recognition technique is used to extract the entity 'e' from the sentence. i "Pomegranate". This approach uses entity concatenation input to enhance the weight of entity features in the retrieved feature sequence. The user question is encoded into a feature sequence t. 检索 ∈R1×d The formula is:

[0073] t 检索 =Embedding(NER(X))+Embedding(X) (2)

[0074] t 检索 =Embedding(e i )+Embedding(X) (3)

[0075] NER(·) is the entity extractor, implemented using the NLTK tool library.

[0076] 3. Retrieve relevant documents. Using vector similarity calculation, retrieve the top N documents (D) from the document vector database that have the highest similarity score to the user's question. 检索 ={D 检1 ,…,D 检N The formula for calculating vector similarity is:

[0077]

[0078] Here, Top N(·) represents the top N documents ranked by similarity score, and Sim(·) is the vector similarity calculation function, using cosine similarity here. (Statistical D) 检索 If the context length is greater than the input length of the generative model, proceed to step 4 to simplify the retrieved content through information compression; otherwise, directly use the retrieved relevant documents as part of the input to the generative model and proceed to step 6.

[0079] 4. Adaptive Information Compression. Using the document's index, retrieve documents from the information compression library. If C′ 检索 ={C′1,…,C′ N If the context length is greater than the input length of the generative model, dynamic information compression is performed. This involves evaluating the relevance value γ between the task and the retrieved documents and document fragments, and removing irrelevant document fragments using a comparison threshold as input to the generative model. A value of 1 (k) is used to retain data, while a value of 0 (k) is used to remove data. The formula is:

[0080] γ=Sim(t 检索 ,C′ i (5)

[0081]

[0082] 5. Knowledge Linking. Using entity e recorded in step 2... i Query for information about entity e i Spectral information Y E and entity e i Related document information YD The graph information contains k associated entities. Relationship between entities Document ID information, etc. This is achieved by analyzing the k entities E = {e1,…,e...} after NER(·). k Generate a graph query statement, set the query degree value for the graph database, and distribute the retrieved knowledge graph results according to the template {"e i and exist The article ID is 111",...," and exist The article ID is 112"} organized into natural language Y E Entity e i Related document information Y D Obtained by performing step 4. E and Y D As input to the generative model.

[0083] 6. Organize relevant knowledge. Summarize the results from steps 3, 4, and 5, and generate a new prompt based on the user's input question: "Please answer the user's question using content from the local knowledge base.\nContent from the knowledge base {C 检索 ,C′ 检索 ,Y e ,Y D User question: "When was the pomegranate model modeled?" Answer:

[0084] 7. Generate the answer. The generated model generates the final answer based on the generation instruction prompt from step 6: "The modeling time for the pomegranate model is June 2024."

[0085] This apparatus for enhancing semantic chain retrieval and generating knowledge-based question answering for long documents includes:

[0086] A processor is configured to execute computer-executable instructions;

[0087] The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the method described above for enhancing semantic chain retrieval and generating knowledge-based question answering for long documents.

[0088] The processor for knowledge question answering of long documents implements semantic chain retrieval enhancement generation processing, wherein the processor is configured to execute computer-executable instructions, and when the computer-executable instructions are executed by the processor, the steps of the above-described method for knowledge question answering of long documents implement semantic chain retrieval enhancement generation processing are implemented.

[0089] The computer-readable storage medium contains a computer program that can be executed by a processor to implement the steps of the method for enhancing semantic chain retrieval generation processing for knowledge question answering of long documents as described above.

[0090] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0091] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution device.

[0092] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0093] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0094] In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "embodiment," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0095] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

[0096] This invention employs a method, apparatus, processor, and computer-readable storage medium for semantic chain retrieval enhancement generation processing in long document knowledge question answering. Through a semantic chain-based RAG method, combined with fine-grained document segmentation, chained storage, information compression, knowledge graphs, and knowledge linking technologies, it achieves highly efficient optimization of semantic retrieval and context preservation in long document knowledge question answering. This method effectively solves the problems of semantic mismatch, context loss, and limited model context capacity in existing technologies for long document retrieval. It has the following technical advantages:

[0097] First, semantic chain storage optimizes context preservation. By fine-grainedly segmenting documents and constructing semantic chains, document fragments are organized into a chain structure according to semantic relationships, effectively maintaining the coherence of contextual information. Compared to traditional slice storage methods, semantic chains can significantly reduce context loss issues and improve the completeness and relevance of search results.

[0098] Second, information compression technology improves retrieval efficiency. By introducing feature extraction, dimensionality reduction, and relevance assessment techniques, information compression is applied to document fragments, significantly reducing storage and computational overhead. Through optimized retrieval efficiency, this method can quickly locate fragments highly relevant to the query semantics in large-scale long document databases, solving the semantic mismatch problem caused by insufficient vector similarity matching accuracy in existing methods.

[0099] Third, knowledge linking enhances semantic integrity. By associating semantic chains with knowledge graphs through knowledge linking technology, and utilizing entity recognition and relation extraction, key information from document fragments is mapped to knowledge graph nodes. This technology ensures the semantic integrity of retrieval results while enhancing the ability to handle complex semantic relationships.

[0100] Fourth, significant performance improvements and enhanced robustness. It significantly improves the ability to handle complex semantic relationships in long documents and enhances the model's robustness to data perturbations.

[0101] In this specification, the invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and variations can be made without departing from the spirit and scope of the invention. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

Claims

1. A method for enhancing semantic chain retrieval and generating long documents through knowledge question answering, characterized in that, The method includes the following steps: (1) Construct a document vector library, an information compression library, and a knowledge graph library; (2) Encode the user's input question with fine-grained semantics to obtain a retrieval vector containing fine-grained semantics; (3) Perform vector similarity calculation, retrieve the corresponding document with high similarity to the current user input question from the document vector library, and count the context length of the document. Perform adaptive information compression processing based on the document length. (4) Evaluate the correlation between user questions and retrieved documents, remove irrelevant document fragments, and use this as input for the generative model; (5) Identify entity information in the user input question, link it with the graph information corresponding to the entity in the knowledge graph library, and obtain the input information of the generation model; (6) Summarize all the semantic information obtained above to obtain the generation instructions based on the user input question; (7) The generative model generates the final answer to the question based on the generative instructions.

2. The method for enhancing semantic chain retrieval and generating long documents through knowledge question answering as described in claim 1, characterized in that, The document vector library is constructed in the following manner: (1.1.1) For content with a length of Document D i The document is divided into document fragments of different sizes according to the semantic sequence of the document. Ordinary documents are divided into document fragments according to chapters, while technical documents are divided into document fragments according to technical architecture. (1.1.2) Then organize the segmented semantic fragments into semantic chains: Among them, c i The document is a fragment, divided into [1, n] segments. To organize document fragments into semantic chains, c titlei c is the name of the i-th segment in the document. contenti This refers to the content of the i-th section; (1.1.3) Encode the semantic chain into a document vector: in, The encoded document vector, and d is the dimension of the vector, R is the set of real numbers, and Embedding(·) is the text vector encoder; (1.1.4) When importing document vectors into the database, follow the instructions... The document vector is stored as a retrieval unit in the document vector library.

3. The method for enhancing semantic chain retrieval and generating long documents through knowledge question answering as described in claim 1, characterized in that, The information compression library specifically involves: converting the context information of the current document into a compact representation, and compressing document fragments according to task requirements in the following manner: Static compression: Performed when a document is encoded and stored in the database, it includes retaining key information and removing redundant information. The static compression semantic chain is as follows: Where, c′ i This is a statically compressed document fragment. It is a semantic chain organized after static compression, including [1, n] segments; Dynamic compression: During the model generation process, when the size of the static compressed data retrieved from the knowledge base still exceeds the length of the model input, dynamic compression is adopted, and the process proceeds to step (4).

4. The method for enhancing semantic chain retrieval and generating long documents through knowledge question answering as described in claim 3, characterized in that, The knowledge graph database is constructed as follows: entity e and relation r are extracted from the user's question in the current document and formed into triples. <e1,r 12 ,e2>, where e1 is the first entity, e2 is the second entity, and r 12 To establish the relationship between e1 and e2, the entity includes its own attributes and document attributes. This triple is stored in the knowledge graph library to facilitate subsequent cross-document knowledge linking.

5. The method for enhancing semantic chain retrieval generation processing for knowledge question answering of long documents according to claim 4, characterized in that, Step (2) specifically includes the following steps: (2.1) The user question is converted into an input sequence X = [x1, ..., x] through word segmentation. i ,…,x n ], where x i For the i-th word; (2.2) Named entity recognition technology is used to extract all entities E = {e1, e2, ...} from the sentence; (2.3) The entity concatenation input method is adopted to concatenate the entity at the beginning of the sentence sequence, thereby enhancing the weight of entity features in the retrieval feature sequence; (2.4) Encode the user question into the retrieved feature sequence t in the following manner. 检索 : t 检索 =Embedding(NER(X)0+Embedding(X) t 检索 =Embedding(E)+Embedding(X) NER(·) is the entity extractor.

6. The method for enhancing semantic chain retrieval and generating long documents through knowledge question answering as described in claim 5, characterized in that, Step (3) specifically includes the following steps: (3.1) By calculating vector similarity, retrieve the top N documents with the highest similarity scores to the user's question from the document vector library: D 检索 ={D 检1 ,…,D 检N }, and record the index where the document is stored. The vector similarity is calculated in the following way: Where Top N(·) represents the top N documents ranked by similarity score, Sim(·) is the vector similarity calculation function, and D 检索 For the retrieved documents, This is the encoded document vector; (3.2) By statistical analysis of D 检索 The document's context length is checked against the input length of the generative model. If it is greater, the context length is reduced through static compression, and the compressed document is retrieved using its index. Otherwise, the relevant document is returned directly. If the document C' is being retrieved... 检索 ={C′1,…,C′ N If the context length is greater than the input length of the generated model, then dynamic information compression is performed, and step (4) is executed.

7. The method for enhancing semantic chain retrieval and generating long documents through knowledge question answering as described in claim 6, characterized in that, Step (4) specifically involves: evaluating the relevance value γ between the user's question and the retrieved documents, and removing irrelevant document fragments by comparing a threshold. And As input to the generative model: Where k is the criterion for whether to retain it.

8. The method for enhancing semantic chain retrieval and generating long documents through knowledge question answering as described in claim 5, characterized in that, Step (5) specifically involves: based on the obtained entity e i Information, query for information about entity e i Spectral information Y E and entity e i Related document information Y D Among them, the spectral information Y E It is a piece of natural language organized using templates, containing k related entities. Relationship between entities And document ID information; Y D The document is retrieved by document ID; the k entities E = {e1,…,e...} after processing by the entity extractor NER(·) are... k Generate a graph query statement, set the query degree value for the graph database, and organize the retrieved knowledge graph results into natural language Y according to a preset template. E The aforementioned spectral information Y E And document information Y D As input to the generative model.

9. An apparatus for enhancing semantic chain retrieval generation processing for knowledge question answering of long documents, characterized in that, The device includes: A processor is configured to execute computer-executable instructions; The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the method for semantic chain retrieval enhancement generation processing for knowledge question answering of long documents as described in any one of claims 1 to 8.

10. A processor for enhancing semantic chain retrieval generation processing for knowledge question answering of long documents, characterized in that, The processor is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the method for semantic chain retrieval enhancement generation processing for knowledge question answering of long documents as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method for semantic chain retrieval enhancement generation processing for knowledge question answering of long documents as described in any one of claims 1 to 8.

Images