Weighted attention mechanism and knowledge base fused intelligent question and answer technology
By constructing a structured triplet and a weighted attention mechanism for sentence selection, the problem of information overload in large language models during question answering is solved, achieving efficient and accurate answer generation and multi-step reasoning, and adapting to real-time updates in different domains.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-11
- Publication Date
- 2026-03-13
AI Technical Summary
Existing question-answering methods suffer from information overload and difficulty in accurately extracting relevant knowledge when using large-scale language models, leading to reduced accuracy and efficiency in answering questions.
Employing a weighted attention mechanism and knowledge base fusion technology, this approach generates answers by constructing structured triples and sentence selection, combined with a large language model. It utilizes the BERT model for sentence embedding and vector database retrieval, selecting the most relevant sentences and fusing them with triples to generate the answer.
It significantly improves the accuracy and efficiency of question answering, reduces information noise interference, enhances the performance of large language models in knowledge-intensive tasks, can quickly locate key facts and perform multi-step reasoning, and adapts to real-time updates in different domains.
Smart Images
Figure CN121658592A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing technology, and in particular to an intelligent question-answering technology that integrates a weighted attention mechanism and a knowledge base. Background Technology
[0002] Currently, document-based question answering methods are mainly divided into two categories: retrieval-based methods and generation-based methods. Retrieval-based methods retrieve documents from large-scale corpora that may contain the answer string, and then use the retrieved documents to generate the correct answer. Early research utilized sparse retrieval methods, while recent work has improved the performance of the retrieval engine to obtain more efficient documents, further improving the accuracy of the model's question answers. Generation-based methods do not rely on external knowledge but instead extract knowledge from the parameters of large language models to generate documents. Recent research shows that large-scale pre-trained models can form an implicit knowledge base containing a large amount of knowledge after pre-training.
[0003] While documents can provide additional knowledge to help answer questions, existing methods utilize all information within the document as supporting knowledge, which may introduce noise irrelevant to the query: firstly, it is highly likely to lead to information overload, thereby reducing the accuracy and efficiency of answering questions; secondly, given the large amount of content involved in the documents, if a large language model needs to process and understand the entire content, it may be difficult to accurately extract and utilize the knowledge relevant to the question. Based on this, we propose an intelligent question-answering technique that integrates a weighted attention mechanism and a knowledge base. Summary of the Invention
[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing an intelligent question-answering technology that integrates a weighted attention mechanism and a knowledge base.
[0005] To achieve the above objectives, the present invention adopts the following technical solution:
[0006] An intelligent question-answering technology that integrates a weighted attention mechanism and a knowledge base includes the following steps:
[0007] S1: Use triples to make effective selections of knowledge in documents;
[0008] S2: Extract the sentences in a document that are most relevant to the triples by sentence selection;
[0009] S3: Answer generation;
[0010] The formula for generating the answer is: A = LM(Q, T, S).
[0011] Preferably, the triples are constructed by using a large language model to generate structured triples based on natural language problems;
[0012] The structured triple is calculated as follows: T = {h} i ,r i ,t i};
[0013] Where i = 1, ..., m, m is the number of triples, h and t are the head entity and the tail entity respectively, and r represents the relationship between the head entity and the tail entity; and formally, T is obtained by the following formula: T = LM(Q).
[0014] Here, LM represents a specific large-scale language model, and Q represents the query question.
[0015] Preferred method for constructing the triplet includes the following:
[0016] A1: Enter your query question and identify the subject entities in the query question;
[0017] A2: Generate a set of information-rich triples based on the subject entity.
[0018] Preferably, the subject entity is an entity extracted from the subject of the query question, and the entity is one of the following: a person's name, a place name, an organization, or other key concepts;
[0019] The subject entity is used as the head entity, and multiple factual triples related to it are automatically completed using a large language model;
[0020] The aforementioned factual triples include the entity's attributes and relationships.
[0021] Preferably, in A1, during the identification of the subject entity, the large language model uses named entity recognition technology to locate representative core entities from the query text and ensure that they can be accurately mapped to unique identifiers in the knowledge graph or embedding space.
[0022] Preferably, the sentence selection process is implemented through a vector database.
[0023] Preferably, the extraction method is to use the BERT model to obtain the constructed triads and the embedding representation of each sentence in the document;
[0024] The embedded expression is: q = Bert(T), K = {k i |k i =Bert(s) i )};
[0025] Where q and k represent the embedding of the triple and the file, respectively.
[0026] Preferably, the specific steps of the extraction are as follows:
[0027] B1: The BERT model captures the semantic information and contextual features of sentences by encoding them into dense vectors;
[0028] ① The document is divided into several independent sentences, and these sentences and elements related to triples are vectorized and encoded using a pre-trained language model;
[0029] ② Store all sentence vectors in a vector database that supports approximate nearest neighbor retrieval;
[0030] B2: Calculate the Euclidean distance between each sentence and the triple based on the embedding representation, and select the k sentences with the closest distance as sentences;
[0031] The index of the sentence is calculated using the following formula:
[0032] L indicates Returns the indices of the k minimum values, where || represents the Euclidean distance;
[0033] The elements associated with the triple include the subject entity or the query text.
[0034] Preferably, the specific method for generating the answer is as follows:
[0035] 1) Integrate triples and sentence selection into a unified knowledge K. f And combine it with the query question;
[0036] 2) Use a large language model to obtain the final answer.
[0037] Preferably, the formula for the fused knowledge is: K f =α·F s (K s )+β·F t (K t );
[0038] Among them, F s With F t Let α and β represent the functions that map structured triples and unstructured sentences to the same semantic embedding space, respectively. α and β are the corresponding weight hyperparameters, satisfying α + β = 1, which are used to balance the contributions of the two to the final representation.
[0039] The beneficial effects of this invention are as follows:
[0040] 1. This invention deeply integrates two forms of knowledge: unstructured text sentences and structured triples. It facilitates interaction and information complementarity between the two, fully leveraging their complementary advantages in question-answering tasks. By uniformly encoding these two types of knowledge and inputting them as prompts into a large language model, the model can balance factual completeness and readability during the answering process, thus significantly improving the accuracy and consistency of the answer. It not only extracts key sentences from documents as natural language evidence but also retrieves triples related to the topic entity, ensuring that the model obtains both detailed contextual descriptions and clear logical relationship chains, thus assisting the large language model and improving the accuracy and efficiency of question answering.
[0041] 2. This invention transforms natural language queries into structured knowledge, enabling downstream models to more efficiently understand and reason about key steps. By identifying the subject entity in the query and generating an information-rich set of triples based on that entity, it provides the model with accurate contextual information (Zhihu column, Wikipedia). This can improve the performance of large language models in knowledge-intensive tasks, not only helping the model better understand the query question but also guiding large language models to perform complex reasoning, ultimately generating accurate and consistent answers.
[0042] 3. The generation of triples in this invention provides structured support for the model's subsequent reasoning and question answering, enabling it to quickly locate key facts, reduce retrieval overhead in massive amounts of unstructured text, and significantly improve the accuracy and consistency of answers (Zhihu column).
[0043] 4. In complex reasoning scenarios, this invention allows multiple triples to form chain-like knowledge, enabling the model to advance layer by layer along the logical chain of "topic-relationship-object" to complete multi-step deduction; moreover, the model can be adapted to different domains through hints or fine-tuning, achieving real-time completion and dynamic updates of new entities and facts, ensuring that the knowledge base is always synchronized with the latest information.
[0044] 5. This invention, through its sentence selection method, firstly significantly reduces the interference of irrelevant information on downstream model reasoning, making answer generation more accurate; secondly, vector retrieval based on the approximate nearest neighbor algorithm has extremely high query efficiency in large-scale scenarios, enabling rapid response; and finally, this method is inherently scalable, allowing for the dynamic addition of new sentence vectors and integration with multimodal retrieval, laying a solid foundation for complex multi-hop reasoning or cross-modal question answering.
[0045] 6. In this invention, when it is necessary to retrieve relevant sentences, the system converts the triples or query text into vectors and quickly performs a similarity search in the database to return the most matching sentences. This process not only efficiently locates the most valuable sentences in massive amounts of data, but also captures deep connections beyond keywords through semantic vector matching.
[0046] 7. Through sentence selection, this invention enables the question-answering system to provide the most direct and compelling evidence for triple completion and answer generation while ensuring relevance and speed. This provides accurate contextual information for subsequent answer generation, significantly improving the quality and accuracy of large language models in question-answering tasks. Compared to directly inputting the entire document into the model, sentence selection can eliminate redundant content irrelevant to the question, allowing the model to focus only on the most crucial evidence sentences, thereby reducing the interference of information noise on the reasoning process and improving the quality of responses. Attached Figure Description
[0047] Figure 1 This is a flowchart illustrating an intelligent question-answering technology that integrates a weighted attention mechanism and a knowledge base, as proposed in this invention.
[0048] Figure 2 This is a schematic diagram of the system flow of an intelligent question-answering technology that integrates a weighted attention mechanism and a knowledge base, as proposed in this invention. Detailed Implementation
[0049] The technical solution of this patent will be further described in detail below with reference to specific embodiments.
[0050] The embodiments of this patent are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this patent, and should not be construed as limiting this patent.
[0051] Example 1:
[0052] An intelligent question-answering technology that integrates a weighted attention mechanism and a knowledge base, such as Figure 1-2 As shown, it includes the following steps:
[0053] S1: Utilize triples to effectively select knowledge from documents; improve the performance of large language models in knowledge-intensive tasks, not only helping the model better understand the query question, but also guiding large language models to perform complex reasoning, ultimately generating accurate and consistent answers.
[0054] Furthermore, the triples are constructed by using a large language model to generate structured triples based on natural language problems;
[0055] Specifically, it includes the following:
[0056] A1: Input the query question and identify the subject entities in the query question; the subject entities can reflect the core intent of the query question.
[0057] Preferably, the subject entity is an entity that is related to the subject of the query question;
[0058] Further preferred entities include personal names, place names, organizations, or other key concepts. These entities can be individuals, locations, organizations, or other entity relationships that reflect the core content of the query question.
[0059] As a supplement, during the identification of topic entities, large language models use techniques such as Named Entity Recognition (NER) to locate representative core entities from the query text and ensure that they can be accurately mapped to unique identifiers in the knowledge graph or embedding space.
[0060] For example, when the query is "What is John's zodiac sign?", the subject entity is explicitly "John", which directly determines the starting point for the retrieval and reasoning required for the subsequent construction of triples.
[0061] A2: Generate a set of information-rich triples based on the topic entity; these triples cover various aspects of knowledge closely related to the query question, providing contextual information to the model from multiple perspectives. In other words, after the topic entity is determined, the triple generation stage begins. At this time, the large language model will automatically generate triples such as (John, birthDate, 1958-11-22) or (John, occupation, actress) based on pre-trained knowledge and contextual information to cover various factual information related to the topic entity, including basic attributes, subordinate relationships, and related people or events.
[0062] Preferably, the subject entity is used as the head entity (Subject, S), and a large language model is used to automatically complete multiple factual triples related to it;
[0063] Further optimization involves multiple factual triples, including entity attributes and relationships, thereby constructing a set of triples covering multi-faceted knowledge.
[0064] As a supplement, the structured triple is calculated as: T = {h} i ,r i ,t i};
[0065] Where i = 1, ..., m, m is the number of triples, h and t are the head entity and the tail entity respectively, and r represents the relationship between the head entity and the tail entity; and formally, T is obtained by the following formula: T = LM(Q).
[0066] Here, LM represents a specific large-scale language model, and Q represents the query question.
[0067] The triple construction process is a key step in transforming natural language queries into structured knowledge, enabling downstream models to understand and reason more efficiently. By identifying thematic entities in the query and generating an information-rich set of triples based on these entities, the model receives precise contextual information. In this invention, triple generation provides structured support for subsequent reasoning and question answering, enabling the model to quickly locate key facts, reduce retrieval overhead in massive amounts of unstructured text, and significantly improve the accuracy and consistency of answers.
[0068] In complex reasoning scenarios, multiple triples can form chain-like knowledge, allowing the model to advance layer by layer along the logical chain of "topic-relationship-object" to complete multi-step deductions. For example, first obtain the birth year from (John, birthDate, 1958-11-22), and then infer the winners of the same year from relevant award triples. The modern triple construction process is highly automated, leveraging the end-to-end capabilities of large language models to extract topic entities and complete triples without human intervention. The model can adapt to different domains through prompts or fine-tuning, achieving real-time completion and dynamic updates of new entities and facts, ensuring that the knowledge base is always synchronized with the latest information.
[0069] S2: Extract the most relevant sentences to the triples in the document through sentence selection; through sentence selection, the question answering system can provide the most direct and powerful evidence support for triple completion and answer generation while ensuring relevance and speed; significantly improve the quality and accuracy of large language models in question answering tasks.
[0070] Preferably, the sentence selection process is implemented through a vector database, which has the advantage of high efficiency. The sentence selection method supported by the vector database of this invention has significant advantages: First, it greatly reduces the interference of irrelevant information on downstream model reasoning, making the answer generation more accurate; second, vector retrieval based on the approximate nearest neighbor algorithm has extremely high query efficiency in large-scale scenarios and can respond quickly; finally, this method is inherently scalable, and can dynamically add new sentence vectors and can be combined with multimodal retrieval, laying a solid foundation for complex multi-hop reasoning or cross-modal question answering.
[0071] Furthermore, the extraction method is as follows: using the BERT model to obtain the constructed triads and the embedding representation of each sentence in the document;
[0072] Preferably, the embedding expression is: q = Bert(T), K = {k i |k i =Bert(s) i )};
[0073] Where q and k represent the embedding and file of the triple, respectively;
[0074] Specifically, the extraction steps are as follows:
[0075] B1: The BERT model captures the semantic information and contextual features of sentences by encoding them into dense vectors;
[0076] Preferably, the specific steps are: ① dividing the document into several independent sentences, and using a pre-trained language model to vectorize and encode these sentences and elements related to the triples;
[0077] Further preferred elements associated with the triple include the main entity or the query text;
[0078] ② Store all sentence vectors in a vector database that supports approximate nearest neighbor retrieval.
[0079] B2: Calculate the Euclidean distance between each sentence and the triple based on the embedding representation, and select the k sentences with the closest distance as sentences; in order to measure semantic similarity.
[0080] The index of a sentence is calculated using the following formula:
[0081] L indicates Returns the indices of the k smallest values, where || represents the Euclidean distance.
[0082] When retrieving relevant sentences, the system converts the triples or query text into vectors and quickly performs a similarity search in the database to return the most matching sentences. This process not only efficiently locates the most valuable sentences in massive amounts of data but also captures deeper connections beyond keywords through semantic vector matching.
[0083] In the S2 sentence selection step, sentences most relevant to the triples can be extracted from a large number of documents. These sentences contain key information related to the query question and provide supporting knowledge for subsequent answer generation. Furthermore, compared to directly using the entire document as sentences, this effective sentence selection eliminates irrelevant information in the document that may hinder the answer. In the entire question-answering system of this invention, sentence selection involves filtering sentences from a large number of documents that are most relevant to the constructed triples, providing accurate contextual information for subsequent answer generation. Compared to directly inputting the entire document into the model, sentence selection can eliminate redundant content irrelevant to the question, allowing the model to focus only on the most crucial evidence sentences, thereby reducing the interference of information noise on the reasoning process and improving the quality of the answer.
[0084] S3: Answer generation;
[0085] Preferably, the formula for generating the answer is: A = LM(Q,T,S);
[0086] Furthermore, the specific method for generating the answer is as follows:
[0087] 1) Integrate triples and sentence selection into a unified knowledge K. f And combine it with the query question;
[0088] Preferred, K f =α·F s (K s )+β·F t (K t );
[0089] Among them, F s With F t Let α and β represent the functions that map structured triples and unstructured sentences to the same semantic embedding space (e.g., based on graph neural networks and sentence embedding models), respectively. α and β are the corresponding weight hyperparameters, satisfying α+β=1, used to balance the contributions of the two to the final representation.
[0090] 2) Utilizing large-scale language models to obtain the final answer. Triples provide structured knowledge, while sentence selection provides detailed information from the document in text format. Effectively combining the advantages of triples and sentence selection, and integrating them with the query question, allows for the fusion of multiple forms of knowledge at different granularities, providing the model with richer context and factual knowledge, thus promoting the generation of more accurate and consistent final answers by large-scale language models.
[0091] In this embodiment, unstructured text sentences and structured triples are deeply integrated to facilitate interaction and information complementarity between the two, fully leveraging their complementary advantages in question-answering tasks. By uniformly encoding these two types of knowledge and feeding them as prompts into a large language model, the model can balance factual completeness and readability during the answering process, thereby significantly improving the accuracy and consistency of the answers.
[0092] This invention not only extracts key sentences from documents as natural language evidence, but also retrieves triples related to the subject entity to ensure that the model can obtain both detailed contextual descriptions and clear logical relationship chains, thereby assisting large language models and improving the accuracy and efficiency of answering questions.
[0093] This invention dynamically controls the proportion of structured and unstructured knowledge in the final prompt through hyperparameters (such as α and β), enabling the same system to easily adapt to different types of question-answering tasks: for scenarios that emphasize logical reasoning, the weight of triples can be appropriately increased; for scenarios that require more contextual and detailed support, the weight of text sentences can be increased. Compared with the rigid configuration of existing technologies (text retrieval and triple retrieval are often performed in parallel and independently), the fusion mechanism of this invention is more flexible and easier to expand and optimize.
[0094] Example 2:
[0095] An intelligent question-answering technology that integrates a weighted attention mechanism and a knowledge base is proposed. This embodiment improves upon Embodiment 1 by performing joint reasoning between a large language model and a knowledge graph within a deep network, rather than simply integrating them at the prompting level.
[0096] Taking QA-GNN as an example: First, a subgraph is constructed in the knowledge graph based on the retrieved triples. Relevant entities and relation nodes are mapped to the input of a graph neural network (GNN) (node features can be obtained by weighted embeddings from a pre-trained language model). Then, the question context is also mapped to nodes or vectors, and information is exchanged between the GNN and graph nodes across multiple layers. Finally, the model completes an end-to-end joint inference on the graph, outputting an answer score or generating an answer. GreaseLM further inserts multiple layers of "modal interaction": the LM context representation and the graph node representation flow between each layer, allowing the text context to "constrain" the graph structure, and vice versa.
[0097] Compared to Example 1, this embodiment preserves the multi-step iterative interaction between graphs and text more effectively by using prompt-level weighted fusion vectors. It is suitable for tasks requiring more complex chain-like reasoning, but the training cost and model architecture are usually more complex.
[0098] Experimental Example 1:
[0099] The table below compares the results of tests between the present invention and existing technologies (intelligent question answering) in terms of response time and accuracy:
[0100] Technology type Existing technology This invention Increase efficiency by % Response time for simple questions (s) 2.1 0.7 66 Response time (s) for complex problems 28 3 89 Single-round fact-finding accuracy rate (%) 70 96 37 Multi-turn interactive question-and-answer accuracy rate (%) 55 93 69 Inference accuracy rate in specialized fields (%) 68 95 40
[0101] The results show that the present invention has improved upon existing technologies in terms of problem response time and question-answering accuracy.
[0102] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. An intelligent question-answering technology that integrates a weighted attention mechanism and a knowledge base, characterized in that, Includes the following steps: S1: Use triples to make effective selections of knowledge in documents; S2: Extract the sentences in a document that are most relevant to the triples by sentence selection; S3: Answer generation; The formula for generating the answer is: A = LM(Q, T, S).
2. The intelligent question-answering technology that integrates a weighted attention mechanism and a knowledge base according to claim 1, characterized in that, The triples are constructed by using a large language model to generate structured triples based on natural language problems; The structured triple is calculated as follows: T = {h} i ,r i ,t i }; Where i = 1, ..., m, m is the number of triples, h and t are the head entity and the tail entity respectively, and r represents the relationship between the head entity and the tail entity; and formally, T is obtained by the following formula: T = LM(Q). Here, LM represents a specific large-scale language model, and Q represents the query question.
3. The intelligent question-answering technology that integrates a weighted attention mechanism and a knowledge base according to claim 2, characterized in that, The method for constructing the triplet specifically includes the following: A1: Enter your query question and identify the subject entities in the query question; A2: Generate a set of information-rich triples based on the subject entity.
4. The intelligent question-answering technology that integrates a weighted attention mechanism and a knowledge base according to claim 3, characterized in that, The subject entity is an entity extracted and related to the query question topic. The entity is one of the following: a person's name, a place name, an organization, or other key concepts. The subject entity is used as the head entity, and multiple factual triples related to it are automatically completed using a large language model; The aforementioned factual triples include the entity's attributes and relationships.
5. The intelligent question-answering technology that integrates a weighted attention mechanism and a knowledge base according to claim 4, characterized in that, In A1, during the identification of topic entities, a large language model uses named entity recognition technology to locate representative core entities from the query text and ensure that they can be accurately mapped to unique identifiers in the knowledge graph or embedding space.
6. The intelligent question-answering technology that integrates a weighted attention mechanism and a knowledge base according to claim 1, characterized in that, The sentence selection process is implemented using a vector database.
7. The intelligent question-answering technology that integrates a weighted attention mechanism and a knowledge base according to claim 6, characterized in that, The extraction method is as follows: using the BERT model to obtain the constructed triads and the embedding representation of each sentence in the document; The embedded expression is: q = Bert(T), K = {k i |k i =Bert(s) i )}; Where q and k represent the embedding of the triple and the file, respectively.
8. The intelligent question-answering technology that integrates a weighted attention mechanism and a knowledge base according to claim 7, characterized in that, The specific steps for extraction are as follows: B1: The BERT model captures the semantic information and contextual features of sentences by encoding them into dense vectors; ① The document is divided into several independent sentences, and these sentences and elements related to triples are vectorized and encoded using a pre-trained language model; ② Store all sentence vectors in a vector database that supports approximate nearest neighbor retrieval; B2: Calculate the Euclidean distance between each sentence and the triple based on the embedding representation, and select the k sentences with the closest distance as sentences; The index of the sentence is calculated using the following formula: L indicates Returns the indices of the k minimum values, where || represents the Euclidean distance; The elements associated with the triple include the subject entity or the query text.
9. The intelligent question-answering technology that integrates a weighted attention mechanism and a knowledge base according to claim 1, characterized in that, The specific method for generating the answer is as follows: 1) Integrate triples and sentence selection into a unified knowledge K. f And combine it with the query question; 2) Use a large language model to obtain the final answer.
10. The intelligent question-answering technology that integrates a weighted attention mechanism and a knowledge base according to claim 9, characterized in that, The formula for the fused knowledge is: K f =α·F s (K s )+β·F t (K t ); Among them, F s With F t Let α and β represent the functions that map structured triples and unstructured sentences to the same semantic embedding space, respectively. α and β are the corresponding weight hyperparameters, satisfying α + β = 1, which are used to balance the contributions of the two to the final representation.