A temporal knowledge question answering method based on sentence granularity hints for large language models

Through the large language model timing knowledge question and answer method based on sentence granularity prompts, the problem of multi-grained time and complex time reasoning in timing knowledge graph questions and answers is solved, and the semantics of sentence granularity are fully mined and the accuracy of answers is improved. Combined with LoRA Method to quickly improve model performance.

CN117271740BActive Publication Date: 2024-11-29NO 15 INST OF CHINA ELECTRONICS TECH GRP +1
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202311307643.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-11
Publication Date
2024-11-29
Estimated Expiration
2043-10-11

AI Technical Summary

Technical Problem

Time series knowledge graph question and answer are difficult in multi-grained time and complex time reasoning, and the existing methods are insufficient in terms of sentence granularity and semantic information.

Method used

Using the time sequence knowledge question and answer method of large language model based on sentence granularity prompts, by transforming the time sequence knowledge graph document blocks into sentence granularity, generating vectors, and matching them with the question vectors, a variety of propt templates are constructed to improve the accuracy of answers, combined with the large language model and LoRA weight adjustments to improve semantic comprehension.

Benefits of technology

Effectively mine the semantic information contained in sentence granularity, improve the accuracy of the answers, and further improve the accuracy of the output answers through the super semantic understanding ability of multiple prompt templates and large language models, significantly reducing the number of parameter adjustments to quickly improve model performance .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117271740B_ABST
    Figure CN117271740B_ABST
Patent Text Reader

Abstract

The present invention discloses a large language model sequential knowledge question-answering method based on sentence granularity prompts, comprising the following steps: obtaining sequential knowledge graph documents based on a local knowledge base path; slicing the obtained sequential knowledge graph documents to obtain N sequential knowledge graph document blocks; performing sentence granularity transformation on the N sequential knowledge graph document blocks and inputting them into an embedding model to obtain N sequential knowledge graph document block vectors; inputting a question into the embedding model to obtain a question vector; based on the question vector and the N sequential knowledge graph document block vectors, selecting the K sequential knowledge graph document block vectors with the highest similarity; filling the original text and question corresponding to the K sequential knowledge graph document block vectors into a pre-built prompt template to obtain a final prompt template; inputting the final prompt template into a large language model to output the answer corresponding to the question. The present invention can improve the accuracy of the output answer.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of language processing technology, and more specifically to a large language model temporal knowledge question answering method based on sentence granularity prompts. Background Art

[0002] Knowledge graph question answering uses the knowledge graph as a knowledge base to answer natural language questions. The answer to the question is usually an entity in the knowledge graph, while the time series knowledge graph adds time series information to the traditional knowledge graph.

[0003] Compared with knowledge graph question answering, time series knowledge graph question answering has two major difficulties. First, the answer to time series knowledge graph question answering contains not only entities, but also multi-granularity time, such as year, month, and day, which greatly increases the complexity of the task. Second, time series knowledge graph question answering requires complex temporal reasoning. Time series knowledge graph question answering includes multi-hop questions about time, and the questions often contain constraints such as "before", "after", "earliest", and "latest". Answering such questions mostly requires complex temporal reasoning, which undoubtedly increases the difficulty of answering questions.

[0004] Traditional time-series knowledge graph question-answering methods often choose to extract entities and relationships from a specific knowledge base, fill the obtained content into a template to generate an answer, or reason about the answer through a deep learning model:

[0005] The first method is a rule- and template-based method that matches questions and answers by defining some predefined rules to achieve question-answering on the knowledge graph. The second method is an information retrieval-based method that uses a deep learning model to generate embedding vectors from large-scale unlabeled data, maps answers to the same space, and matches answers based on similarity, dealing with the complexity of natural language expressions.

[0006] However, the core of the rule- and template-based method lies in the construction of a template library, which requires a lot of work in advance. Therefore, this method is usually applicable to some simple and structured question-answering tasks, and has poor usability in multi-granularity temporal knowledge question-answering.

[0007] The information retrieval-based method uses deep learning models such as BERT and Transformer to generate embedding vectors from large-scale unlabeled data, map the answers to the same space, and match the answers based on similarity. However, this method only compares relevance at the word level and ignores the semantic information contained in the sentence level.

[0008] Therefore, how to provide a knowledge question answering method based on sentence granularity prompts is an urgent problem that technicians in this field need to solve. Summary of the invention

[0009] In view of this, the purpose of the present invention is to provide a large language model temporal knowledge question answering method based on sentence granularity prompts.

[0010] In order to achieve the above object, the present invention adopts the following technical solution:

[0011] A large language model temporal knowledge question answering method based on sentence granularity prompts, comprising the following steps:

[0012] S1: Based on the local knowledge base path, obtain the temporal knowledge graph document stored in the path;

[0013] Slice the acquired temporal knowledge graph document to obtain N temporal knowledge graph document blocks, where N ≥ 1;

[0014] The N temporal knowledge graph document blocks are transformed into sentence granularity and then input into the embedding model to obtain N temporal knowledge graph document block vectors;

[0015] Inputting the question into the embedding model to obtain a question vector;

[0016] S2: Based on the question vector and the N temporal knowledge graph document block vectors, select the K temporal knowledge graph document block vectors with the highest similarity; where 1≤K≤N;

[0017] S3: Fill the original text corresponding to the K temporal knowledge graph document block vectors and the question into the pre-built prompt template to obtain the final prompt template;

[0018] S4: Input the final prompt template into the large language model, and output the answer corresponding to the question.

[0019] Preferably, S1 further comprises:

[0020] If the local knowledge base path stores a document, the document is directly read as the temporal knowledge graph document;

[0021] If the local knowledge base path stores a folder, all documents in the folder are read and merged as the temporal knowledge graph document.

[0022] Preferably, S1 further comprises:

[0023] The temporal knowledge graph is stored in the local knowledge base in the form of a four-tuple - "head entity relationship tail entity time";

[0024] The method for sentence granularity transformation is:

[0025] The word on is added between the tail entity and the time, and the quadruple after the word on is added is transformed into a character string.

[0026] Preferably, the method of transforming the four-tuple after adding the word on into a character string includes: deleting special characters in the four-tuple;

[0027] The special characters include "-" and "_".

[0028] Preferably, S2 further comprises:

[0029] Calculate the cosine similarity between the question vector and the N temporal knowledge graph document block vectors, and select the K temporal knowledge graph document block vectors with the highest cosine similarity.

[0030] Preferably, S3 further comprises:

[0031] The pre-built prompt template includes a first basic template, a first one-shot learning template, a first few-shot learning template, a second basic template, a second one-shot learning template and a second few-shot learning template;

[0032] The first basic template is a template that does not provide examples and limits the large language model to output only one answer;

[0033] The first one-shot learning template is a template that provides an example and limits the large language model to output only one answer;

[0034] The first few-shot learning template is a template that provides multiple examples and limits the large language model to output only one answer;

[0035] The second basic template is a template that does not provide examples and does not limit the number of answers output by the large language model;

[0036] The second one-shot learning template is a template that provides an example and does not limit the number of answers output by the large language model;

[0037] The second few-shot learning template is a template that provides multiple examples and does not limit the number of answers output by the large language model.

[0038] Preferably, S3 further comprises:

[0039] The example is constructed as follows:

[0040] Based on the entity and time of the question, obtain an example of the question from the knowledge graph in a string matching manner;

[0041] The examples are used to prompt the large prediction model to output the correct answer.

[0042] Preferably, S3 further comprises:

[0043] Replace the original text corresponding to the K temporal knowledge graph document blocks with "{context}" in the pre-built prompt template;

[0044] Replace "{question}" in the pre-built prompt template with the question.

[0045] Preferably, S3 further comprises:

[0046] If the temporal knowledge graph document described in S1 is an unstructured document, the original text corresponding to the K temporal knowledge graph document block vectors, the original text corresponding to the adjacent document block vectors of the K temporal knowledge graph document block vectors, and the question are filled into the pre-constructed prompt template to obtain the final prompt template.

[0047] Preferably, S4 further comprises:

[0048] The large language model is a pre-trained language model; the pre-trained language model uses the LoRA method to adjust the weight parameters;

[0049] The output of the pre-trained language model is:

[0050] y=W.x+W A .W B .x (2)

[0051] Wherein, x is the input of the pre-trained language model, y is the output of the pre-trained language model, and W is the weight matrix of the pre-trained language model; W A and W B is the rank decomposition matrix of W, that is, W = W A .W B ;W∈R d×k , W A ∈R r×k , W B ∈R d×r ; R represents a matrix with a rank r<<min(d, k); d and k represent the number of rows and columns of the weight matrix W respectively.

[0052] Preferably, the calculation formula of the cosine similarity is:

[0053]

[0054] Among them, A represents the question vector, and B represents a vector among the K temporal knowledge graph document block vectors.

[0055] It can be seen from the above technical solution that, compared with the prior art, the present invention discloses a large language model temporal knowledge question answering method based on sentence granularity prompts, which can achieve the following beneficial technical effects:

[0056] 1) The present invention transforms the knowledge graph document block into sentence granularity, which can fully mine the semantic information contained in the sentence granularity and improve the accuracy of the output answer.

[0057] 2) The present invention constructs a variety of sentence-granularity prompt templates (i.e., prompt templates) for the questions that need to be answered, and relies on the super-strong semantic understanding ability of the large language model to obtain the final answer from the sentence-granularity prompts, which further improves the accuracy of the output answer.

[0058] 3) The large language model (i.e., pre-trained language model) of the present invention adopts the LoRA (low rank adaptation) method to adjust the weight parameters, which can significantly reduce the number of adjustment parameters, thereby quickly improving the semantic understanding ability of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0060] Figure 1 It is a flow chart of the overall method of the present invention;

[0061] Figure 2 A flowchart for obtaining a document block vector library for the present invention;

[0062] Figure 3 The sentence granularity transformation process and results for a certain four-tuple;

[0063] Figure 4 To prompt learning related schematic diagrams;

[0064] Figure 5 Learn relevant schematics for context;

[0065] Figure 6 This is a schematic diagram of LoRA fine-tuning of the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] like Figure 1 As shown, the embodiment of the present invention discloses a large language model temporal knowledge question answering method based on sentence granularity prompts, comprising the following steps:

[0068] S1: Based on the local knowledge base path, obtain the temporal knowledge graph document stored in the path;

[0069] Further:

[0070] If the local knowledge base path stores a document, the document is directly read as the temporal knowledge graph document;

[0071] If the local knowledge base path stores a folder, all documents in the folder are read and merged as the temporal knowledge graph document.

[0072] Slice the acquired temporal knowledge graph document to obtain N temporal knowledge graph document blocks, where N ≥ 1;

[0073] The N temporal knowledge graph document blocks are transformed into sentence granularity and then input into the embedding model to obtain N temporal knowledge graph document block vectors;

[0074] The N temporal knowledge graph document block vectors constitute a temporal knowledge graph document block vector library.

[0075] Figure 2 The construction process of the temporal knowledge graph document block vector library is shown.

[0076] Taking the document block "Xiao Ming hosted the 2023 annual meeting" as an example, the solution process of the document block vector is described in detail:

[0077] Input "Xiao Ming hosted the 2023 Annual Meeting" into the embedding model to get the vector of each word:

[0078] 1) Xiao Ming: {0.7 0.6 0.7};

[0079] 2) Host: {0.3 0.4 0.3};

[0080] 3) 2023: {0.5 0.5 0.8};

[0081] 4) Annual meeting: {0.1 0.9 0.6};

[0082] 5) Calculate the average value of each word in each dimension to obtain the vector of the document block "Xiao Ming hosted the 2023 annual meeting": {0.4 0.6 0.6};

[0083] in:

[0084] 0.4 = (0.7 + 0.3 + 0.5 + 0.1) / 4;

[0085] 0.6 = (0.6 + 0.4 + 0.5 + 0.9) / 4;

[0086] 0.6=(0.7+0.3+0.8+0.6)) / 4.

[0087] Further:

[0088] The temporal knowledge graph is stored in the local knowledge base in the form of a four-tuple - "head entity relationship tail entity time";

[0089] The method for sentence granularity transformation is:

[0090] The word on is added between the tail entity and the time, and the quadruple after the word on is added is transformed into a character string.

[0091] Further:

[0092] The method of transforming the four-tuple after adding the word on into a string includes: deleting special characters in the four-tuple;

[0093] The special characters include "-" and "_".

[0094] Figure 3 The sentence granularity transformation process and results of a certain quadruple are shown.

[0095] Inputting the question into the embedding model to obtain a question vector;

[0096] S2: Based on the question vector and the N temporal knowledge graph document block vectors, select the K temporal knowledge graph document block vectors with the highest similarity; where 1≤K≤N;

[0097] Further:

[0098] Calculate the cosine similarity between the question vector and the N temporal knowledge graph document block vectors, and select the K temporal knowledge graph document block vectors with the highest cosine similarity.

[0099] Further:

[0100] The calculation formula of the cosine similarity is:

[0101]

[0102] Among them, A represents the question vector, and B represents a vector among the K temporal knowledge graph document block vectors.

[0103] S3: Fill the original text corresponding to the K temporal knowledge graph document block vectors and the question into the pre-built prompt template to obtain the final prompt template;

[0104] Further:

[0105] Replace the original text corresponding to the K temporal knowledge graph document blocks with "{context}" in the pre-built prompt template;

[0106] Replace "{question}" in the pre-built prompt template with the question.

[0107] Further:

[0108] The pre-built prompt template includes a first basic template, a first one-shot learning template, a first few-shot learning template, a second basic template, a second one-shot learning template and a second few-shot learning template;

[0109] The first basic template is a template that does not provide examples and limits the large language model to output only one answer;

[0110] The first one-shot learning template is a template that provides an example and limits the large language model to output only one answer;

[0111] The first few-shot learning template is a template that provides multiple (five in one embodiment) examples and limits the large language model to output only one answer;

[0112] The second basic template is a template that does not provide examples and does not limit the number of answers output by the large language model;

[0113] The second one-shot learning template is a template that provides an example and does not limit the number of answers output by the large language model;

[0114] The second few-shot learning template is a template that provides multiple (five in one embodiment) examples and does not limit the number of answers output by the large language model.

[0115] The following table is a prompt template constructed by the present invention:

[0116]

[0117]

[0118]

[0119]

[0120] Further:

[0121] The example is constructed as follows:

[0122] Based on the entity and time of the question, obtain an example of the question from the knowledge graph in a string matching manner;

[0123] The knowledge graph mentioned here can be the knowledge graph corresponding to the question or other knowledge graphs.

[0124] The examples are used to prompt the large prediction model to output the correct answer.

[0125] In order to more clearly understand the role of example prompts, the concepts of prompt learning and context learning are explained below:

[0126] 1) Prompt learning concepts

[0127] Prompted learning is a method of guiding a language model to perform a specific task by designing natural language prompts or instructions. Figure 4 As shown in the figure, the concept of prompt learning is explained by taking the sentiment classification task as an example: when using a masked language model to solve the text sentiment classification task, for the input of "I love this poem.", you can add a prompt template such as "The poem is___" after the input, and then let the masked language model fill in the blanks with words representing emotions such as "wonderful", "terrible", etc., and finally convert the words into sentiment classification labels. In this way, by selecting a suitable prompt template, the output of the model can be controlled, so that a masked language model trained on a general knowledge dataset can be used to handle downstream tasks such as text sentiment classification.

[0128] 2) Contextual Learning Concepts

[0129] As a special form of prompt learning, contextual learning was first applied in GPT3. Its core idea is to learn from analogies. Based on the generalization ability of large language models, contextual learning only needs a few examples to enable the model to quickly adapt to the downstream tasks required. Figure 5 As shown in Figure 1, with only K examples of downstream problems, a large language model pre-trained on general knowledge data can be quickly adapted to the text sentiment classification task.

[0130] Based on the ideas of prompt learning and context learning, the present invention constructs several examples for the questions that need to be answered to improve the accuracy of the output results of the large language model.

[0131] Further:

[0132] If the temporal knowledge graph document described in S1 is an unstructured document, the original text corresponding to the K temporal knowledge graph document block vectors, the original text corresponding to the adjacent document block vectors of the K temporal knowledge graph document block vectors, and the question are filled into the pre-constructed prompt template to obtain the final prompt template.

[0133] The neighboring document vectors include a left neighboring document vector and / or a right neighboring document vector.

[0134] S4: Input the final prompt template into the large language model, and output the answer corresponding to the question.

[0135] Further:

[0136] The large language model is a pre-trained language model; the pre-trained language model uses the LoRA method to adjust the weight parameters;

[0137] The output of the pre-trained language model is:

[0138] y=W.x+W A .W B .x (2);

[0139] Wherein, x is the input of the pre-trained language model, y is the output of the pre-trained language model, and W is the weight matrix of the pre-trained language model; W A and W B is the rank decomposition matrix of W, that is, W = W A ·W B ;W∈R d×k , W A ∈R r ×k , W B ∈R d×r ; R represents a matrix with a rank r<<min(d, k); d and k represent the number of rows and columns of the weight matrix W respectively.

[0140] It should be noted that Low-Rank Adaptation (LoRA) is an efficient fine-tuning method for large language models, which aims to solve the problems of slow fine-tuning speed and high computational overhead of large language models. Figure 6As shown in Figure 2, when LoRA is fine-tuned, the weights of the pre-trained language model are frozen and a trainable layer, namely the rank decomposition matrix, is injected into each Transformer block. The rank decomposition matrix consists of a dimensionality reduction matrix A and a dimensionality increase matrix B. The A matrix is ​​initialized by a random Gaussian distribution, and the B matrix is ​​initialized by a zero matrix to ensure that the rank decomposition matrix is ​​a zero matrix at the beginning of training. As shown in formula (2), the input and output dimensions of the model remain unchanged during LoRA fine-tuning. Finally, the output of the pre-trained language model is the output W of the rank decomposition matrix branch. A .W B .x is the sum of the output Wx of the pre-trained language model weight matrix branch;

[0141] Compared with traditional fine-tuning, which requires training the parameters of W, LoRA fine-tuning only trains W A and W B The parameters of W, where the rank r<<min(d, k), therefore the parameters that need to be adjusted for LoRA fine-tuning are far less than those for traditional fine-tuning. A and W B The semantic understanding ability of the large language model can be quickly improved by simply changing the parameters of the pre-trained language model without modifying the weight matrix parameters of the pre-trained language model. LoRA fine-tuning can significantly reduce the number of adjustment parameters, and the performance is basically the same as full parameter fine-tuning.

[0142] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0143] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large language model temporal knowledge question answering method based on sentence granularity prompts, characterized in that: The following steps are involved: S1: Based on the local knowledge base path, obtain the temporal knowledge graph document stored in the path; Slice the acquired temporal knowledge graph document to obtain N temporal knowledge graph document blocks, where N ≥ 1; The N temporal knowledge graph document blocks are transformed into sentence granularity and then input into the embedding model to obtain N temporal knowledge graph document block vectors; Inputting the question into the embedding model to obtain a question vector; S1 further includes: The temporal knowledge graph is stored in the local knowledge base in the form of a four-tuple - "head entity relationship tail entity time"; The method for sentence granularity transformation is: Add the word on between the tail entity and the time, and transform the quadruple after adding the word on into a string; The method of transforming the four-tuple after adding the word on into a string includes: deleting special characters in the four-tuple; The special characters include "-" and "_"; S2: Based on the question vector and the N temporal knowledge graph document block vectors, select the K temporal knowledge graph document block vectors with the highest similarity; where 1≤K≤N; S3: Fill the original text corresponding to the K temporal knowledge graph document block vectors and the question into the pre-built prompt template to obtain the final prompt template; S3 further includes: The pre-built prompt template includes a first basic template, a first one-shot learning template, a first Few-shot learning template, a second basic template, a second one-shot learning template and a second Few-shot learning template; The first basic template is a template that does not provide examples and limits the large language model to output only one answer; The first one-shot learning template is a template that provides an example and limits the large language model to output only one answer; The first few-shot learning template is a template that provides multiple examples and limits the large language model to output only one answer; The second basic template is a template that does not provide examples and does not limit the number of answers output by the large language model; The second one-shot learning template is a template that provides an example and does not limit the number of answers output by the large language model; The second few-shot learning template is a template that provides multiple examples and does not limit the number of answers output by the large language model; The example is constructed as follows: Based on the entity and time of the question, obtain an example of the question from the knowledge graph in a string matching manner; The examples are used to prompt the large prediction model to output the correct answer; Replace "{context}" in the pre-built prompt template with the original text corresponding to the K temporal knowledge graph document blocks; Replace "{question}" in the pre-built prompt template with the question; S4: Input the final prompt template into the large language model, and output the answer corresponding to the question.

2. The large language model temporal knowledge question answering method based on sentence granularity prompts according to claim 1 is characterized in that: S1 further includes: If the local knowledge base path stores a document, the document is directly read as the temporal knowledge graph document; If the local knowledge base path stores a folder, all documents in the folder are read and merged as the temporal knowledge graph document.

3. The large language model temporal knowledge question answering method based on sentence granularity prompts according to claim 1 is characterized in that: S2 further includes: Calculate the cosine similarity between the question vector and the N temporal knowledge graph document block vectors, and select the K temporal knowledge graph document block vectors with the highest cosine similarity.

4. The large language model temporal knowledge question answering method based on sentence granularity prompts according to claim 1 is characterized in that: S3 further includes: If the temporal knowledge graph document described in S1 is an unstructured document, the original text corresponding to the K temporal knowledge graph document block vectors, the original text corresponding to the adjacent document block vectors of the K temporal knowledge graph document block vectors, and the question are filled into the pre-constructed prompt template to obtain the final prompt template.

5. The large language model temporal knowledge question answering method based on sentence granularity prompts according to any one of claims 1 to 4, characterized in that: S4 further includes: The large language model is a pre-trained language model; the pre-trained language model uses the LoRA method to adjust the weight parameters; The output of the pre-trained language model is: y=W.x+W A .W B .x (2); Wherein, x is the input of the pre-trained language model, y is the output of the pre-trained language model, and W is the weight matrix of the pre-trained language model; W A and W B is the rank decomposition matrix of W, that is, W = W A .W B ;W∈R d×k , W A ∈R r×k , W B ∈R d×r ; R represents a matrix with a rank r<<min(d, k); d and k represent the number of rows and columns of the weight matrix W respectively.

Citation Information

Patent Citations

  • Auxiliary intelligent question answering method and device for operation and maintenance of wind turbine generator

    CN115730053A

  • Transaction transaction-oriented intention recognition method in online dialogue

    CN115982337A

  • Vehicle fault diagnosis and prevention method

    CN116414990A

  • Government affair service field multi-strategy fusion dialogue method based on knowledge graph

    CN116628172A

  • Question and answer method and system based on knowledge graph and large language model

    CN116775847A