Retrieval enhancement generation system and method based on contextual knowledge flow

Abstract

The invention discloses a retrieval enhancement generation system and method based on context knowledge flow, and the method comprises the steps: firstly, retrieving related knowledge fragments from a large-scale knowledge base through employing a mixed retrieval strategy; then, the knowledge fragments and dialogue context information are coded, and vector representation is generated; adjusting the weight of the knowledge fragment according to the context vector; updating a vector representation of the knowledge fragment according to the context and the generation result; updating the knowledge state diagram according to the condition of each round of dialogue; adjusting a retrieval strategy according to the knowledge state diagram; and finally, fusing the processed knowledge fragment vectors with the context vectors, and inputting the fused knowledge fragment vectors and the context vectors into a generative model to obtain final output contents. According to the method, a dynamic knowledge flowing and updating mechanism is introduced, the limitation of an existing retrieval enhancement generation model RAG in multiple rounds of dialogues and complex reasoning tasks is solved, the utilization efficiency of knowledge is remarkably improved, the continuity and accuracy of generated content are enhanced, and meanwhile the flexibility of the system is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and relates to a retrieval enhancement generation system based on contextual knowledge flow, and also relates to a retrieval enhancement generation method based on contextual knowledge flow. Background Art

[0002] With the development of natural language processing technology, the Retrieval-Augmented Generation (RAG) model has shown great application potential in many natural language processing tasks. By combining a retrieval module with a generation module, the RAG model can leverage external knowledge bases to enrich the knowledge information of generated content, improving the accuracy and professionalism of answers.

[0003] However, existing RAG models exhibit significant limitations when handling complex task scenarios. In multi-turn conversations, due to a lack of effective contextual awareness and knowledge updating mechanisms, the model struggles to adapt its knowledge usage in real time to the dynamics of the conversation, resulting in inadequate coherence and accuracy in generated responses. For example, in intelligent customer service conversations, when users delve deeper into a question or change the topic over multiple rounds, existing RAG models may be unable to promptly associate and apply relevant knowledge, resulting in inappropriate or inaccurate responses.

[0004] In complex reasoning tasks, existing RAG models often fail to effectively integrate and utilize multiple retrieved knowledge fragments for deep reasoning. When dealing with problems requiring cross-domain, multi-step reasoning, they struggle to establish effective connections between knowledge, resulting in less reliable and logical reasoning results.

[0005] While some research has attempted to improve the RAG model, such as introducing dynamic knowledge updates and multi-round dialogue management mechanisms, these approaches mostly rely on static knowledge storage and retrieval, lacking effective processing of the dynamic flow and interaction of knowledge during the generation process. This prevents the model from fully utilizing knowledge when faced with complex and ever-changing task requirements, making it difficult to meet the requirements of practical applications. Summary of the Invention

[0006] The purpose of this invention is to provide a retrieval enhancement generation method based on contextual knowledge flow to solve the problems existing in the existing RAG model in multi-round dialogue and complex reasoning tasks.

[0007] The purpose of the present invention is to provide a retrieval enhancement generation system based on contextual knowledge flow.

[0008] The first technical solution adopted by the present invention is a retrieval enhancement generation system based on contextual knowledge flow, comprising:

[0009] The knowledge representation module uses the BERT model to encode the knowledge fragments and conversation context information retrieved from the knowledge base and generate vector representations;

[0010] The knowledge flow network module, based on the attention mechanism, dynamically adjusts the weight of knowledge fragments according to the context during the generation process;

[0011] The knowledge update module uses the gradient descent method to update the vector representation of knowledge fragments in real time according to context changes during the generation process;

[0012] The multi-round dialogue management module uses a graph neural network to construct a knowledge state graph to manage and update the knowledge state of multi-round dialogues;

[0013] The dynamic retrieval module dynamically adjusts the retrieval strategy according to the current knowledge state graph and retrieves knowledge fragments from the knowledge base.

[0014] The present invention is also characterized in that:

[0015] The knowledge representation module combines knowledge fragments and context information into an input sequence, uses special separators to distinguish the two, and generates vector representations through the Transformer's multi-layer attention mechanism;

[0016] The knowledge flow network module uses a dot product attention mechanism, takes the context vector as the query, and the knowledge fragment vector as the key and value, and calculates the weight of the knowledge fragment through a multi-head attention mechanism;

[0017] In each generation step, the knowledge update module calculates the loss function based on the current context and generation results, calculates the gradient through the backpropagation algorithm, and updates the vector representation of the knowledge fragment;

[0018] In each round of dialogue, the multi-round dialogue management module updates the knowledge state diagram based on user input and system responses, recording the addition of knowledge fragments, weight changes, and association relationship updates;

[0019] The dynamic retrieval module determines the search keywords and scope by analyzing the weights of nodes and the relationship between edges in the knowledge state graph, and prioritizes retrieving the latest knowledge fragments that are relevant to the current dialogue needs.

[0020] The second technical solution adopted by the present invention is: a retrieval enhancement generation method based on contextual knowledge flow, which is specifically implemented according to the following steps:

[0021] Step 1: When receiving a question or conversation content input by the user, retrieve the knowledge fragments related to the input from the large-scale knowledge base; construct a knowledge state graph, where the nodes of the knowledge state graph represent the knowledge fragments;

[0022] Step 2: Combine the retrieved knowledge fragments and conversation context information into an input sequence, segment the words using the BERT tokenizer, complete input embedding and forward propagation, and generate context vectors and knowledge fragment vector representations;

[0023] Step 3: Based on the current context vector, the dot product attention mechanism is used to calculate the correlation score between the context and the knowledge fragment vector. The score is converted into a probability distribution through softmax to obtain the initial weight of each knowledge fragment. The multi-head attention mechanism is then used to optimize the weight distribution. The final weight is determined through linear transformation and normalization, and the node weights of the knowledge state graph are updated accordingly.

[0024] Step 4: Extract the weighted knowledge fragment vector and context vector from the knowledge state graph, and weightedly fuse them into the input vector. Input the intermediate result into the generative model GPT, calculate the loss function of the generative model GPT and update the knowledge fragment vector.

[0025] Step 5: Update the knowledge state diagram based on the situation of each round of dialogue, recording the newly introduced knowledge fragments, changes in the weights of knowledge fragments, and updates on the associations between knowledge;

[0026] Step 6: Based on the current knowledge state diagram, adjust the search keywords and scope and conduct a new round of knowledge search to obtain knowledge fragments that better meet the current conversation needs;

[0027] Step 7: The weight-adjusted and updated knowledge fragment vector is fused with the context vector and input into the generative model GPT to generate the final output content; the generative model GPT will generate answers or text that conform to the context logic and user needs based on the weight and semantic information of the knowledge.

[0028] The present invention is also characterized in that:

[0029] The retrieval process described in step 1 adopts a hybrid retrieval strategy, namely the keyword matching algorithm BM25 and the semantic similarity calculation method BERT. The semantic similarity calculation method BERT is used to calculate the semantic similarity between the candidate knowledge fragments and the user input after the preliminary screening of the keyword matching algorithm BM25. The candidate knowledge is sorted according to the semantic similarity, and the results with highly semantic relevance are retained.

[0030] The calculation process of the keyword matching algorithm BM25 is as follows:

[0031] (1) Calculate the IDF value of each keyword

[0032]

[0033] Among them, N is the total number of knowledge fragments, df(t i ) is a string containing the keyword t iThe number of documents;

[0034] (2) Calculate the TF value of each keyword in the document:

[0035]

[0036] Among them, the frequency (t i ,D) is the keyword t i The number of times it appears in the knowledge segment D, z is the total number of words in the knowledge segment D;

[0037] (3) Calculation of BM25 score

[0038]

[0039] Among them, D is the knowledge fragment, Q is the query content input by the user, and t i is the i-th keyword in the input, e is the total number of keywords in the input; IDF(t i ) is the keyword t i The inverse document frequency; TF(t i ,D) is the keyword t i The word frequency in the knowledge segment D; k1 is an adjustment parameter with a value of 1.2 to 2.0; b is an adjustment parameter with a value of 0.5 to 0.8, |D| is the length of the knowledge segment D in terms of the number of words; avgdl is the average length of all documents.

[0040] Step 2 is as follows:

[0041] The query content Q and knowledge fragment D input by the user are combined into an input sequence and separated by a delimiter, as shown in formula (4):

[0042] [CLS]Q[SEP]D[SEP] (4)

[0043] Among them, [CLS] is the token of the BERT model, which represents the semantic information of the entire sequence, and [SEP] is the separator of the BERT model, which is used to separate different text fragments.

[0044] In step 3, the embedding vector of the conversation context information is extracted: let context C = [C1, C2, ..., C m ], extract the embedding vector of the token obtained after each context is segmented by the bert segmenter These embedding vectors are aggregated as shown in formula (5):

[0045]

[0046] Where m is the number of tokens in the conversation context, is the embedding vector of the i-th token; the final vC It is the vector representation of the context; similarly, the embedding vector of the knowledge fragment is extracted from the output of the knowledge representation module, and the knowledge fragment set P = [D1, D2, ..., D n ], extract the embedding vector of the token obtained after each knowledge fragment is segmented by the bert segmenter:

[0047] Aggregate the embedding vectors of the knowledge fragments to obtain the vector representation of the knowledge fragments:

[0048]

[0049] Among them, n k is the number of tags of the k-th knowledge fragment, is the embedding vector of the jth token in the kth knowledge fragment; the final It is the vector representation of the kth knowledge fragment;

[0050] Then calculate the relevance score between the context vector and each knowledge fragment vector, using the dot product to calculate their relevance;

[0051] For each knowledge fragment D k , calculate its vector representation With the context vector v C The dot product of is shown in formula (7):

[0052]

[0053] Use the softmax function for normalization, as shown in formula (8):

[0054]

[0055] Among them, α k is the normalized weight of the kth knowledge fragment; in order to capture semantic information at different levels, a multi-head attention mechanism is used to capture semantic information at different levels. Assume that there are a total of h attention heads, and the weight calculation method of each head is shown in formula (9):

[0056]

[0057] in, is the vector representation of the k-th knowledge fragment under the h-th attention head, and d is the dimension of the vector;

[0058] The outputs of all attention heads are concatenated as shown in formula (10):

[0059]

[0060] Perform a linear transformation on the concatenated vector to obtain the final knowledge fragment weight as shown in formula (11):

[0061]

[0062] Among them, W is a learnable linear transformation matrix, which is used to map the multi-head output to the final weight space and normalize the obtained weights.

[0063] In step 4, the weighted knowledge fragment vector v D With the context vector v C Fusion is performed to form the input vector v input ; The fusion method is weighted summation, as shown in formula (12):

[0064]

[0065] in, is the weight of the kth knowledge fragment, is the vector representation of the kth knowledge fragment; the fused input vector v input Input into the generative model GPT, the generative model GPT generates a text sequence step by step according to the input vector, and generates a token at each step, as shown in formula (13):

[0066] y=Generate(v input ) (13)

[0067] Calculate the loss function L based on the generated intermediate result y and the target output y*; define the target output The result is y=[y1,y2,…,y t ], then the loss function is expressed as shown in formula (14):

[0068]

[0069] Where T is the length of the generated sequence, is the i-th token in the target output sequence, After generating the first i-1 tokens, the model predicts the i-th token to be The probability of the knowledge fragment vector is then calculated by the back propagation algorithm. As shown in formula (15):

[0070]

[0071] gradient It reflects the effectiveness of the vector representation of the knowledge fragment in the current context and the direction that needs to be adjusted. The vector representation of the knowledge fragment is updated using the gradient descent method. The update formula is shown in formula (16):

[0072]

[0073] in, is the current vector representation of the kth knowledge fragment, α is the learning rate, which controls the update step size; for each knowledge fragment D k , according to its corresponding gradient Adjust its vector representation.

[0074] Step 5 is as follows:

[0075] Construct a knowledge state graph for each round of dialogue, initialize the nodes and edges, and the knowledge state graph G is a directed graph, where the node V represents the knowledge fragment in the knowledge state graph, and the edge E represents the association relationship between the knowledge fragments; each node Saved knowledge fragment D k Vector representation of and weights

[0076] Initialize the edge based on the semantic similarity between knowledge fragments, calculate the cosine similarity between each pair of knowledge fragments, and if the similarity is higher than the threshold θ, establish an edge between them, as shown in Equation (17):

[0077]

[0078] in, It is a knowledge fragment D i and D j The cosine similarity between the vector representations of ;

[0079] If a new knowledge fragment D′ is detected, its vector representation v is retrieved from the knowledge base. D′ , and add a new node v to the knowledge state graph D′ ; Initialize its weight W D′ ; Update the weights of existing nodes according to the context and generation results of the current conversation; the weight update formula is shown in formula (18):

[0080]

[0081] in, is the weight adjustment calculated based on the current conversation context;

[0082] According to the dynamic association relationship between knowledge fragments, the edges in the knowledge state graph are updated. If the similarity between two knowledge fragments is higher than the threshold θ, an edge is established or retained between them; otherwise, the edge is deleted.

[0083] Step 6 is as follows:

[0084] Extract nodes with higher weights from the knowledge state graph G, which represent more important or relevant knowledge fragments in the current conversation; set a weight threshold θ w , extract all nodes whose weight is higher than the threshold, as shown in formula (19):

[0085]

[0086] Among them, V high is a set of high-weight nodes, is a node Analyze the association relationship between high-weight nodes; find other nodes directly associated with high-weight nodes through the edge E of the knowledge state graph. For each high-weight node Find the set of nodes connected to it, as shown in formula (20):

[0087]

[0088] Extract key topics and concepts from the knowledge fragments of high-weight nodes and associated nodes. These topics and concepts serve as keywords for subsequent retrieval, and the extracted key topics and concepts serve as new retrieval keywords.

[0089] Step 7 is as follows:

[0090] The knowledge fragment vector is weightedly fused with the context vector to form the input vector v of the generative model input , the fusion formula is shown in formula (21):

[0091]

[0092] The fused vector v input Input into the generative model, the generative model will gradually generate a text sequence based on the semantic information of the input vector and the weight of the knowledge fragment. The text sequence generation process can be expressed as shown in formula (22):

[0093] y t =Generate(y ＜t ,v input ) (twenty two)

[0094] Among them, y t is the tth generated token, y <t is the token sequence generated previously.

[0095] Compared with the prior art, the present invention has the following beneficial effects:

[0096] 1. Improve knowledge utilization efficiency: Through the knowledge flow mechanism, knowledge can flow and update dynamically according to the context during the generation process, enabling the model to more accurately utilize knowledge related to the current task, avoiding knowledge redundancy and misuse, thereby significantly improving knowledge utilization efficiency.

[0097] 2. Enhance the coherence and accuracy of generated content: In multi-turn conversations, the Knowledge Update Module and the Multi-turn Dialogue Management Module ensure that the model adjusts its knowledge state in real time based on the conversation's development, making generated responses more coherent and accurate. In complex reasoning tasks, the Knowledge Flow Network Module helps the model better integrate and apply multiple pieces of knowledge for deep reasoning, improving the reliability and logic of the reasoning results.

[0098] 3. Improve the adaptability and flexibility of the system: The dynamic retrieval module can dynamically adjust the retrieval strategy according to the knowledge state diagram, enabling the system to quickly adapt to different task requirements and dialogue scenarios, thereby improving the adaptability and flexibility of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Figure 1 It is a system architecture diagram of the present invention;

[0100] Figure 2 is a hybrid search flow chart of the present invention;

[0101] Figure 3 is a flow chart of the encoding and updating module of the present invention; DETAILED DESCRIPTION

[0102] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0103] Example 1

[0104] The present invention provides a retrieval enhancement generation system based on contextual knowledge flow, comprising:

[0105] The knowledge representation module uses the BERT model to encode the knowledge fragments and conversation context information retrieved from the knowledge base and generate vector representations. The knowledge representation module combines the knowledge fragments and context information into a single input sequence, using a special separator to distinguish the two, and generates a vector representation through the Transformer's multi-layer attention mechanism.

[0106] The knowledge flow network module uses the dot product attention mechanism, takes the context vector as the query, and the knowledge fragment vector as the key and value, and calculates the weight of the knowledge fragment through the multi-head attention mechanism.

[0107] Knowledge update module,In each generation step, the knowledge update module calculates the loss function based on the current context and generation results, calculates the gradient through the backpropagation algorithm, and updates the vector representation of the knowledge fragment;

[0108] The multi-round dialogue management module uses a graph neural network to construct a knowledge state graph. In each round of dialogue, the knowledge state graph is updated based on user input and system responses, recording the addition of knowledge fragments, changes in weights, and updates to associations.

[0109] The dynamic retrieval module determines the search keywords and scope by analyzing the weights of nodes and the relationship between edges in the knowledge state graph, and prioritizes retrieving the latest knowledge fragments that are relevant to the current dialogue needs.

[0110] Example 2

[0111] The retrieval enhancement generation method based on contextual knowledge flow is implemented in the following steps:

[0112] Step 1: When receiving a question or conversation content input by the user, retrieve the knowledge fragments related to the input from the large-scale knowledge base; construct a knowledge state graph, where the nodes of the knowledge state graph represent the knowledge fragments;

[0113] Step 2: Combine the retrieved knowledge fragments and conversation context information into an input sequence, segment the words using the BERT tokenizer, complete input embedding and forward propagation, and generate context vectors and knowledge fragment vector representations;

[0114] Step 3: Based on the current context vector, the dot product attention mechanism is used to calculate the correlation score between the context and the knowledge fragment vector. The score is converted into a probability distribution through softmax to obtain the initial weight of each knowledge fragment. The multi-head attention mechanism is then used to optimize the weight distribution. The final weight is determined through linear transformation and normalization, and the node weights of the knowledge state graph are updated accordingly.

[0115] Step 4: Extract the weighted knowledge fragment vector and context vector from the knowledge state graph, and weightedly fuse them into the input vector. Input the intermediate result into the generative model GPT, calculate the loss function of the generative model GPT and update the knowledge fragment vector.

[0116] Step 5: Update the knowledge state diagram based on the situation of each round of dialogue, recording the newly introduced knowledge fragments, changes in the weights of knowledge fragments, and updates on the associations between knowledge;

[0117] Step 6: Based on the current knowledge state diagram, adjust the search keywords and scope and conduct a new round of knowledge search to obtain knowledge fragments that better meet the current conversation needs;

[0118] Step 7: The weight-adjusted and updated knowledge fragment vector is fused with the context vector and input into the generative model GPT to generate the final output content; the generative model GPT will generate answers or text that conform to the context logic and user needs based on the weight and semantic information of the knowledge.

[0119] Example 3

[0120] Based on Example 2, the retrieval process described in step 1 adopts a hybrid retrieval strategy, namely the keyword matching algorithm BM25 and the semantic similarity calculation method BERT. The semantic similarity calculation method BERT is used to calculate the semantic similarity between the candidate knowledge fragments and the user input after the preliminary screening of the keyword matching algorithm BM25. The candidate knowledge is sorted according to the semantic similarity, and the results with highly semantic relevance are retained.

[0121] The calculation process of the keyword matching algorithm BM25 is as follows:

[0122] (1) Calculate the IDF value of each keyword

[0123]

[0124] Among them, N is the total number of knowledge fragments, df(t i ) is a string containing the keyword t i The number of documents;

[0125] (2) Calculate the TF value of each keyword in the document:

[0126]

[0127] Among them, the frequency (t i ,D) is the keyword t i The number of times it appears in the knowledge segment D, z is the total number of words in the knowledge segment D;

[0128] (3) Calculation of BM25 score

[0129]

[0130] Among them, D is the knowledge fragment, Q is the query content input by the user, and t i is the i-th keyword in the input, e is the total number of keywords in the input; IDF(t i ) is the keyword t i The inverse document frequency; TF(t i ,D) is the keyword t iThe word frequency in the knowledge segment D; k1 is an adjustment parameter with a value of 1.2 to 2.0; b is an adjustment parameter with a value of 0.5 to 0.8, |D| is the length of the knowledge segment D in terms of the number of words; avgdl is the average length of all documents.

[0131] The query content Q and knowledge fragment D input by the user are combined into an input sequence and separated by a delimiter, as shown in formula (4):

[0132] [CLS]Q[SEP]D[SEP] (4)

[0133] Among them, [CLS] is the token of the BERT model, which represents the semantic information of the entire sequence, and [SEP] is the separator of the BERT model, which is used to separate different text fragments.

[0134] 1. Use BERT's tokenizer to tokenize the input sequence. The tokenizer breaks the input text into a series of subwords or tokens. For example, if the user inputs query Q, "How to use the BERT model," and candidate knowledge fragment D, "How to use the BERT model," the resulting input sequence after tokenization is as follows:

[0135] [CLS][how][to][use][bert][model][SEP][bert][model][usage][method][SEP]

[0136] 2. Input embedding. The processed input sequence is input into the BERT model, which generates an embedding vector for each token. The input embedding of the BERT model consists of three parts:

[0137] Token Embeddings: The embedding vector corresponding to each token.

[0138] Segment Embeddings: Embedding vectors used to distinguish different text segments. In this example, the query Q and the knowledge segment D belong to different segments and therefore require different Segment Embeddings.

[0139] Position Embeddings: Embedding vectors used to represent the position information of tags in the sequence.

[0140] The final input embedding is the sum of these three parts:

[0141] Input Embedding=Token Embedding+Segment Embedding+Position Embedding

[0142] 3. Forward propagation. The input embedding is fed into the BERT model. After forward propagation through the multi-layer Transformer architecture, a contextual embedding vector for each token is obtained. Particular attention is paid to the embedding vector at the [CLS] position, which represents the semantic information of the entire sequence. The BERT model, through the self-attention mechanism of the multi-layer Transformer architecture, is able to capture the semantic information and contextual relationships in the input sequence.

[0143] 4. Calculate semantic similarity. Use cosine similarity to calculate the semantic similarity between query Q and knowledge fragment D.

[0144] (1) Extract the embedding vector of the [CLS] position

[0145] Extract the embedding vector of the [CLS] position from the output of the BERT model. Assume:

[0146] v Q is the embedding vector of the [CLS] position of the query Q.

[0147] v D is the embedding vector of the [CLS] position of the knowledge fragment D.

[0148] These two vectors represent the semantic information of the query and the knowledge fragment respectively; calculate v Q and v D Cosine similarity of:

[0149] Where: v D is the vector v Q and v D The dot product of ||v Q || is the vector v Q The norm (length). ||v D || is the vector v D Norm (length);

[0150] The specific calculation steps are as follows:

[0151] Compute the dot product:

[0152] Compute the norm:

[0153] Calculate cosine similarity:

[0154] Finally, the top M knowledge fragments after preliminary BM25 filtering and BERT semantic ranking are used as the final search results. This hybrid search strategy combines the efficiency of keyword matching with the accuracy of semantic similarity, enabling more accurate retrieval of knowledge fragments that are highly relevant to the user input.

[0155] In step 3, the embedding vector of the conversation context information is extracted: let context C = [C1, C2, ..., C m ], extract the embedding vector of the token obtained after each context is segmented by the bert segmenter These embedding vectors are aggregated as shown in formula (5):

[0156]

[0157] Where m is the number of tokens in the conversation context, is the embedding vector of the i-th token; the final v C It is the vector representation of the context; similarly, the embedding vector of the knowledge fragment is extracted from the output of the knowledge representation module, and the knowledge fragment set P = [D1, D2, ..., D n ], extract the embedding vector of the token obtained after each knowledge fragment is segmented by the bert segmenter:

[0158] Aggregate the embedding vectors of the knowledge fragments to obtain the vector representation of the knowledge fragments:

[0159]

[0160] Among them, n k is the number of tags of the k-th knowledge fragment, is the embedding vector of the jth token in the kth knowledge fragment; the final It is the vector representation of the kth knowledge fragment;

[0161] Then calculate the relevance score between the context vector and each knowledge fragment vector, using the dot product to calculate their relevance;

[0162] For each knowledge fragment D k , calculate its vector representation With the context vector v C The dot product of is shown in formula (7):

[0163]

[0164] Use the softmax function for normalization, as shown in formula (8):

[0165]

[0166] Among them, α k is the normalized weight of the kth knowledge fragment; in order to capture semantic information at different levels, a multi-head attention mechanism is used to capture semantic information at different levels. Assume that there are a total of h attention heads, and the weight calculation method of each head is shown in formula (9):

[0167]

[0168] in, is the vector representation of the k-th knowledge fragment under the h-th attention head, and d is the dimension of the vector;

[0169] The outputs of all attention heads are concatenated as shown in formula (10):

[0170]

[0171] Perform a linear transformation on the concatenated vector to obtain the final knowledge fragment weight as shown in formula (11):

[0172]

[0173] Among them, W is a learnable linear transformation matrix, which is used to map the multi-head output to the final weight space and normalize the obtained weights.

[0174] Example 4

[0175] Based on Example 3, in step 4, the weight-adjusted knowledge segment vector v D With the context vector v C Fusion is performed to form the input vector v input ; The fusion method is weighted summation, as shown in formula (12):

[0176]

[0177] in, is the weight of the kth knowledge fragment, is the vector representation of the kth knowledge fragment; the fused input vector v input Input into the generative model GPT to generate the intermediate result y. The intermediate result y is a tag sequence that represents the model's understanding of the current input and the generated content. The intermediate result y is shown in formula (13):

[0178] y=Generate(v input ) (13)

[0179] Calculate the loss function L based on the generated intermediate result y and the target output y*; define the target output The result is y=[y1,y2,…,y t ], then the loss function is expressed as shown in formula (14):

[0180]

[0181] Where T is the length of the generated sequence, is the i-th token in the target output sequence, After generating the first i-1 tags, the model predicts the i-th tag to be The probability of the knowledge fragment vector is then calculated by the back propagation algorithm. As shown in formula (15):

[0182]

[0183] gradient It reflects the effectiveness of the vector representation of the knowledge fragment in the current context and the direction that needs to be adjusted. The vector representation of the knowledge fragment is updated using the gradient descent method. The update formula is shown in formula (16):

[0184]

[0185] in, is the current vector representation of the kth knowledge fragment, α is the learning rate, which controls the update step size; for each knowledge fragment D k , according to its corresponding gradient Adjust its vector representation. If a knowledge fragment has a large gradient, it indicates a significant impact on the current generated result and requires a larger adjustment. If a knowledge fragment has a small gradient, it indicates a smaller impact on the current generated result and can be adjusted more gently. Through step 4, knowledge updating, the model dynamically adjusts the vector representation of the knowledge fragment based on the current context and generated results. This process ensures that the knowledge fragment remains consistent with the current context, improving the timeliness and relevance of the knowledge, thereby enhancing the quality and accuracy of the generated content.

[0186] Example 5

[0187] Based on Example 4, in step 5, the knowledge state graph is updated according to the situation of each round of dialogue, recording the addition of knowledge fragments, weight changes, and association relationship updates, so as to better manage the knowledge flow and state changes in the dialogue. A knowledge state graph is constructed for each round of dialogue, and the nodes and edges are initialized. The knowledge state graph G is a directed graph, in which the node V represents the knowledge fragment and the edge E represents the association relationship between the knowledge fragments. Each node Saved knowledge fragment D k Vector representation of and its weight At the beginning of the conversation, based on the initially retrieved knowledge fragments D1, D2, ..., D N Initialize the nodes of the knowledge state graph. Each knowledge fragment D k Corresponding to a node Its initial vector is expressed as The initial weight is Initialize edges based on the semantic similarity between knowledge fragments. Calculate the cosine similarity between each pair of knowledge fragments, and if the similarity is above a certain threshold θ, establish an edge between them, as shown in Equation (17).

[0188]

[0189] in, It is a knowledge fragment D i and D j The cosine similarity between the vector representations of ;

[0190] During the conversation, new knowledge fragment nodes are dynamically added and existing nodes are updated based on user input and system-generated content. In each round of conversation, the user input Q and the system-generated answer y are analyzed to detect whether new knowledge fragments D′ have been introduced. If the user asks a new question or changes the topic, knowledge fragments related to the new question may be introduced.

[0191] If a new knowledge fragment D′ is detected, its vector representation v is retrieved from the knowledge base. D′ , and add a new node v to the knowledge state graph D′ ; Initialize its weight W D′ ; Update the weights of existing nodes according to the context and generated results of the current conversation; if a knowledge fragment is frequently cited in the current conversation and is considered important, its weight should be increased; the weight update formula is shown in formula (18):

[0192]

[0193] in, is the weight adjustment calculated based on the current conversation context;

[0194] According to the dynamic association relationship between knowledge fragments, the edges in the knowledge state graph are updated. If the similarity between two knowledge fragments is higher than the threshold θ, an edge is established or retained between them; otherwise, the edge is deleted.

[0195] The updated knowledge state graph is passed to subsequent modules to provide support for dynamic retrieval and content generation. After each round of dialogue, the updated knowledge state graph G′ is passed to the dynamic retrieval module and the content generation module. The knowledge state graph contains the state information of all knowledge fragments in the current dialogue, including the weights of nodes and the associations of edges. The dynamic retrieval module adjusts the retrieval strategy based on the node weights and edge relationships in the knowledge state graph. For example, if a knowledge fragment has a high weight and is closely related to other knowledge fragments, the dynamic retrieval module will give priority to retrieving more content related to the knowledge fragment. The content generation module dynamically adjusts the use of knowledge fragments based on the information in the knowledge state graph. For example, when generating answers, knowledge fragments with higher weights are given priority, and the associations between knowledge fragments are considered to generate more coherent and accurate answers.

[0196] Through step 5, multi-round dialogue management and knowledge state updates, the model can dynamically manage the flow of knowledge and state changes during the dialogue. This process ensures that the knowledge state diagram can reflect the knowledge usage in the dialogue in real time, providing strong support for subsequent dynamic retrieval and content generation.

[0197] In step 6, according to the current knowledge state graph, the retrieval strategy is dynamically adjusted to obtain knowledge fragments that are more in line with the current dialogue needs: nodes with higher weights are extracted from the knowledge state graph G, which represent more important or more relevant knowledge fragments in the current dialogue; a weight threshold θ is set w , extract all nodes whose weight is higher than the threshold, as shown in formula (19):

[0198]

[0199] Among them, V high is a set of high-weight nodes, is a node Analyze the association relationship between high-weight nodes; find other nodes directly associated with high-weight nodes through the edge E of the knowledge state graph. For each high-weight node Find the set of nodes connected to it, as shown in formula (20):

[0200]

[0201] Extract key topics and concepts from the knowledge fragments of high-weight nodes and associated nodes. These topics and concepts serve as keywords for subsequent retrieval, and the extracted key topics and concepts serve as new retrieval keywords.

[0202] These associated nodes may contain extended information or supplementary content related to the current conversation. Extract key topics and concepts from the knowledge fragments of high-weight nodes and associated nodes. These topics and concepts can be used as keywords for subsequent retrieval. Use the extracted key topics and concepts as new retrieval keywords. Dynamically adjust the retrieval scope based on the progress of the conversation and changes in the knowledge state graph. If the conversation involves multiple topics, knowledge fragments related to these topics can be retrieved simultaneously. Use the generated retrieval keywords to construct a retrieval query, and the process returns to step 1 to continue performing related operations. Integrate the newly retrieved knowledge fragments into the knowledge state graph to support subsequent conversation management.

[0203] Through step 6, dynamic retrieval adjustment, the model can dynamically adjust its retrieval strategy based on the progress of the current conversation and changes in the knowledge state graph, obtaining knowledge fragments that better meet the needs of the current conversation. This process ensures that information in the knowledge base can be retrieved and utilized in a timely and accurate manner, thereby improving the response quality and adaptability of the dialogue system.

[0204] In step 7, the weighted and updated knowledge fragment vectors are fused with the context vector and input into the generative model to generate the final output content. and the corresponding weights After dynamic retrieval adjustment and multi-round dialogue management, these knowledge fragments are highly relevant to the current context. The vector representation v of the current context is extracted from the multi-round dialogue management module. C This vector reflects the semantic information and topic direction of the current conversation. The knowledge fragment vector is weighted and fused with the context vector to form the input vector v of the generative model. input , the fusion formula is shown in formula (21):

[0205]

[0206] The fused vector v input Input into the generative model, the generative model will gradually generate a text sequence based on the semantic information of the input vector and the weight of the knowledge fragment. The text sequence generation process can be expressed as shown in formula (22):

[0207] y t =Generate(y <t ,v input ) (twenty two)

[0208] Among them, y t is the tth generated token, y ＜tis the token sequence generated previously. Set the maximum length T to ensure that the generated text is not too long or too short. The generated length can be adjusted according to the specific task requirements. The generation process ends when the maximum length T is reached or the end marker (such as [EOS]) is generated. The final text sequence y is generated. output It is the answer or text generated by the model based on the input vector that conforms to the context logic and user needs.

[0209] Through step 7, the generative model integrates the processed and updated knowledge fragments with contextual information to generate high-quality responses or text that meet user needs and the logic of the conversation. This process ensures the accuracy and coherence of the generated content, improving the user experience and response quality of the conversation system.

[0210] Example 6

[0211] Step 1: Imagine a scenario where a user inquires about an electronic product in an intelligent customer service conversation. The user enters: "What is the battery life of this phone?"

[0212] Knowledge fragment retrieval: After receiving user input, the system first searches the knowledge base for relevant knowledge fragments. The retrieval process uses a hybrid retrieval strategy that combines methods based on keyword matching (BM25) and semantic similarity calculation (BERT).

[0213] (1) BM25 search: Quickly filter out relevant documents in the knowledge base based on the keywords entered by the user (such as "mobile phone" and "battery life").

[0214] (2) BERT semantic ranking: For knowledge fragments that have been initially screened by BM25, the BERT model is used to calculate their semantic similarity with the user input, and knowledge fragments with highly relevant semantics are retained.

[0215] (3) Search results: Assume that the retrieved knowledge fragments include: "The battery capacity of this mobile phone is 5000mAh, and the battery life is strong." "The mobile phone supports fast charging technology and the charging speed is fast."

[0216] Step 2: Input the retrieved knowledge fragments and the contextual information entered by the user into the knowledge representation module, encode them using the BERT model, and generate vector representations.

[0217] (1) Combine user input and knowledge fragments into an input sequence: [CLS] How is the battery life of this phone? [SEP] The battery capacity of this phone is 5000mAh, which has a long battery life; [SEP] The phone supports fast charging technology and charges quickly. [SEP]

[0218] (2) Use the BERT model to segment, embed, and forward propagate the input sequence to generate a contextual embedding vector for each token.

[0219] (3) Extract the embedding vector of the [CLS] position as the semantic representation of the entire sequence.

[0220] (4) The context vector of the user input and the vector representation of each knowledge fragment.

[0221] Step 3: The knowledge flow network module performs attention calculation on the vector representation of the knowledge fragment based on the context vector and dynamically adjusts the weight of each knowledge fragment.

[0222] (1) Calculate the relevance score (dot product) between the context vector and each knowledge fragment vector.

[0223] (2) Use the softmax function to convert the relevance score into a probability distribution to obtain the weight of each knowledge fragment.

[0224] (3) The multi-head attention mechanism is used to capture semantic information at different levels and further optimize the weight distribution.

[0225] (4) Assume that after calculation, the knowledge fragment "The battery capacity of this mobile phone is 5000mAh and has a long battery life" has a higher weight because it is more directly related to the user's problem.

[0226] Step 4: During the answer generation process, the knowledge update module dynamically updates the vector representation of the knowledge fragment based on the current context and generation results.

[0227] (1) The weighted knowledge fragment vector is fused with the context vector and input into the generative model (such as GPT) to generate an intermediate result.

[0228] (2) Calculate the loss function based on the generated intermediate results and target output.

[0229] (3) Use gradient descent to update the vector representation of the knowledge fragment to make it more consistent with the current dialogue requirements.

[0230] Step 5: The multi-round dialogue management module updates the knowledge state diagram according to the situation of each round of dialogue, recording the addition of knowledge fragments, weight changes and association relationship updates.

[0231] (1) Initialize the knowledge state graph, use the retrieved knowledge fragments as nodes, and initialize the edges according to semantic similarity.

[0232] (2) During the conversation, if the user asks further questions (such as “What about its charging speed?”), new knowledge needs are detected, and relevant knowledge fragments are retrieved from the knowledge base and added to the knowledge state graph.

[0233] (3) Update the weights of existing nodes and adjust the weights according to the importance of the knowledge fragment in the conversation.

[0234] (4) Update the association relationship between knowledge fragments and dynamically adjust the existence and weight of edges.

[0235] (5) Results: The knowledge state diagram can reflect the knowledge usage in the conversation in real time, providing support for subsequent dynamic retrieval and content generation.

[0236] Step 6: The dynamic retrieval module adjusts the retrieval strategy based on the current knowledge state graph to obtain knowledge fragments that better meet the current dialogue requirements.

[0237] (1) Extract nodes with higher weights and their associated nodes from the knowledge state graph and analyze key topics and concepts.

[0238] (2) Use the extracted key topics and concepts as new search keywords and dynamically adjust the search scope.

[0239] (3) Conduct a new round of knowledge retrieval to obtain more relevant knowledge fragments.

[0240] Step 7: The weight-adjusted and updated knowledge fragment vector is fused with the context vector and input into the generative model to generate the final output content.

[0241] (1) Extract the vector representation and corresponding weights of all knowledge fragments in the knowledge state graph.

[0242] (2) The knowledge fragment vector and the context vector are weightedly fused to form the input vector of the generative model.

[0243] (3) The generative model gradually generates text sequences based on the semantic information of the input vector and the weights of the knowledge fragments.

[0244] (4) Final output: "This phone has a strong battery life. The battery capacity is 5000mAh, which can meet the needs of daily use. At the same time, it also supports fast charging technology, which charges quickly and can quickly restore power."

[0245] From the above specific implementation process, it can be seen that the contextual knowledge flow-based retrieval enhancement generation method of the present invention can effectively improve the knowledge utilization efficiency and the quality of generated content in complex dialogue scenarios, providing users with a better service experience.

Claims

1. A retrieval enhancement generation system based on contextual knowledge flow, characterized by: include: The knowledge representation module uses the BERT model to encode the knowledge fragments and conversation context information retrieved from the knowledge base and generate vector representations; The knowledge flow network module, based on the attention mechanism, dynamically adjusts the weight of knowledge fragments according to the context during the generation process; The knowledge update module uses the gradient descent method to update the vector representation of knowledge fragments in real time according to context changes during the generation process; The multi-round dialogue management module uses a graph neural network to construct a knowledge state graph to manage and update the knowledge state of multi-round dialogues; The dynamic retrieval module dynamically adjusts the retrieval strategy according to the current knowledge state graph and retrieves knowledge fragments from the knowledge base.

2. The context-based knowledge flow-based retrieval enhancement generation system according to claim 1, characterized in that: The knowledge representation module combines knowledge fragments and context information into an input sequence, uses separators to distinguish the two, and generates vector representations through the Transformer's multi-layer attention mechanism; The knowledge flow network module adopts the dot product attention mechanism, takes the context vector as the query, and the knowledge fragment vector as the key and value, and calculates the weight of the knowledge fragment through the multi-head attention mechanism; The knowledge update module calculates the loss function according to the current context and generation results in each generation step, calculates the gradient through the back-propagation algorithm and updates the vector representation of the knowledge fragment; The multi-round dialogue management module updates the knowledge state diagram in each round of dialogue based on user input and system response, recording the addition of knowledge fragments, weight changes, and association relationship updates; The dynamic retrieval module determines the retrieval keywords and scope by analyzing the weights of nodes and the relationship between edges in the knowledge state graph, and prioritizes retrieval of the latest knowledge fragments that are relevant to the current dialogue needs.

3. A retrieval enhancement generation method based on contextual knowledge flow, characterized in that: Please follow the steps below to implement it: Step 1: When receiving a question or conversation content input by the user, retrieve the knowledge fragments related to the input from the large-scale knowledge base; Construct a knowledge state graph, where the nodes of the knowledge state graph represent knowledge fragments; Step 2: Combine the retrieved knowledge fragments and conversation context information into an input sequence, segment the words using the BERT tokenizer, complete input embedding and forward propagation, and generate context vectors and knowledge fragment vector representations; Step 3: Based on the current context vector, the dot product attention mechanism is used to calculate the correlation score between the context and the knowledge fragment vector, and the softmax is used to convert it into a probability distribution to obtain the initial weight of each knowledge fragment. Then, the multi-head attention mechanism is used to optimize the weight distribution, and the final weight is determined through linear transformation and normalization, and the node weights of the knowledge state graph are updated accordingly; Step 4: Extract the weighted knowledge fragment vector and context vector from the knowledge state graph, and weightedly fuse them into the input vector. Input the intermediate result into the generative model GPT, calculate the loss function of the generative model GPT and update the knowledge fragment vector. Step 5: Update the knowledge state diagram based on the situation of each round of dialogue, recording the newly introduced knowledge fragments, changes in the weights of knowledge fragments, and updates on the associations between knowledge; Step 6: Based on the current knowledge state diagram, adjust the search keywords and scope and conduct a new round of knowledge search to obtain knowledge fragments that better meet the current conversation needs; Step 7: The weighted and updated knowledge fragment vector is fused with the context vector and input into the generative model GPT to generate the final output content; The generative model GPT generates answers or text that conforms to contextual logic and user needs based on the weight of knowledge and semantic information.

4. The context-based knowledge flow-based retrieval enhancement generation method according to claim 3, characterized in that: The retrieval process described in step 1 adopts a hybrid retrieval strategy, namely the keyword matching algorithm BM25 and the semantic similarity calculation method BERT. The semantic similarity calculation method BERT is used to calculate the semantic similarity between the candidate knowledge fragments and the user input after the preliminary screening of the keyword matching algorithm BM25. The candidate knowledge is sorted according to the semantic similarity, and the results with highly semantic relevance are retained.

5. The context-based knowledge flow-based retrieval enhancement generation method according to claim 3, characterized in that: The calculation process of the keyword matching algorithm BM25 is as follows: (1) Calculate the IDF value of each keyword Among them, N is the total number of knowledge fragments, df(t i ) is a string containing the keyword t i The number of documents; (2) Calculate the TF value of each keyword in the document: Among them, the frequency (t i ,D) is the keyword t i The number of times it appears in the knowledge segment D, z is the total number of words in the knowledge segment D; (3) Calculation of BM25 score Among them, D is the knowledge fragment, Q is the query content input by the user, and t i is the i-th keyword in the input, e is the total number of keywords in the input; IDF(t i ) is the keyword t i The inverse document frequency; TF(t i ,D) is the keyword t i The word frequency in the knowledge segment D; k1 is an adjustment parameter with a value of 1.2 to 2.0; b is an adjustment parameter with a value of 0.5 to 0.8, |D| is the length of the knowledge segment D in terms of the number of words; avgdl is the average length of all documents.

6. The context-based knowledge flow-based retrieval enhancement generation method according to claim 5, characterized in that: Step 2 is as follows: The query content Q and knowledge fragment D input by the user are combined into an input sequence and separated by a delimiter, as shown in formula (4): [CLS]Q[SEP]D[SEP] (4) Among them, [CLS] is the token of the BERT model, which represents the semantic information of the entire sequence, and [SEP] is the separator of the BERT model, which is used to separate different text fragments.

7. The context-based knowledge flow-based retrieval enhancement generation method according to claim 6, characterized in that: In step 3, the embedding vector of the conversation context information is extracted: let context C = [C1, C2, ..., C m ], extract the embedding vector of the token obtained after each context is segmented by the bert segmenter These embedding vectors are aggregated as shown in formula (5): Where m is the number of tokens in the conversation context, is the embedding vector of the i-th token; the final v C It is the vector representation of the context; similarly, the embedding vector of the knowledge fragment is extracted from the output of the knowledge representation module, and the knowledge fragment set P = [D1, D2, ..., D n ], extract the embedding vector of the token obtained after each knowledge fragment is segmented by the bert segmenter: Aggregate the embedding vectors of the knowledge fragments to obtain the vector representation of the knowledge fragments: Among them, n k is the number of tags of the k-th knowledge fragment, is the embedding vector of the jth token in the kth knowledge fragment; the final It is the vector representation of the kth knowledge fragment; Then calculate the relevance score between the context vector and each knowledge fragment vector, using the dot product to calculate their relevance; For each knowledge fragment D k , calculate its vector representation With the context vector v C The dot product of is shown in formula (7): Use the softmax function for normalization, as shown in formula (8): Among them, α k is the normalized weight of the kth knowledge fragment; in order to capture semantic information at different levels, a multi-head attention mechanism is used to capture semantic information at different levels. Assume that there are a total of h attention heads, and the weight calculation method of each head is shown in formula (9): in, is the vector representation of the k-th knowledge fragment under the h-th attention head, and d is the dimension of the vector; The outputs of all attention heads are concatenated as shown in formula (10): Perform a linear transformation on the concatenated vector to obtain the final knowledge fragment weight as shown in formula (11): Among them, W is a learnable linear transformation matrix, which is used to map the multi-head output to the final weight space and normalize the obtained weights.

8. The context-based knowledge flow-based retrieval enhancement generation method according to claim 7, characterized in that: In step 4, the weighted knowledge fragment vector v D With the context vector v C Fusion is performed to form the input vector v input ; The fusion method is weighted summation, as shown in formula (12): in, is the weight of the kth knowledge fragment, is the vector representation of the kth knowledge fragment; the fused input vector v input Input into the generative model GPT, the generative model GPT generates a text sequence step by step according to the input vector, and generates a token at each step, as shown in formula (13): y=Generate(v input ) (13) Calculate the loss function L based on the generated intermediate result y and the target output y*; define the target output The result is y=[y1,y2,…,y T ], then the loss function is expressed as shown in formula (14): Where T is the length of the generated sequence, is the i-th token in the target output sequence, After generating the first i-1 tokens, the model predicts the i-th token to be The probability of the knowledge fragment vector is then calculated by the back propagation algorithm. As shown in formula (15): gradient It reflects the effectiveness of the vector representation of the knowledge fragment in the current context and the direction that needs to be adjusted. The vector representation of the knowledge fragment is updated using the gradient descent method. The update formula is shown in formula (16): in, is the current vector representation of the kth knowledge fragment, α is the learning rate, which controls the update step size; for each knowledge fragment D k , according to its corresponding gradient Adjust its vector representation.

9. The context-based knowledge flow-based retrieval enhancement generation method according to claim 8, characterized in that: Step 5 is as follows: Construct a knowledge state graph for each round of dialogue, initialize the nodes and edges, and the knowledge state graph G is a directed graph, where the node V represents the knowledge fragment in the knowledge state graph, and the edge E represents the association relationship between the knowledge fragments; each node Saved knowledge fragment D k Vector representation of and weights Initialize the edge based on the semantic similarity between knowledge fragments, calculate the cosine similarity between each pair of knowledge fragments, and if the similarity is higher than the threshold θ, establish an edge between them, as shown in Equation (17): in, It is a knowledge fragment D i and D j The cosine similarity between the vector representations of ; If a new knowledge fragment D is detected ′ , retrieve its vector representation v from the knowledge base D′ , and add a new node v to the knowledge state graph D′ ; Initialize its weight W D′ ; Update the weights of existing nodes according to the context and generation results of the current conversation; the weight update formula is shown in formula (18): in, is the weight adjustment calculated based on the current conversation context; According to the dynamic association relationship between knowledge fragments, the edges in the knowledge state graph are updated. If the similarity between two knowledge fragments is higher than the threshold θ, an edge is established or retained between them; otherwise, the edge is deleted.

10. The context-based knowledge flow-based retrieval enhancement generation method according to claim 9, characterized in that: Step 6 is as follows: Extract nodes with higher weights from the knowledge state graph G, which represent more important or relevant knowledge fragments in the current conversation; set a weight threshold θ w , extract all nodes whose weight is higher than the threshold, as shown in formula (19): Among them, V high is a set of high-weight nodes, is a node Analyze the association relationship between high-weight nodes; find other nodes directly associated with high-weight nodes through the edge E of the knowledge state graph. For each high-weight node Find the set of nodes connected to it, as shown in formula (20): Extract key topics and concepts from the knowledge fragments of high-weight nodes and related nodes. These topics and concepts serve as keywords for subsequent retrieval, and the extracted key topics and concepts serve as new retrieval keywords. Step 7 is as follows: The knowledge fragment vector is weightedly fused with the context vector to form the input vector v of the generative model input , the fusion formula is shown in formula (21): The fused vector v input Input into the generative model, the generative model will gradually generate a text sequence based on the semantic information of the input vector and the weight of the knowledge fragment. The text sequence generation process can be expressed as shown in formula (22): and t =Generate(y <t ,v input ) (22) Among them, y t is the tth generated token, y ＜t is the token sequence generated previously.

Images