A large language model dialogue recommendation method based on reinforcement knowledge selection

Abstract

The application discloses a large language model dialogue recommendation method based on reinforcement knowledge selection, and belongs to the field of natural language processing.Combining the encoder of the T5 model and the RGCN model, the application improves the relevance and richness of dialogue content, making the dialogue more natural, smooth and closely meeting the user's needs.The T5 model is used as a strategy model, and reinforcement learning is combined to effectively improve the quality and accuracy of dialogue generation and ensure the efficiency of knowledge selection.Experiments prove that the model of the application exceeds the current latest benchmark model in both automatic evaluation and manual evaluation.The method effectively solves the above problems through an innovative reinforcement learning framework and algorithm, and provides more intelligent, accurate and efficient dialogue recommendation services for users.The application of the method not only marks a great progress in dialogue recommendation technology, but also provides a new direction for the research and development of future intelligent dialogue agents.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing, specifically relating to a large language model dialogue recommendation method based on reinforcement knowledge selection. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially in the field of Natural Language Processing (NLP), Large Language Models (LLMs) such as ChatGPT and Gemini have become a significant driving force in this field. The emergence of these models has greatly enhanced machines' ability to understand and generate natural language, laying the foundation for building more intelligent dialogue systems. Based on this, dialogue recommendation systems have become a research hotspot, aiming to understand user needs and provide corresponding suggestions and assistance through dialogue interaction.

[0003] In the past, dialogue recommendation systems mainly relied on techniques such as pattern matching, keyword triggering, and static decision trees. These methods often proved inadequate in complex dialogue scenarios. They typically lacked a deep understanding of context and the ability to process it flexibly, resulting in limited accuracy and user satisfaction in the recommendation results. Furthermore, these systems often struggled to provide personalized and accurate recommendations when dealing with large amounts of dynamically changing user information and preferences.

[0004] Knowledge graphs, by constructing networks of relationships between entities, provide a rich source of semantic information, enabling dialogue systems to understand and infer the complex relationships behind user intent. Utilizing knowledge graphs to provide more informative and context-sensitive recommendations guides the conversation towards the recommended goal by integrating user interests and feedback. Furthermore, knowledge graphs allow dialogue systems to continuously track topics throughout multi-turn conversations, maintaining dialogue coherence and enhancing the overall interactive experience. Through these technological advancements, dialogue recommendation systems can not only handle more complex recommendation tasks, but this approach also makes conversational recommendations more natural and effective compared to traditional monolithic dialogue systems.

[0005] However, single large language model systems generally perform poorly in dialogue recommendation tasks. Researchers have begun exploring methods combining large language models with reinforcement learning to optimize their application in dialogue recommendation systems. Reinforcement learning is a machine learning method that allows models to learn optimal behavioral strategies through interaction with their environment. It enables models to continuously optimize their recommendation strategies during interactions with users. In this way, models can not only improve the relevance of recommendations but also better adapt to changes in user behavior and preferences, thereby providing more personalized services.

[0006] A dialogue recommendation method combining large language models with reinforcement learning can improve the language model's knowledge selection and application capabilities in specific domains through continuous learning and adaptation while maintaining its generative capabilities. It comprehensively considers user-provided information and historical interaction data during the dialogue process, thereby achieving more accurate knowledge selection and recommendations.

[0007] However, reinforcement learning still faces many challenges in practical applications. For example, how to design effective reward mechanisms to properly guide model learning, how to balance exploration and exploitation, and how to avoid overfitting are all key issues that current technology needs to address.

[0008] This invention proposes a large language model-based dialogue recommendation method based on reinforcement knowledge selection. This method effectively solves the aforementioned problems through an innovative reinforcement learning framework and algorithm, providing users with a more intelligent, accurate, and efficient dialogue recommendation service. This method represents a significant advancement in dialogue recommendation technology and also provides a new direction for the future research and development of intelligent dialogue agents. Summary of the Invention

[0009] This invention aims to solve the aforementioned technical problems by providing a large language model-based dialogue recommendation method based on reinforcement knowledge selection. This invention employs a small, adjustable policy model (e.g., T5) to perform knowledge selection for each input instance while simultaneously generating auxiliary directional stimulus cues, guiding the large language model to generate the required recommended dialogue.

[0010] The technical solution of the present invention:

[0011] A dialogue recommendation method based on a large language model with reinforcement knowledge selection, comprising the following steps:

[0012] (1) Data preprocessing: First, the original dataset is selected, which includes user information, dialogue target, dialogue scenario, relevant knowledge triples extracted from the knowledge graph and multi-turn recommendation dialogue; then, the original dataset is cleaned and text standardized; second, a multi-turn recommendation dialogue in the original dataset is split into multiple data, so that the multi-turn recommendation dialogue in the original dataset is decomposed into dialogue context and target response.

[0013] (2) Constructing a new dataset: Based on the original dataset in step (1), input the user information, dialogue target, dialogue scenario, relevant knowledge triples, dialogue context, and target response text of the original dataset into ChatGPT, so that it selects the 3-5 most relevant knowledge triples, and constructs a new dataset D′={(x,z * )}, where x represents the user information, dialogue target, dialogue scenario, relevant knowledge triples, dialogue context, and target response text of the original dataset, i.e., the input of ChatGPT, and z* Represents the most relevant knowledge triples; the new dataset consists of input stimulus pairs (x, z) * The new dataset is composed of [a set of data] and is used to conduct supervised training of the policy model.

[0014] (3) Converting text into word embedding vectors: For the new dataset constructed in step (2), user information, dialogue context, dialogue scene, and dialogue target are input text into the encoder of the T5 (Text-to-Text Transfer Transformer) model. The encoder of the T5 model, through its attention mechanism, can understand the complex relationships between words. It segments, encodes, and extracts features from the input text to generate a word embedding vector matrix, which is represented as: These embeddings represent the semantic information of the input text, where n W This indicates the number of words in the input text.

[0015] (4) Encoding of the knowledge graph: The relevant knowledge triples from the original dataset in step (1) are input into the relational graph convolutional network RGCN. RGCN is used to process graph-structured data and can encode the relationships between entities, outputting a triplet embeddings vector matrix. The triplet embedding vector matrix captures the semantic information about the relationships between entities, where n T It represents the number of knowledge triples.

[0016] (5) Model Embedding Combination: The word embedding vector matrix generated by the encoder of the T5 model and the triplet embedding vector matrix generated by RGCN are combined using a specific function f to form a comprehensive combined embedding representation: E Combined =f(E Word E Triplet ), where E Word It is the word embedding vector matrix obtained through the encoder of the T5 model, E Triplet This is the triplet embedding vector matrix obtained through the RGCN model. The function f is the vector concatenation strategy or fusion strategy. The fusion strategy can be a weighted average or a neural network fusion layer. This method adopts the vector concatenation strategy.

[0017] (6) Training the Policy Model: The T5 model is used as the policy model to determine which knowledge information should be used in the dialogue and to generate targeted stimulus cues. The word embedding vector matrix E obtained in step (5) is then used to train the policy model. combinedThe target label for each training instance, i.e., z* from the new dataset constructed in step (2), is input into the T5 model for supervised training, thereby initially improving the performance of the policy model on specific tasks. The policy model is fine-tuned by maximizing the log-likelihood:

[0018]

[0019] in, This means that for all samples (x, z) in the dataset * ), calculate the average value of the loss function; logp POL (z * |x) represents the policy model generating the correct output z given the input x. * The logarithmic probability.

[0020] The strategy model learns through supervised training how to select and output the most relevant knowledge triples based on the current dialogue context, dialogue goal, dialogue scenario, and relevant knowledge triples. These triples are used as prompts to guide the generation of subsequent dialogue content.

[0021] (7) Fine-tuning the large language model: Before generating dialogue content, the large language model is fine-tuned using QLoRA (Efficient Finetuning of Quantized LLMs) with the input of user information, dialogue target, dialogue scene, dialogue context, and most relevant knowledge triples from the new dataset and the target response as the target output.

[0022] (8) Application of the large language model: The prompts selected by the strategy model (i.e., knowledge triples), the initial dialogue context, the dialogue goal, and user information are compiled into a pre-set Prompt template and input into a large language model. The large language model is responsible for generating natural, fluent, and relevant dialogue content based on these inputs. This method uses the Chatglm3-6B large language model.

[0023] (9) Evaluation of dialogue content: The quality of the generated dialogue content is evaluated using the BLEU score automatic evaluation method.

[0024]

[0025] Among them, l ref It is the length of the target dialogue content, l cand p is the length of the dialogue content output by the large language model. n It is the geometric mean of the n-gram precision, where N represents the length of the longest n-gram.

[0026] (10) Reinforcement Learning-Based Optimization: The policy model is trained using BLEU scores as rewards. In this way, the policy model learns how to better select knowledge triples to improve the quality of dialogue generation. Reinforcement learning gradually optimizes the policy model's decision-making process through trial and error, enabling it to make more effective knowledge selections in future dialogue scenarios. The reward function can be expressed as:

[0027] R(z) = BLEU(Output, Reference)

[0028] Where Output is the dialogue content generated by the policy model, and Reference is the ideal dialogue content. The objective function of reinforcement learning can be expressed as:

[0029]

[0030] Where, π θ The policy network is determined by the parameter θ, z is the sequence of behaviors sampled from this policy network, and R(z) is the reward function. To prevent the policy network from deviating from the initial policy, a KL divergence penalty term is added, and the final reward function becomes:

[0031]

[0032] Among them, R LLM This is the evaluation function of LLM, where π(z|x) is the probability that the policy network π chooses action z given input x. POL This is the initial policy network. The coefficient β is dynamically adjusted based on the KL divergence and the objective value KL. target Adjust the difference:

[0033] β t+1 =β t (1+K(KL(π t ,π POL ))-KL target )

[0034] Where K is the adjustment coefficient, β t It is a coefficient that is dynamically adjusted in each training step to control the degree to which the policy network deviates from the initial policy during the reinforcement learning process, KL(π). t ,π POL ) is the current policy π at time t. t and initial policy π POL The KL divergence between them.

[0035] The beneficial effects of this invention are:

[0036] 1. By combining the encoder of the T5 model and the RGCN model, the relevance and richness of the dialogue content are improved, making the dialogue more natural, fluent and closely aligned with user needs.

[0037] 2. By adopting the T5 model as the policy model and combining it with reinforcement learning, the quality and accuracy of dialogue generation are effectively improved, ensuring the efficiency of knowledge selection.

[0038] 3. Experiments have shown that the model of this invention surpasses the latest benchmark models in both automatic and manual evaluation. Attached Figure Description

[0039] Figure 1 This is a large language model-based dialogue recommendation system that uses reinforcement knowledge selection. Detailed Implementation

[0040] A dialogue recommendation method based on a large language model with reinforcement knowledge selection, comprising the following steps:

[0041] (1) Data preprocessing: First, select the original dataset (e.g., DuRecDial is a Chinese dialogue dataset used for research on dialogue recommendation systems). The original dataset includes user information, dialogue target, dialogue scenario, relevant knowledge triples extracted from a knowledge graph (a knowledge graph is a structured database that stores entities (such as people, places, things) and their relationships) and multi-turn recommendation dialogues. Then, perform data cleaning and text standardization on the original dataset. Next, split a multi-turn recommendation dialogue in the original dataset into multiple data points, so that the multi-turn recommendation dialogue in the original dataset is decomposed into dialogue context and target response.

[0042] (2) Constructing a new dataset: Based on the original dataset in step (1), input the user information, dialogue target, dialogue scenario, relevant knowledge triples, dialogue context, and target response text of the original dataset into ChatGPT, so that it selects the 3-5 most relevant knowledge triples, and constructs a new dataset D′={(x,z * )}, where x represents the user information, dialogue target, dialogue scenario, relevant knowledge triples, dialogue context, and target response text of the original dataset, i.e., the input of ChatGPT, and z * Represents the most relevant knowledge triples; the new dataset consists of input stimulus pairs (x, z) * The new dataset is composed of [a set of data] and is used to conduct supervised training of the policy model.

[0043] (3) Converting text into word embedding vectors: For the new dataset constructed in step (2), user information, dialogue context, dialogue scene, and dialogue target are input text into the encoder of the T5 (Text-to-Text Transfer Transformer) model. The encoder of the T5 model, through its attention mechanism, can understand the complex relationships between words. It segments, encodes, and extracts features from the input text to generate a word embedding vector matrix, which is represented as: These embeddings represent the semantic information of the input text, where n T This indicates the number of words in the input text.

[0044] (4) Encoding of the knowledge graph: The relevant knowledge triples (e.g., [entity-relationship-entity]) from the original dataset in step (1) are input into the relational graph convolutional network RGCN. RGCN is used to process graph structured data and can encode the relationships between entities, outputting a triplet embeddings vector matrix. The triplet embedding vector matrix captures the semantic information about the relationships between entities, where n E It represents the number of knowledge triples.

[0045] (5) Model Embedding Combination: The word embedding vector matrix generated by the encoder of the T5 model and the triplet embedding vector matrix generated by RGCN are combined using a specific function f to form a comprehensive combined embedding representation: E Combined =f(E Word E Triplet This combination not only captures the semantic information of words but also integrates the structural information of entities and relationships, thus providing a rich information source for downstream text processing tasks. Among them, E Word It is the word embedding vector matrix obtained through the encoder of the T5 model, E Triplet This is the triplet embedding vector matrix obtained through the RGCN model. The function f is the vector concatenation strategy or fusion strategy. The fusion strategy can be a weighted average or a neural network fusion layer. This method adopts the vector concatenation strategy.

[0046] (6) Training the Policy Model: The T5 model is used as the policy model to determine which knowledge information should be used in the dialogue and to generate targeted stimulus cues. The word embedding vector matrix E obtained in step (5) is then used to train the policy model. combinedThe target label for each training instance, i.e., z* from the new dataset generated in step (2), is input into the T5 model for supervised training, thereby initially improving the performance of the policy model on specific tasks. The policy model is fine-tuned by maximizing the log-likelihood:

[0047]

[0048] in, This means that for all samples (x, z) in the dataset * ), calculate the average value of the loss function; logp POL (z * |x) represents the policy model generating the correct output z given the input x. * The logarithmic probability.

[0049] The strategy model learns through supervised training how to select and output the most relevant knowledge triples based on the current dialogue context, dialogue goal, dialogue scenario, and relevant knowledge triples. These triples are used as prompts to guide the generation of subsequent dialogue content.

[0050] (7) Fine-tuning the large language model: Before generating dialogue content, the large language model is fine-tuned using QLoRA (Efficient Finetuning of Quantized LLMs) with user information, dialogue target, dialogue scenario, dialogue context, and the most relevant knowledge triplet from a new dataset as input, and the target response as the target output. QLoRA is an improvement on the attention mechanism that can improve the model's performance on specific tasks, especially when dealing with task-specific queries. Such fine-tuning helps the large language model better understand and respond to dialogue-specific topics and content.

[0051] (8) Application of the large language model: The prompts selected by the strategy model (i.e., knowledge triples), the initial dialogue context, the dialogue goal, and user information are compiled into a pre-set Prompt template and input into a large language model. The large language model is responsible for generating natural, fluent, and relevant dialogue content based on these inputs. This method uses the Chatglm3-6B large language model.

[0052] (9) Evaluation of Dialogue Content: The quality of the generated dialogue content is evaluated using an automatic BLEU score method. The BLEU score is a commonly used machine translation quality assessment metric, used here to measure how close the generated dialogue content is to the ideal dialogue content. This evaluation can be mathematically represented as:

[0053]

[0054] Among them, l ref It is the length of the target dialogue content, l cand p is the length of the dialogue content output by the large language model. n It is the geometric mean of the n-gram precision, where N represents the length of the longest n-gram.

[0055] (10) Reinforcement Learning-Based Optimization: The policy model is trained using BLEU scores as rewards. In this way, the policy model learns how to better select knowledge triples to improve the quality of dialogue generation. Reinforcement learning gradually optimizes the policy model's decision-making process through trial and error, enabling it to make more effective knowledge selections in future dialogue scenarios. The reward function can be expressed as:

[0056] R(z) = BLEU(Output, Reference)

[0057] Where Output is the dialogue content generated by the policy model, and Reference is the ideal dialogue content. The objective function of reinforcement learning can be expressed as:

[0058]

[0059] Where, π θ The policy network is determined by the parameter θ, and z is the sequence of behaviors sampled from this policy network. This is to prevent the policy network from deviating from the initial policy π. POL A KL divergence penalty term was added, and the final reward function became:

[0060]

[0061] Among them, R LLM This is the evaluation function of LLM, where π(z|x) is the probability that the policy network π chooses action z given input x. POL This is the initial policy network. The coefficient β is dynamically adjusted based on the KL divergence and the objective value KL. target Adjust the difference:

[0062] β t+1 =β t (1+K(KL(π t ,π POL ))-KL target )

[0063] Where K is the adjustment coefficient, KL(π) t ,π POL ) is the current policy π at time t. t and initial policy π POLThe KL divergence between them.

[0064] This study used two goal-oriented, knowledge-driven datasets. The first is DuConv (Wu W, Guo Z, Zhou X, et al. Proactive human-machine conversation with explicit conversation goals[J]. arXiv preprint arXiv:1906.05572,2019.), an active dialogue dataset containing approximately 30,000 dialogues and 270,000 utterances. Each dialogue contains an average of 9.1 utterances, each utterance contains an average of 10.6 words, and each dialogue contains an average of 96.2 words and 17.1 knowledge facts. In each round of dialogue, the machine needs to proactively initiate a conversation that combines explicit dialogue goals with relevant knowledge triples. The second is DuRecDial (Liu Z, Wang H, Niu ZY, et al. Towards conversational recommendation over multi-type dialogs[J]. arXiv preprint arXiv:2005.03954,2020.), a goal-oriented, knowledge-driven dialogue recommendation dataset containing approximately 10,000 dialogues and 156,000 utterances. Each dialogue session contains an average of 15.32 utterances and 21.93 knowledge facts, with an average word count of 11.53 per utterance and 12.73 per knowledge fact. In each round of dialogue, the machine acts as a recommender, guiding multi-type dialogues while fully considering the dialogue goal and approaching the recommendation goal. The dialogue goal is a string of similar sequences obtained from knowledge triples. In this study, we assume that the complete dialogue goal is explicitly given at the beginning of the dialogue.

[0065] Our proposed method is applicable to the multi-turn dialogue domain, covering the entire process from data preprocessing to dialogue generation. First, we use the DuRecDial Chinese dialogue dataset, which includes user information and dialogue objectives, for data cleaning and text standardization. Next, the dataset is decomposed into multiple data points to extract dialogue context and target responses. User information and dialogue context are encoded using the encoder of a T5 model to capture deep semantic information. Simultaneously, entity embeddings are obtained by processing triples from a knowledge graph through a Relational Graph Convolutional Network (RGCN). These data are combined through a fusion step to form a rich semantic information foundation. Next, ChatGPT is used to select relevant knowledge triples, and the T5 policy model is trained in a supervised manner, enabling it to select the most relevant knowledge information based on the dialogue context. Then, the QLoRA method is used to fine-tune the large language model, improving its performance on specific tasks. The large language model, combined with the knowledge triples selected by the policy model, generates natural and fluent dialogue content. Finally, BLEU scores are used to evaluate the quality of the dialogue content, and reinforcement learning is employed to optimize the policy model, improving the effectiveness of knowledge selection and the quality of dialogue generation. Our model is implemented using PyTorchFramework, and we use a vocabulary of 15k words. Instead of using pre-trained word embeddings, we set the word embedding size to 300 and randomly initialized the embedding vectors. We use the Adam optimizer (Kingma and Ba 2014) to minimize the loss.

[0066] Benchmark Models – On DuConv, our model is compared to the following models: PGN: Capable of copying words from input sources, this model demonstrates superior performance on multiple natural language generation tasks. KIC: This model demonstrates state-of-the-art performance on the DuConv dataset in 2020. Furthermore, we compare it with retrieval-based and generation-based benchmark models mentioned by Wu et al. (2019). On DuRecDial, our model is compared to the following models: Seq2Seq: A basic sequence-to-sequence model. PostKS: A knowledge-based response generation model that uses a posterior knowledge selection mechanism to choose appropriate knowledge for response generation. MGCG R / G: A proposed retrieval-based and generation-based model that demonstrates state-of-the-art results on DuRecDial.

[0067] Automated Evaluation – We used both human and automated evaluation to assess each model, with evaluation metrics adapted from Wu et al. (2019) and Liu et al. (2020). Automated evaluation metrics included Bleu-1 / 2, F1 score, Distinct-1 / 2, and Perplexity. The F1 score measures the precision and recall of character-level response generation; the Bleu score estimates the fluency of n-gram-level responses; Distinct-1 / 2 assesses response diversity; and perplexity is a commonly used method to evaluate the likelihood of a model generating realistic responses.

[0068] Table 1 Evaluation scores for each model

[0069]

[0070]

[0071] Human evaluation—automatically assessing the quality of dialogue systems has always been a challenging problem. Current mainstream evaluation metrics all have various shortcomings, so BLEU scores cannot accurately measure the quality of responses generated by the model. Due to the complexity and diversity of human language, some sentences may have no overlapping words and be completely unrelated without considering the context, yet they can still serve as responses to the same dialogue situation. In this case, this embodiment introduces human evaluation to further measure the quality of responses generated by the proposed model and various benchmark models. The human evaluation metrics set in this embodiment include relevance and fluency. Relevance measures whether the generated response is consistent with the context and whether it connects to the dialogue. The relevance score ranges from 0 to 2. 0 represents no relevance and cannot be used as a response to the current dialogue; 1 represents moderate relevance and can be used as a response to the current dialogue; 2 represents a very fitting response to the current dialogue situation. Fluency measures whether the generated response is grammatically fluent and error-free. The fluency score ranges from 0 to 1. 0 represents an incoherent sentence with grammatical errors; 1 represents a fluent sentence without grammatical errors. For each model, this embodiment randomly selected 200 test sentences for human evaluation. The evaluators were completely unaware of the experimental content. The evaluation results are shown in Table 2. The model in this embodiment achieved the best results in terms of both relevance and fluency.

[0072] Table 2 Results of Manual Assessment

[0073]

[0074] Example Analysis – Table 3 shows typical cases of the PGN model, which is the best-performing model among the benchmark models in this example. As can be seen, the input text includes dialogue instructions, user information, background knowledge triples, dialogue goals, and dialogue history. The input text contains a large amount of information, and the PGN model cannot select appropriate information from the input text and make an appropriate response; the overall fluency of the response is also insufficient. Our model first selects the three knowledge triples most relevant to the dialogue and makes a correct and fluent response based on the triples.

[0075] Table 3 Dialogue Examples

[0076]

[0077]

[0078] This invention proposes a dialogue recommendation method based on a large language model with reinforced knowledge selection. This method, through an innovative algorithmic framework, effectively improves the recommendation quality and accuracy of dialogue recommendation systems in complex interactions. Experimental results show that this method significantly improves upon benchmark models in various evaluation metrics, as well as dialogue relevance and fluency, opening up new research directions for the development of dialogue recommendation systems.

Claims

1. A dialogue recommendation method based on a large language model with reinforcement knowledge selection, characterized in that, The steps are as follows: (1) Data preprocessing: First, select the original dataset, which includes user information, dialogue target, dialogue scenario, relevant knowledge triples extracted from the knowledge graph, and multi-turn recommendation dialogue; Then, the original dataset is cleaned and the text is standardized. Secondly, the original dataset's multi-turn recommendation dialogue is split into multiple data entries, thus decomposing the original dataset's multi-turn recommendation dialogue into dialogue context and target response; (2) Constructing a new dataset: Based on the original dataset in step (1), input the user information, dialogue target, dialogue scenario, relevant knowledge triples, dialogue context, and target response text of the original dataset into ChatGPT, so that it selects the 3-5 most relevant knowledge triples, and constructs a new dataset D′={(x,z * )}, where x represents the user information, dialogue target, dialogue scenario, relevant knowledge triples, dialogue context, and target response text of the original dataset, i.e., the input of ChatGPT, and z * Represents the most relevant knowledge triples; the new dataset consists of input stimulus pairs (x, z) * The new dataset is composed of [various components] and is used to conduct supervised training of the policy model. (3) Converting text into word embedding vectors: For the new dataset constructed in step (2), user information, dialogue context, dialogue scene, and dialogue target are input text into the encoder of the T5 model. The encoder of the T5 model, through its attention mechanism, can understand the complex relationships between words. It performs word segmentation, encoding, and feature extraction on the input text to generate a word embedding vector matrix, which is represented as: These embeddings represent the semantic information of the input text, where n W This indicates the number of words in the input text; (4) Encoding of the knowledge graph: Input the relevant knowledge triples from the original dataset in step (1) into the relational graph convolutional network RGCN; RGCN is used to process graph structure data and can encode the relationships between entities. The output triple embedding vector matrix The triplet embedding vector matrix captures the semantic information about the relationships between entities, where n T It is the number of knowledge triples; (5) Model Embedding Combination: The word embedding vector matrix generated by the encoder of the T5 model and the triplet embedding vector matrix generated by RGCN are combined using a specific function f to form a comprehensive combined embedding representation: E Combined =f(E Word E Triplet ), where E Word It is the word embedding vector matrix obtained through the encoder of the T5 model, E Triplet It is the triplet embedding vector matrix obtained through the RGCN model; the function f is the vector concatenation or fusion strategy. (6) Training the strategy model: The T5 model is used as the strategy model to determine which knowledge information should be used in the dialogue and to generate targeted stimulus cues; the word embedding vector matrix E obtained in step (5) is then used to train the strategy model. combined The target label for each training instance, i.e., z* in the new dataset constructed in step (2), is input into the T5 model for supervised training, thereby initially improving the performance of the policy model on a specific task; the policy model is fine-tuned by maximizing the log-likelihood. in, This means that for all samples (x, z) in the dataset * ), calculate the average of the loss function; logp POL (z * |x) represents the policy model generating the correct output z given the input x. * The logarithmic probability; The strategy model learns in a supervised manner how to select and output the most relevant knowledge triples based on the current dialogue context, dialogue goal, dialogue scenario, and relevant knowledge triples; these triples are used as cues to guide the generation of subsequent dialogue content. (7) Fine-tuning the large language model: Before generating dialogue content, use user information, dialogue target, dialogue scenario, dialogue context, and most relevant knowledge triplet from the new dataset as input, and use the target response as the target output to fine-tune the large language model using QLoRA. (8) Application of large language model: The prompts, initial dialogue context, dialogue goals and user information selected by the strategy model are compiled into a pre-set Prompt template and input into a large language model; the large language model is responsible for generating natural, fluent and relevant dialogue content based on these inputs; (9) Evaluation of dialogue content: The quality of the generated dialogue content is evaluated using the BLEU score automatic evaluation method. Among them, l ref It is the length of the target dialogue content, l cand p is the length of the dialogue content output by the large language model. n It is the geometric mean of the n-gram precision, where N represents the length of the longest n-gram; (10) Optimization based on reinforcement learning: The policy model is trained using BLEU scores as rewards; in this way, the policy model learns how to better select knowledge triples to improve the quality of dialogue generation; reinforcement learning gradually optimizes the decision-making process of the policy model through trial and error, enabling it to make more effective knowledge selections in future dialogue scenarios; the reward function can be expressed as: R(z) = BLEU(Output, Reference) Where Output is the dialogue content generated by the policy model, and Reference is the ideal dialogue content; the objective function of reinforcement learning can be expressed as: Where, π θ The policy network is determined by the parameter θ, z is the sequence of behaviors sampled from this policy network, and R(z) is the reward function. To prevent the policy network from deviating from the initial policy, a KL divergence penalty term is added, and the final reward function becomes: Among them, R LLM This is the evaluation function of LLM, where π(z|x) is the probability that the policy network π chooses action z given input x. POL This is the initial policy network; the coefficient β is dynamically adjusted based on the KL divergence and the objective value KL. target Adjust the difference: β t+1 =β t (1+K(KL(π t ,π POL ))-KL target ) Where K is the adjustment coefficient, β t It is a coefficient that is dynamically adjusted in each training step to control the degree to which the policy network deviates from the initial policy during the reinforcement learning process, KL(π). R ,π POL ) is the current policy π at time t. t and initial policy π POL The KL divergence between them.

Images