Subsequent question generation method based on knowledge graph and large language model

Abstract

The invention discloses a follow-up question generation method based on a knowledge graph and a large language model, which is mainly used for generating follow-up questions in a dialogue system, helping a user to explore information and providing better user experience. According to the method, for given historical question and answer information, firstly intention recognition is carried out, relevant background knowledge is expanded, keywords are extracted from dialogues, and query is constructed to retrieve the most relevant Wikipedia page. And then, constructing a real-time knowledge graph by taking an entity corresponding to the page as a center. And then, selecting a node most relevant to the current dialogue based on two dimensions of node importance and relevancy, thereby determining external background knowledge needing to be introduced. Therefore, the model can access wider knowledge resources, and the depth and relevancy of the generated problem are improved. And finally, a knowledge fusion operation is designed, external Wikipedia knowledge obtained previously is written continuously based on the context by instructing the large language model, the understanding and cognition of the model on the context are further enhanced, and subsequent problem generation is guided. A series of experiments show that the method has high overall performance in subsequent problem generation tasks, problems with richer information and higher cognitive complexity can be generated, and dialogues can be promoted to a deeper level.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method for generating follow-up questions based on a knowledge graph and a large language model. Background Art

[0002] Asking questions is a fundamental way for humans to learn new knowledge. Question generation is an important task in the field of natural language processing, which aims to generate questions based on a given text. High-quality questions are crucial for dialogue systems, as an excellent system should be able to interact well with users by asking and answering questions. With the rapid development of artificial intelligence technology, generative AI dialogue systems have been widely used in many fields such as education, healthcare, and legal consultation. However, while large language models (LLMs) such as ChatGPT can respond to queries, they are usually passive - they only respond to users' queries rather than actively guide the conversation or ask their own queries. To address this limitation of initiative, the task of follow-up question generation is introduced. In dialogue systems, follow-up questions usually refer to continuation questions generated based on user input or the system's initial answer. This type of question is significantly different from the questions generated by traditional question generation tasks, in which the source text can be used to answer the generated questions. In contrast, follow-up questions cannot be answered in the previous context. Intuitively, a good follow-up question must meet two requirements while maintaining coherence and fluency: (1) ensuring contextual relevance. Follow-up questions should be highly relevant to the current topic of conversation and should not deviate from the previous conversation content; (2) they should be designed to explore new information, thereby leading the next response to provide more novel information and advance the conversation to a deeper level.

[0003] However, some scholars have found that it is difficult for machines to generate relevant questions by integrating background knowledge and examples. Compared with humans, the amount of information in the generated follow-up questions is significantly different. Moreover, humans can generate follow-up questions through advanced cognitive skills such as analogy and association. According to research, due to the limitations of training data and preset models, most of the questions generated by machines remain at the level of surface language relevance and lack flexibility and creativity. Figure 2 The example in intuitively demonstrates the limitations of machine-generated follow-up questions. When discussing eye color, humans can think of other factors not mentioned in the context, such as melanin and iris. However, because the machine can only rely on contextual information, the generated questions are related to the previous context, but the depth and breadth of the content are often insufficient. The technical problem to be solved by the present invention is that the questions generated by the existing follow-up question generation model are often limited to shallow contextual questions, lack inspiration and relevance, and have insufficient information, which is far from the level of humans. A follow-up question generation method based on knowledge graph and large language model is provided. Summary of the invention

[0004] In view of the above limitations, the present invention proposes a method for generating follow-up questions based on knowledge graph and large language model. Figure 3 As shown, the method of the present invention is divided into three stages, namely, identification, selection and fusion. The first stage is to generate questions by identifying contextual topics, the second stage is to build a knowledge graph online and select the most relevant background knowledge, and the third stage is to combine Wikipedia knowledge with a large language model to generate the final question. The present invention generates coherent, clear, information-rich and exploratory follow-up questions by introducing external common sense knowledge and performing knowledge fusion operations.

[0005] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention to solve its technical problem is: a method for generating subsequent questions based on a knowledge graph and a large language model. For a given historical question and answer information, the method first performs intent recognition, expands relevant background knowledge, extracts keywords from the conversation, and constructs a query to retrieve the most relevant Wikipedia page. Next, the present invention constructs a real-time knowledge graph centered on the entity corresponding to the page. Then, based on the two dimensions of node importance and relevance, the nodes most relevant to the current conversation are selected to determine the external background knowledge that needs to be introduced. This allows the model to access a wider range of knowledge resources and improve the depth and relevance of the generated questions. Finally, the present invention designs a knowledge fusion operation to further enhance the model's understanding and cognition of the context and guide the generation of subsequent questions by instructing the large language model to continue writing previously acquired external Wikipedia knowledge based on the context.

[0006] The method for generating subsequent questions based on a knowledge graph and a large language model of the present invention comprises the following steps:

[0007] 1) For a given historical conversation, extract subject words and keywords to identify the overall conversation intent;

[0008] 2) for the combination of subject terms and keywords obtained in step 1), identify the core topics by constructing queries to retrieve the most relevant Wikipedia pages and extract the correct contextual information to generate follow-up questions;

[0009] 3) For the core topics obtained in step 2), build a real-time knowledge graph to obtain background knowledge related to the conversation;

[0010] 4) For the knowledge graph constructed in step 3), evaluate the relevance of entities and conversation context in the graph in order to select meaningful background knowledge and improve the accuracy and reliability of generating subsequent questions;

[0011] 5) Prompt learning is used to instruct the large language model to continue writing the external Wikipedia knowledge obtained in step 4) based on the context, and the fused knowledge is used to guide subsequent question generation.

[0012] As a further improvement of the present invention, in step 1), the present invention simulates the overall conversation by combining subject words and keywords to identify the conversation intent. The present invention inputs a given question-answer pair into a large language model and instructs it to extract key information from the question-answer pair. During the extraction process, the large language model extracts a subject word and n keywords (n≥1). Because the subject word is a highly concise description of the entire question and answer, it is used to capture the user's main question intent, while the keyword extracts finer-grained details from the question and answer.

[0013] As a further improvement of the present invention, in step 2), in order to obtain context information, the present invention searches Wikipedia based on subject terms and keywords to obtain relevant context entities in an iterative search. Specifically, the present invention first searches for pages whose titles contain subject terms, recorded as a set Then, the present invention traverses the keywords one by one, adds each keyword to the query condition, and The pages containing the keywords are retrieved until all the keyword traversals are completed. By narrowing the query scope, the present invention obtains the most relevant pages. In the dynamic search process, if a unique matching result is found, the search process is terminated in advance.

[0014] As a further improvement of the present invention, in step 2), since the word overlap score of the search engine may not accurately reflect the text relevance, the present invention introduces a re-ranker to re-rank the relevant pages. The present invention selects the T5 (Text-to-Text Transfer Transformer) model as the basic model of the re-ranker. After the re-ranking is completed, the page ranked first will be regarded as a topic entity that can represent the historical conversation.

[0015] As a further improvement of the present invention, in the step 3), the present invention designs a method for online construction of a knowledge graph using a large language model, injecting common sense knowledge into the question generation process, and making the question more knowledge-based. Specifically, the present invention first uses customized prompts to guide the large language model to extract relevant entities and relationships from Wikipedia pages. These prompts typically include contextual information related to the input text, and the output format (such as JSON) expected to be generated by the large language model. Through customized prompts, the method can more effectively utilize the capabilities of the large language model to extract entities and relationships that are closely related to the input text. At the same time, the output in a specific format is also easier to be converted into nodes and edges in a knowledge graph. The present invention then represents entities as nodes and relationships as edges connecting nodes, thereby constructing a knowledge graph.

[0016] As a further improvement of the present invention, in step 4), in order to select meaningful background knowledge, the present invention proposes to evaluate the entity nodes in the knowledge graph from the dimension of node importance. The present invention uses the Page Rank algorithm to analyze the overall structure of the knowledge graph and assigns a weight ω to each node. i Entities with global importance are given higher weights. Then, by running the random walk algorithm and recording the i Number of visits Calculate the importance score of each node:

[0017]

[0018] As a further improvement of the present invention, in step 4), in order to select meaningful background knowledge, the present invention proposes to evaluate the entity nodes in the knowledge graph from the dimension of semantic similarity. The present invention uses BERT to encode the query in step 2) and the definition of each entity, respectively, denoted as q and e i Then calculate the semantic similarity S between the two i , as shown below:

[0019]

[0020] As a further improvement of the present invention, in step 4), since each node similarity S i The range is from -1 to 1, the importance of the invention is i Perform max-min normalization to control I i The threshold range to balance I i and S i The normalized calculation is as follows:

[0021]

[0022] Among them I max =max{I1,I2,...,I n}, I min =min{I1,I2,...,I n}.

[0023] As a further improvement of the present invention, in step 4), in order to select the node most relevant to the current conversation, the present invention designs an algorithm for measuring the relevance of entity nodes, combining node importance and semantic similarity to jointly calculate the final comprehensive relevance score R of each entity. i , the specific formula is as follows:

[0024] R i =I i +β×S i

[0025] Among them, β is a weighting factor that balances the influence between node importance and semantic similarity.

[0026] The present invention selects the entity with the highest score and uses the corresponding entity definition as the introduced Wikipedia knowledge to provide meaningful background knowledge for subsequent question generation, thereby improving accuracy and reliability.

[0027] As a further improvement of the present invention, in step 5), the present invention designs a knowledge fusion operation and innovatively designs a text continuation task, by instructing the large language model to continue writing the previously acquired external Wikipedia knowledge based on the context. Specifically, the system combines the conversation context (including the historical conversation content and the highly relevant entity information calculated from the knowledge graph) with the external Wikipedia knowledge fragment, and then inputs it to the large language model in the form of a prompt. Specifically, the constructed prompt content includes: (1) a historical conversation summary or context description, so that the large language model can understand the scope of topics discussed in the current conversation and the user's intention. (2) The Wikipedia external knowledge text fragment selected in step 4). The above knowledge text is not directly given in parallel, but is instructed to expand or continue writing it as part of the context. In the continuation and expansion process, the large language model will spontaneously try to combine the external Wikipedia knowledge points with the topic context of the current conversation. Specifically, the large language model refers to the given Wikipedia content to extract the key information of the entity under the conversation topic. Connect the topics mentioned before the conversation with external knowledge, so as to naturally embed external knowledge points in the generated text. On the one hand, this knowledge fusion operation can stimulate the large language model to integrate its internal world knowledge and provide more common sense knowledge; on the other hand, by fusing knowledge with context through natural language generation, it can ensure that the generated subsequent questions have strong context relevance.

[0028] Beneficial effects:

[0029] Compared with the prior art, the present invention has the following advantages: 1) The questions generated by the existing methods are often limited to shallow contextual questions, lack inspiration, and are insufficient in information. The present invention designs a strategy to inject common sense knowledge into the question generation process by building a knowledge graph online, making the questions more knowledge-based and rich in information. 2) The subsequent questions generated by the existing methods are not highly correlated with the context, and the dialogue level is low. The present invention innovatively designs a text continuation task, requiring the model to continue to write Wikipedia knowledge according to the context, and fuses the knowledge with the context through natural language generation to ensure that the generated questions have strong context relevance. This knowledge fusion operation can improve the cognitive level of the generated questions and help guide the dialogue to a deeper level. Experimental analysis has proved that the subsequent question generation method based on the knowledge graph and the large language model proposed by the present method has played an improved role in the subsequent question generation model, greatly improved the model performance, and can use background knowledge to generate subsequent questions with richer information and higher cognitive complexity.

[0030] In theory, by dynamically building a knowledge graph and incorporating a large language model in the process of generating subsequent questions, this solution does not rely on specific fields or narrow corpora, and the questions it generates can be continuously iterated as the knowledge base and large language model are updated. This mechanism theoretically provides a basis for the generalization ability of the model, making it more adaptable to emerging concepts, events, and fields. In addition, the construction and fusion process of the knowledge graph can provide a traceable path for knowledge points in question generation, thereby improving the interpretability of the dialogue system. Users can not only obtain in-depth questions, but also trace the source of knowledge behind them when necessary. In actual application scenarios, such as smart assistants, educational Q&A, corporate customer service, and scientific research knowledge Q&A, users often need more in-depth information exploration. This solution stimulates users' desire to explore by naturally introducing new knowledge points in subsequent questions, thereby improving the interaction satisfaction and stickiness between users and the system. In addition, factual knowledge in many scenarios will be continuously updated. This solution is based on the path of "online construction of knowledge graph + continuation of large model knowledge writing", so that the system can integrate the latest data resources at any time. When the knowledge base is updated, there is no need to retrain the system on a large scale. Instead, the system can quickly respond to the latest information by simply injecting, updating or replacing relevant knowledge modules. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a basic process schematic diagram of the present invention;

[0032] Figure 2 is a comparative example diagram of follow-up questions generated by humans and machines of the present invention;

[0033] Figure 3 It is a model framework diagram of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with the embodiments and the accompanying drawings.

[0035] Example:

[0036] The basic process diagram of the subsequent question generation method based on the knowledge graph and large language model of the present invention is as follows: Figure 1 As shown in the figure, given a question-answer pair in a historical conversation, the present invention first identifies the key information of the context and retrieves the most relevant Wikipedia page. Then, a knowledge graph is constructed and the nodes most relevant to the conversation are selected as background knowledge. Finally, the background knowledge is integrated into the process of question generation to guide the large model to generate the final follow-up questions. Specifically, the following steps are included:

[0037] 1) For a given historical conversation, extract subject words and keywords to identify the overall conversation intent. Input a given question-answer pair into the large language model and instruct it to extract key information from the question-answer pair. During the extraction process, the large language model extracts a subject word and n keywords (n≥1). Because the subject word is a highly concise description of the entire question and answer, it is used to capture the user's main question intent, while the keyword extracts finer-grained details from the question and answer. The present invention believes that the combination of subject words and keywords can better simulate the overall conversation and identify the conversation intent. Figure 3 is a model framework diagram of the present invention, showing the three main steps of the present invention - identification, selection and fusion. Figure 3 As shown, in the first step "recognition" module of the present invention, for the input questions and answers, the subject word "fish" and the keywords "gill", "water" and "oxygen" are extracted.

[0038] 2) For the combination of subject terms and keywords obtained in step 1), the core topic is identified by constructing a query to retrieve the most relevant Wikipedia page, and the correct context information is extracted to generate follow-up questions. Wikipedia is a multilingual online encyclopedia whose content covers almost all known fields. The present invention queries Wikipedia based on subject terms and keywords to obtain relevant context entities in an iterative retrieval manner. Specifically, Figure 3 As shown, in the first step "identification" module, after obtaining the combination of subject words and keywords, the present invention first searches for pages whose titles contain subject words, which are recorded as a set Then, the present invention traverses the keywords one by one, adds each keyword to the query condition, and The pages containing the keywords are retrieved until all the keyword traversals are completed. By narrowing the query scope, the present invention obtains the most relevant pages. In the dynamic search process, if a unique matching result is found, the search process is terminated in advance.

[0039] The present invention uses Elasticsearch as a specific implementation of the search engine. Since the word overlap score of the search engine may not accurately reflect the text relevance, the present invention introduces a reranker to rerank the relevant pages. Inspired by previous work, the reranker of the present invention measures the relevance of the page by calculating the following probability:

[0040]

[0041] Among them, Q is the query formed by connecting the subject word and the keyword, θ represents the parameters of the pre-trained language model, and p i represents the definition of the entity in each page. The conditional probability of the entire output sequence Q is calculated by multiplying the conditional probabilities of all time steps, where P(Q t |Q <t ,p i ; θ) is the conditional probability of the model generating the current word at each time step, which can be expressed as:

[0042] P(Q t |Q <t ,p i ; θ) = Softmax(f(Q <t ,x;θ))

[0043] Among them, f(Q <t ,x;θ) is the output of the model, indicating the generation of the current word Q t The scores are converted into probability distributions through the softmax function. The present invention selects the T5 (Text-to-Text Transfer Transformer) model as the basic model of the re-ranker. After the re-ranking is completed, the page ranked first will be regarded as a topic entity that can represent the historical conversation.

[0044] 3) For the core subject entities obtained in step 2), a real-time knowledge graph is constructed to obtain background knowledge related to the conversation. The follow-up questions generated by existing methods are too superficial, such as Figure 2As shown, when talking about eye color, humans can rely on common sense to associate other factors that are not mentioned in the context, such as melanin and iris. However, since the machine can only rely on contextual information, the generated questions are related to the previous context, but the depth and breadth of the content are often insufficient. In order to solve this limitation, the present invention proposes to construct a knowledge graph to introduce relevant knowledge to assist the machine in generating questions. The present invention designs a method for constructing a knowledge graph using a large language model to construct a knowledge graph for candidate pages. Specifically, customized prompts are first used to guide the large language model to extract relevant entities and relationships from the Wikipedia page. These prompts typically include contextual information related to the input text, as well as the output format (such as JSON) that the large language model is expected to generate. Through customized prompts, the method can more effectively utilize the capabilities of the large language model to extract entities and relationships that are closely related to the input text. At the same time, the output in a specific format is also easier to be converted into nodes and edges in the knowledge graph. The present invention then represents entities as nodes and relationships as edges connecting nodes, thereby constructing a knowledge graph. As shown Figure 3 As shown, in the second step "selection" module, the present invention constructs a knowledge graph based on the URL of the Wikipedia page representing the core topic obtained in step 2), and designates the entity "fish" of the page as the central node of the result knowledge graph. Since other nodes of the output knowledge graph, such as "aquatic animals", "marine life", "vertebrates", etc., are associated with the central node, and each node stores the definition of its corresponding entity, the knowledge graph reflects the background knowledge related to the context.

[0045] 4) For the knowledge graph constructed in step 3), evaluate the relevance of entities in the graph and the conversation context in order to select meaningful background knowledge and improve the accuracy and reliability of generating subsequent questions. Since the knowledge graph contains many candidate entities, such as Figure 3 Among the "aquatic animals", "marine life" and "vertebrates", the present invention needs to give priority to the most relevant and well-known entities to ensure that the introduced background knowledge is in line with common sense and relevant to the context. To this end, the present invention designs two evaluation dimensions, namely node importance and semantic similarity.

[0046] For node importance, the present invention uses Page Rank algorithm to analyze the overall structure in the knowledge graph and assigns a weight ω to each node. i Entities with global importance are assigned higher weights.

[0047] Then, by running the Random Walk algorithm and recording the nodes v in the graph i Number of visits Calculate the importance score of each node:

[0048]

[0049] For semantic similarity, the present invention uses BERT to encode the query in step 2) and the definition of each entity, denoted as q and e respectively. i Then calculate the semantic similarity S between the two i , as shown below:

[0050]

[0051] Due to S i The range is from -1 to 1. i Perform max-min normalization to control I i The threshold range to balance I i and S i The normalized calculation is as follows:

[0052]

[0053] Among them I max =max{I1,I2,...,I n}, I min =min{I1,I2,...,I n By combining the importance score of the node I i ′ and semantic similarity S i , the final comprehensive relevance score R for each entity i for:

[0054] R i =I i ′+β×S i

[0055] Among them, β is a weighted factor that balances the influence between node importance and semantic similarity. The present invention selects the entity with the highest score and uses the corresponding entity definition as the introduced Wikipedia knowledge to provide meaningful background knowledge for subsequent question generation, thereby improving accuracy and reliability. Figure 3 As shown in the “Selection” module, the entity with the highest score is “aquatic animals”, and the definition of this entity will be used to assist question generation later.

[0056] 5) Prompt learning is used to instruct the large language model to continue writing the external Wikipedia knowledge obtained in step 4) based on the context, and the fused knowledge is used to guide the generation of subsequent questions. The large language model acquires a large amount of knowledge from a massive text corpus. Some scholars have pointed out that in addition to learning language knowledge, the large language model also learns a large amount of world knowledge (or factual knowledge). Unlike the traditional knowledge base where information is explicitly stored, in these large language models, knowledge is embedded in parameters. The model needs to be properly guided to "speak" knowledge. Inspired by existing work, the present invention uses prompt learning to try to extract knowledge from the large language model. The present invention innovatively designs a text continuation task, requiring the model to continue writing the Wikipedia knowledge output by the selection module in step 4) based on the context. There are two purposes for this: on the one hand, to stimulate the large language model to integrate its internal world knowledge and provide more common sense knowledge; on the other hand, to fuse knowledge with context through natural language generation to ensure that the generated questions have strong context relevance.

[0057] For the continued writing prompt, the present invention designs it as: "Given a question-answer pair: [question], [answer]. Please continue to write the following sentence based on the question-answer pair, and use a few sentences to reflect the relationship with it." During the continued writing process, the large language model conducts in-depth analysis of the context and Wikipedia knowledge to improve the level of understanding of background knowledge.

[0058] Finally, the present invention instructs the large language model to generate a follow-up question based on the known information. In order to obtain high-quality follow-up questions, the present invention provides a clear task description and context in the prompt, including questions, answers, background knowledge fragments, and language style requirements. The specific design is as follows: "Given the following information: [question], [answer], [related knowledge]. Based on this information, please ask a follow-up question that is relevant and thoughtful to the question and answer content." At this point, the present invention obtains the final follow-up question. Figure 3 As shown in the third step “fusion” module of , by giving the above prompt, the large language model generates a follow-up question - "Why do some aquatic animals have gills while others extract oxygen through their skin?".

[0059] The above embodiments are only preferred implementation modes of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and equivalent substitutions without departing from the principles of the present invention. These technical solutions after improvements and equivalent substitutions to the claims of the present invention all fall within the protection scope of the present invention.

[0060] This example uses the following indicators for evaluation:

[0061] For the follow-up question generation task, in order to more comprehensively evaluate the quality of generating follow-up questions, two types of indicators based on automatic evaluation and manual evaluation are used for comparison. In terms of automatic evaluation, the topic consistency (denoted as Consistency), mutual information (denoted as MI), Distinct-n and type tag ratio (denoted as TTR) of the generated follow-up questions are mainly evaluated. The topic consistency is used to reflect the relevance of the follow-up questions. The present invention uses the LDA topic model to extract topics from the conversation context and follow-up questions, selects the top N words with the highest probability in each topic, and calculates their co-occurrence frequency for scoring. The average score of all topics determines the consistency between the context and the question. The higher the score, the stronger the topic relevance. The mutual information is used to reflect the amount of information in the follow-up question, and the mutual information between the follow-up question and the initial question is calculated. When the mutual information is small, the initial question reveals less information about the follow-up question, which means that the generated question contains more information. Distinct-n and type tag ratio are used to evaluate the diversity of generated follow-up questions. In terms of manual evaluation, complexity (denoted as COM.) refers to the complexity of generating subsequent questions, and is scored in the range of 0 to 1. The higher the score, the more complex it is; relevance (denoted as REL.) refers to the degree of association of generating subsequent questions, and is scored in the range of 0 to 1. The higher the score, the more relevant it is; informativeness (denoted as INF.) refers to the richness of information contained in the generated subsequent questions, and is scored in the range of 1 to 3. The higher the score, the richer the information.

[0062] In order to verify the effect of the present invention, the model was evaluated on the FOLLOWUPQG dataset. The experimental results are as follows:

[0063] Table 1 Automatic evaluation experimental results of different follow-up question generation models on the FOLLOWUPQG dataset

[0064]

[0065] Table 2. Experimental results of manual evaluation of different follow-up question generation models on the FOLLOWUPQG dataset

[0066]

[0067] Compared with other methods, the present invention has a significant improvement, especially in generating follow-up questions with richer information and higher cognitive complexity.

[0068] It should be noted that the above embodiments are not intended to limit the protection scope of the present invention, and equivalent changes or substitutions made on the basis of the above technical solutions all fall within the protection scope of the claims of the present invention.

Claims

1. A method for generating follow-up questions based on knowledge graph and large language model, characterized in that: The method comprises the following steps: 1) For a given historical conversation, extract subject words and keywords to identify the overall conversation intent; 2) for the combination of subject terms and keywords obtained in step 1), identify the core topics by constructing queries to retrieve the most relevant Wikipedia pages and extract the correct contextual information to generate follow-up questions; 3) For the core topics obtained in step 2), build a real-time knowledge graph to obtain background knowledge related to the conversation; 4) For the knowledge graph constructed in step 3), evaluate the relevance of entities and conversation context in the graph in order to select meaningful background knowledge and improve the accuracy and reliability of generating subsequent questions; 5) Prompt learning is used to instruct the large language model to continue writing the external Wikipedia knowledge obtained in step 4) based on the context, and the fused multi-source knowledge is used to guide the generation of subsequent questions.

2. The method for generating subsequent questions based on a knowledge graph and a large language model according to claim 1, characterized in that: In the step 1), the overall conversation is simulated by combining subject words and keywords to identify the conversation intent, a given question-answer pair is input into the large language model, and it is instructed to extract key information from the question-answer pair. During the extraction process, the large language model extracts a subject word and n keywords (n≥1).

3. The method for generating subsequent questions based on a knowledge graph and a large language model according to claim 1, characterized in that: In step 2), first search for pages whose titles contain the keyword, which is recorded as a set Then, go through the keywords one by one, add each keyword to the query condition, and The pages containing keywords are retrieved until all keyword traversals are completed. By narrowing the query scope, the relevant pages with the highest correlation are obtained. In the dynamic search process, if a unique matching result is found, the search process is terminated early. In the step 2), the T5 (Text-to-Text Transfer Transformer) model is selected as the basic model of the re-ranker. After the re-ranking is completed, the page ranked first will be regarded as a topic entity that can represent the historical conversation.

4. The method for generating follow-up questions based on a knowledge graph and a large language model according to claim 1, characterized in that: In the step 3), the present invention designs a method for online construction of a knowledge graph using a large language model, injecting common sense knowledge into the question generation process to make the question more knowledge-based. Specifically, the present invention first uses customized prompts to guide the large language model to extract relevant entities and relationships from Wikipedia pages. These prompts usually include contextual information related to the input text and the output format expected to be generated by the large language model. Through customized prompts, the method can more effectively utilize the capabilities of the large language model to extract entities and relationships that are closely related to the input text. At the same time, the output in a specific format is also easier to be converted into nodes and edges in the knowledge graph. Then, the present invention represents entities as nodes and relationships as edges connecting nodes, thereby constructing a knowledge graph.

5. The method for generating subsequent questions based on a knowledge graph and a large language model according to claim 1, characterized in that: In step 4), the Page Rank algorithm is used to analyze the overall structure of the knowledge graph and assign a weight ω to each node. i , entities with global importance are given higher weights, Then, by running the random walk algorithm and recording the nodes v in the graph i Number of visits Calculate the importance score of each node:

6. The method for generating follow-up questions based on a knowledge graph and a large language model according to claim 1, characterized in that: In step 4), BERT is used to encode the query in step 2) and the definition of each entity, respectively, denoted as q and e i , and then calculate the semantic similarity S between the two i , as shown below:

7. The method for generating follow-up questions based on a knowledge graph and a large language model according to claim 1, characterized in that: In step 4), since each node similarity S i The range is from -1 to 1, which indicates the importance of i Perform max-min normalization to control I i The threshold range to balance I i and S i The normalized calculation is as follows: Part I max = max{I1,I2,...,I n }, I min = min{I1,I2,...,I n }.

8. The method for generating follow-up questions based on a knowledge graph and a large language model according to claim 1, characterized in that: In step 4), an algorithm for measuring the relevance of entity nodes is designed, combining node importance and semantic similarity to jointly calculate the final comprehensive relevance score R of each entity. i , the specific formula is as follows: R i =I i +β×S i Among them, β is a weighting factor that balances the influence between node importance and semantic similarity. The entity with the highest score is selected and the corresponding entity definition is used as the introduced Wikipedia knowledge to provide meaningful background knowledge for subsequent question generation, thereby improving accuracy and reliability.

9. The method for generating follow-up questions based on a knowledge graph and a large language model according to claim 1, characterized in that: In the step 5), a knowledge fusion operation is designed, and a text continuation task is innovatively designed. By instructing the large language model to continue writing the previously acquired external Wikipedia knowledge based on the context, specifically, the system will combine the conversation context (including historical conversation content and highly relevant entity information calculated from the knowledge graph) with the external Wikipedia knowledge fragment, and then input it to the large language model in the form of a prompt. Specifically, the constructed prompt content includes: (1) historical conversation summary or context description, so that the large language model can understand the scope of topics discussed in the current conversation and user intentions, and (2) the Wikipedia external knowledge text fragment selected in step 4). The above knowledge text is not directly given in parallel, but is required by instructions to the large language model to expand or continue writing it as part of the context. In the continuation and expansion process, the large language model will spontaneously try to combine the external Wikipedia knowledge points with the topic context of the current conversation. The large language model refers to the given Wikipedia content, extracts the key information of the entity under the conversation topic, and connects the topics mentioned before the conversation with the external knowledge, thereby naturally embedding the external knowledge points in the generated text.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for generating subsequent questions based on the knowledge graph and the large language model as described in any one of claims 1 to 8 above is implemented.

Images