Controllable multi-hop problem generation method, system and equipment based on large language model

Abstract

The invention discloses a method, a system and equipment for generating a controllable multi-hop problem based on a large language model. The method comprises the following steps of: inputting a given context C and a hop count h of a problem to be generated into a pre-established large language model; obtaining support evidence from the context C according to the question, and dividing multi-hop question generation into combined multi-hop question generation and comparative multi-hop question generation according to whether triple predicates in the support evidence are the same or not; and sequentially generating a combined multi-hop question and a comparative multi-hop question by using the large language model to obtain the multi-hop question and an answer thereof. According to the controllable multi-hop problem generation method, the performance is remarkably improved, and the deep structure and semantic information of the problem can be deeply captured, so that the relationships are more accurately understood and processed, and errors and deviations possibly occurring when complex problems are processed are reduced. According to the method, the whole multi-hop problem generation process can be finely controlled, and the generated problems are ensured to be logically coherent and accurate in hop count.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a controllable multi-hop question generation method, system and device based on a large language model. Background Art

[0002] Controllable multi-hop question generation means that based on given text materials and parameters, the machine aggregates and infers multiple discrete information fragments in the text material to generate questions that meet the input parameter requirements. Compared with single-hop question generation, multi-hop question generation is more challenging. It requires the model to understand and integrate multiple information fragments in the context to generate questions that require multi-step reasoning. Controllable multi-hop question generation adds control over the generation process on the basis of multi-hop question generation, that is, the questions generated by the model not only require multi-hop reasoning, but also meet the requirements of specific parameters, such as question type, number of hops, etc. Compared with single-hop question answering datasets, multi-hop question answering datasets usually have longer context lengths and therefore contain richer information. The model needs to eliminate the interference of noise information and filter out relevant information fragments from long texts to generate questions.

[0003] Existing question generation methods mainly include rule-based methods and neural network-based methods. Rule-based methods usually rely on manually designed transformation rules to convert a given text into a corresponding question. This method usually uses templates extracted from the training set to create questions for the corresponding facts in the test set, which is more suitable for use in a specific field. Rule-based question generation methods allow people to have good control over the question generation process with very little data. However, such methods strongly rely on hand-crafted transformation and generation rules, resulting in a significant lack of diversity in the generated questions and lack of scalability in covering different fields. With the rise of data-driven learning methods and the emergence of large-scale datasets, neural network-based question generation methods have gradually become mainstream. Neural network-based question generation models are mainly divided into two types: sequence-to-sequence Seq2Seq deep learning model architecture and graph neural network-based. Most neural network-based question generation models follow the Seq2Seq framework, which first converts the input context text and optional target answers into intermediate representations through an encoder, and then uses a decoder to generate questions from the intermediate representation. In order to solve the problem that Seq2Seq-based models are not good at solving long-distance dependency problems, some studies have begun to focus on question generation models based on graph neural networks. This method usually converts unstructured natural text into a highly structured graph (such as a dependency syntactic graph), and then uses the powerful structural modeling capabilities of graph neural networks to represent long texts and guide question generation. The question generation method based on graph neural networks alleviates the problem that Seq2Seq-based methods are not effective in dealing with long texts. Since the context length of multi-hop question-answering datasets is long, long-distance dependencies will occur in long texts, and there is no good solution yet. Summary of the invention

[0004] The purpose of the present invention is to provide a controllable multi-hop question generation method, system and device based on a large language model to address the problems in the above-mentioned prior art, which can generate multi-hop questions with a specified number of hops with high quality.

[0005] In order to achieve the above object, the present invention has the following technical solutions: In a first aspect, a controllable multi-hop question generation method based on a large language model is provided, comprising: Input a given context C and the number of hops h required to generate a question into the pre-established large language model; According to the question, supporting evidence is obtained from context C, and according to whether the triple predicates in the supporting evidence are the same, multi-hop question generation is divided into combination multi-hop question generation and comparison multi-hop question generation; The large language model is used to generate combinatorial multi-hop questions and comparative multi-hop questions in sequence to obtain multi-hop questions and their answers.

[0006] As a preferred solution, for the generation of the combinatorial multi-hop question, a large language model is used to extract multiple h-hop reasoning chains in a given context C, and the large language model is used to generate multiple h-hop multi-hop questions and their answers based on the extracted reasoning chains.

[0007] As a preferred solution, using a large language model to extract multiple h-hop reasoning chains in a given context C includes: Use a large language model to extract subject-verb-object (SPO) triples from a given context C; According to the extracted subject-verb-object SPO triples, the term frequency-inverse document frequency TF-IDF algorithm is used to merge similar entities and construct a context graph; Extract all connected subgraphs Gsub containing h+1 nodes from the context graph; Randomly sample a node in the connected subgraph Gsub as the answer node; A spanning tree Gt is extracted with the selected answer node as the root node, wherein the spanning tree Gt contains a total of h+1 nodes, and a reasoning chain for generating an h-hop problem is obtained based on the spanning tree Gt containing a total of h+1 nodes.

[0008] As a preferred solution, the step of generating multiple h-hop multi-hop questions and their answers using a large language model according to the extracted reasoning chain includes: The nodes in the reasoning chain are traversed in a hierarchical traversal manner. When the root node is traversed for the first time, the reasoning relationship between the root node and its child nodes is converted into a coherent sentence, and a question is asked with the corresponding sentence as the context and the root node as the answer. Then, the non-leaf nodes in the reasoning chain are traversed continuously. Each time a new non-leaf node is traversed, the reasoning relationship between the new non-leaf node and its child nodes is rewritten into a new sentence, and then the new sentence is used to replace the common reality entity in the question after the last rewriting, so as to obtain a multi-hop question with a clear logical structure.

[0009] As a preferred solution, the step of merging similar entities using the term frequency-inverse document frequency TF-IDF algorithm based on the extracted subject-verb-object SPO triples to construct a context graph includes: For two entities , first calculate the TF-IDF value of each word in the entity; for the entity Words in , calculate the TF-IDF value according to the following expression:

[0010] In the formula, Yes word In the entity IDFk is the word frequency in The inverse document frequency of The TF-IDF value is used to construct the vector representation of the entity. The expression is as follows:

[0011] Based on the vector representation of the entity, the similarity between two entities is calculated using cosine similarity as follows:

[0012] Where v1·v2 is the vector inner product, |v1| and |v2| are the moduli of vectors v1 and v2 respectively; Set a similarity threshold. When the feature similarity of two entities exceeds the similarity threshold, they are considered to refer to the same entity. All entities referring to the same entity are merged to obtain a context graph.

[0013] As a preferred solution, the context graph regards entities in the text as nodes and the relationships between entities as edges, thereby forming a connected graph; each node can be connected to other nodes, so that the relationships between entities can be expressed and organized in multiple ways.

[0014] As a preferred solution, for the comparative multi-hop question generation, a large language model is used to extract entities of predefined entity types and attributes of entities, and h entities are randomly sampled from entities of the same type according to the number of hops h of the question to be generated; Perform attribute screening on the randomly sampled h entities to filter out the attributes that exist in all entities; The h entities and their corresponding attributes and attribute values ​​are constructed into entity-attribute triples, which are then input into the large language model to generate corresponding comparative questions. According to the type of question words, they can be divided into Yes / No questions and Wh questions. The answer to the Yes / No question is "Yes" or "No", while the Wh question provides specific details, descriptions or explanations. When generating comparative multi-hop questions, a Yes / No question is generated first, and then different answers are selected to generate the corresponding Wh question based on whether the attribute values ​​of the entities are the same.

[0015] In a second aspect, a controllable multi-hop question generation system based on a large language model is provided, comprising: A data input module is used to input a given context C and the number of hops h required to generate a question into a pre-established large language model; A question division module is used to obtain supporting evidence from context C according to the question, and to divide the multi-hop question generation into combinational multi-hop question generation and comparative multi-hop question generation according to whether the triple predicates in the supporting evidence are the same; The question generation module is used to use the large language model to sequentially generate combinational multi-hop questions and comparative multi-hop questions to obtain multi-hop questions and their answers.

[0016] As a preferred solution, when generating a combined multi-hop question, the question generation module uses a large language model to extract multiple h-hop reasoning chains in a given context C, and generates multiple h-hop multi-hop questions and their answers based on the extracted reasoning chains using the large language model; When generating a comparative multi-hop question, the question generation module uses a large language model to extract entities of predefined entity types and attributes of the entities, and randomly samples h entities from entities of the same type according to the number of hops h of the question to be generated; Perform attribute screening on the randomly sampled h entities to filter out the attributes that exist in all entities; The h entities and their corresponding attributes and attribute values ​​are constructed into entity-attribute triples, which are then input into the large language model to generate corresponding comparative questions. According to the type of question words, they can be divided into Yes / No questions and Wh questions. The answer to the Yes / No question is "Yes" or "No", while the Wh question provides specific details, descriptions or explanations. When generating comparative multi-hop questions, a Yes / No question is generated first, and then different answers are selected to generate the corresponding Wh question based on whether the attribute values ​​of the entities are the same.

[0017] According to a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the controllable multi-hop question generation method based on a large language model is implemented.

[0018] Compared with the prior art, the present invention has at least the following beneficial effects: In view of the long context length of the multi-hop question-answering dataset, which leads to the problem of long-distance dependency in long texts, the present invention proposes a multi-hop question generation method with controllable hop count based on a large language model. The method of the present invention is free from the reliance on manually annotated supporting evidence. By dividing the multi-hop question generation into combinational multi-hop question generation and comparative multi-hop question generation, and using the large language model to sequentially generate combinational multi-hop questions and comparative multi-hop questions, the combinational multi-hop question generation and comparative multi-hop question generation work together to finally generate a multi-hop question with a specified number of hops. Experiments have proved that the controllable multi-hop question generation method based on a large language model of the present invention has achieved significant performance improvements, can deeply capture the deep structure and semantic information of the problem, so as to more accurately understand and process these relationships, and reduce errors and deviations that may occur when processing complex problems. The controllable multi-hop question generation method of the present invention can finely control the entire multi-hop question generation process, ensure that the generated questions are logically coherent and accurate in hop count, thereby meeting the needs of practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A schematic diagram of the architecture of a controllable multi-hop question generation method based on a large language model according to an embodiment of the present invention; Figure 2 The context graph constructed by the embodiment of the present invention; Figure 3 The embodiment of the present invention uses a large language model to extract a schematic diagram of SPO triples from the context; Figure 4 The embodiment of the present invention utilizes a large language model to extract entity schematic diagrams of predefined entity types; Figure 5 A schematic diagram of an embodiment of the present invention using a large language model to extract attributes of an entity; Figure 6 Schematic diagram of the impact of whether the attribute values ​​of entities are the same on the Wh problem in an embodiment of the present invention; Figure 7 A schematic diagram of a prompt for generating a Yes / No question according to an embodiment of the present invention; Figure 8 Schematic diagram of prompts for generating Wh questions according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, ordinary technicians in this field can also obtain other embodiments without making creative work.

[0022] See also Figure 1 In view of the problem that the context length of a multi-hop question-answering dataset is long, which leads to long-distance dependencies in long texts, an embodiment of the present invention proposes a controllable multi-hop question generation method based on a large language model, including: Input a given context C and the number of hops h required to generate a question into the pre-established large language model; According to the question, supporting evidence is obtained from context C, and according to whether the triple predicates in the supporting evidence are the same, multi-hop question generation is divided into combination multi-hop question generation and comparison multi-hop question generation; The large language model is used to generate combinatorial multi-hop questions and comparative multi-hop questions in sequence to obtain multi-hop questions and their answers.

[0023] In a possible implementation, the model used in the embodiment of the present invention is mainly composed of two modules: combinational multi-hop question generation and comparative question generation. The two modules work together to generate a multi-hop question with a specified number of hops. In the question generation process, the combinational multi-hop question is first generated, and then the comparative multi-hop question is generated.

[0024] (1) Combinatorial multi-hop question generation: First, extract SPO triples from the context, then build a context graph about these triples through the coreference resolution technique, then extract the reasoning chain from the graph as the basis for generating multi-hop questions, and finally use the large language model to gradually rewrite the reasoning chain to generate a multi-hop question that meets the given number of hops. (2) Comparative multi-hop question generation: First, extract entities of the same type and their attributes, then filter out the attributes that exist in all entities, and finally construct these entities, attributes, and attribute values ​​into entity-attribute triples, and input them into the large language model to generate the corresponding comparative questions.

[0025] In the step of obtaining supporting evidence from context C according to the question, the supporting evidence is the information related to answering the question.

[0026] Further, for the generation of combined multi-hop problems, the embodiment of the present invention includes the following steps: 1.1 Reasoning Chain Extraction With the advancement of computer technology, research on large language models in the field of open information extraction has made significant progress. Open information extraction aims to extract triple information such as entities and relations from unstructured texts without pre-defining entity types and relationship categories. Since the large language model is exposed to texts of various fields and types during the pre-training stage, it has accumulated rich language knowledge and deep semantic information. This enables the large language model to deeply understand and extract the complex relationships implied in the text in different fields and diverse scenarios without the need for separate training for each specific field.

[0027] The embodiment of the present invention uses a large language model to extract SPO triples from the context, and constructs prompts such as Figure 3 After successfully extracting the SPO triples of the context, the TF-IDF algorithm is further used to merge similar entities. Specifically, suppose there are two entities, namely Before calculating feature similarity, we need to calculate the TF-IDF value of each word in the entity. Words in , its TF-IDF value is calculated as follows:

[0028] in, Yes word In the entity IDFk is the word frequency in The inverse document frequency of .

[0029] Then use the TF-IDF value to construct the vector representation of the entity. The specific formula is:

[0030] Next, use cosine similarity to calculate the similarity between two entities. Cosine similarity measures the similarity between two vectors by calculating the angle between them. The formula is:

[0031] Where: v1·v2 is the vector inner product, |v1| and |v2| are the moduli of vectors v1 and v2 respectively.

[0032] Finally, the embodiment of the present invention sets the similarity threshold to 0.4, that is, when the feature similarity of two entities exceeds 0.4, they are considered to refer to the same entity. After merging all co-referencing entities, the following is obtained: Figure 2 The context-based context graph G shown in the figure helps to solve the long-distance dependency problem in long texts. In long texts, there are complex associations and dependencies between entities, and these relationships often span multiple sentences or paragraphs. By constructing a context graph, the entities in the text are regarded as nodes, and the relationships between entities are regarded as edges, thus forming a connected graph. Each node can be connected to other nodes, so that the relationships between entities can be expressed and organized in a variety of ways. This graphical representation method helps to extract important information from long texts and makes the complex relationships in the text clearer and easier to understand.

[0033] After building the context graph, we need to extract the reasoning chain from the graph: First, all connected subgraphs Gsub containing h+1 nodes are extracted from the context graph. These connected subgraphs constitute the candidate set of reasoning chains. Next, in order to generate a specific reasoning chain, a node is randomly sampled in the connected subgraph Gsub as the answer node. Then, a spanning tree Gt is extracted with the selected answer node as the root node. The spanning tree contains a total of h+1 nodes. Through this method, reasoning chains for generating h-hop questions are constructed. These reasoning chains not only reflect the deep relationship between entities, but also clearly show the reasoning process through the tree structure.

[0034] 1.2 Gradually rewrite After extracting the reasoning chains, the next step is to generate multi-hop questions based on these reasoning chains. In order to improve the interpretability of multi-hop question generation, a step-by-step rewriting method is used to generate multi-hop questions with a specific number of hops.

[0035] Specifically, the nodes in the reasoning chain are traversed in a layer-order traversal manner. When the root node is traversed for the first time, the reasoning relationship between the root node and its child nodes is converted into a coherent sentence, and a question is asked with the sentence as the context and the root node as the answer. Subsequently, the non-leaf nodes in the reasoning chain continue to be traversed, and each time a new non-leaf node is traversed, the reasoning relationship between the non-leaf node and its child node is rewritten into a new sentence, and then the new sentence is used to replace the common entity (i.e., the bridge entity) in the question after the last rewrite. Through this method, the embodiment of the present invention can not only generate a multi-hop question with a clear logical structure, but also ensure the interpretability and coherence of the question.

[0036] Furthermore, in the multi-hop problem reasoning type, in addition to the combination type, there is also a comparison type, which usually involves the comparison of entity attributes. For the generation of comparison multi-hop problems, the embodiment of the present invention includes the following steps: First, the entities of predefined entity types and their attributes are extracted with the help of a large language model. The specific entities and their attribute prompts are as follows: Figure 4 and Figure 5 As shown in the figure. After the entity attributes are extracted, h entities are randomly sampled from the same entity according to the input hop count h. Then the sampled entities are screened for attributes to ensure that the screened attributes exist in all these entities. This step is crucial because if an attribute does not exist on one of the entities, the attribute information of the entity cannot be accurately obtained from the context, which makes the generated question unanswerable. These h entities and their corresponding attributes and attribute values ​​together constitute the core elements for generating comparative multi-hop questions.

[0037] Through case analysis in the dataset, it is found that comparative multi-hop questions can be divided into two categories according to the type of question words: Yes / No questions and Wh questions. The answer to Yes / No questions is usually a simple "Yes" or "No", while Wh questions require the respondent to provide specific details, descriptions or explanations. Such questions usually begin with special question words such as "What", "Where", "When", "Howmany", etc. Furthermore, it can be observed that whether the attribute values ​​of the entities are the same has a significant impact on the answer to the Wh question. Specifically, if Figure 6As shown in , when the attribute values ​​to be compared are the same, the question tends to be asked with the attribute value of the entity as the answer; when the attribute values ​​are different, the question tends to be asked with the entity itself as the answer. This difference reflects the importance of attribute value identity in the generation of Wh questions. However, for Yes / No questions, whether the attribute values ​​are the same does not affect them. Therefore, when generating comparison questions, a Yes / No question is generated first; then, depending on whether the attribute values ​​of the entities are the same, different answers are selected to generate the corresponding Wh questions. The prompts generated for Yes / No questions and Wh questions are shown below. Figure 7 and Figure 8 shown.

[0038] In addition, examples learned in context have a significant impact on the quality of generated questions. This is because examples provide contextual information for model learning, and the quality and quantity of this information directly determine what the model can understand and learn. First, high-quality examples can help the model better capture the inherent laws and patterns of the data, thereby improving the performance of the model. These examples are usually representative and can reflect the true distribution of the data, so that the model can learn effective features and representations during training. Second, diverse examples can enhance the generalization ability of the model. When the model faces new, unseen data, if it can recall similar examples it has encountered during training, it will be easier to understand and process this data. Therefore, diverse examples can make the model better adapt to a variety of different scenarios and situations.

[0039] Another embodiment of the present invention further provides a controllable multi-hop question generation system based on a large language model, comprising: A data input module is used to input a given context C and the number of hops h required to generate a question into a pre-established large language model; A question division module is used to obtain supporting evidence from context C according to the question, and to divide the multi-hop question generation into combinational multi-hop question generation and comparative multi-hop question generation according to whether the triple predicates in the supporting evidence are the same; The question generation module is used to use the large language model to sequentially generate combinational multi-hop questions and comparative multi-hop questions to obtain multi-hop questions and their answers.

[0040] In a possible implementation, when generating a combined multi-hop question, the question generation module uses a large language model to extract multiple h-hop reasoning chains from a given context C, and generates multiple h-hop multi-hop questions and their answers based on the extracted reasoning chains using the large language model; When generating a comparative multi-hop question, the question generation module uses a large language model to extract entities of predefined entity types and attributes of the entities, and randomly samples h entities from entities of the same type according to the number of hops h of the question to be generated; Perform attribute screening on the randomly sampled h entities to filter out the attributes that exist in all entities; The h entities and their corresponding attributes and attribute values ​​are constructed into entity-attribute triples, which are then input into the large language model to generate corresponding comparative questions. According to the type of question words, they can be divided into Yes / No questions and Wh questions. The answer to the Yes / No question is "Yes" or "No", while the Wh question provides specific details, descriptions or explanations. When generating comparative multi-hop questions, a Yes / No question is generated first, and then different answers are selected to generate the corresponding Wh question based on whether the attribute values ​​of the entities are the same.

[0041] The effect of the controllable multi-hop question generation method based on a large language model in an embodiment of the present invention is analyzed through comparative experiments.

[0042] The embodiment of the present invention conducts relevant experiments on the multi-hop question-answering dataset R4C, focusing on the generation of questions without answers. In order to effectively evaluate the performance of the model, the entity most similar to the labeled answer is selected as the answer node to guide the generation of multi-hop questions. Specifically, the TF-IDF algorithm is used to calculate the feature similarity between the entity in the reasoning chain and the labeled answer in the dataset, and then the most similar entity node is selected as the predicted answer. In addition, for a context, there are often multiple reasoning chains, which means that multiple questions can be generated. To this end, the entity relationship pairs on the reasoning chain are spliced ​​together and compared with the supporting evidence given in the dataset. Then, the reasoning chain with the highest score is selected with the help of the BLEU-1 indicator to generate the corresponding multi-hop questions. For comparative questions, since the labeled questions in the data are usually one of the Yes / No type and the Wh type, the corresponding questions are further selected according to the answer type for evaluation to enhance the pertinence and accuracy of the evaluation. By decomposing the multi-hop question generation task into a series of carefully designed intermediate subtasks, the embodiment of the present invention achieves a significant improvement in performance, as shown in Table 1, and is significantly better than ReasoningCircuits in the comparative experiment. By gradually and orderly decomposing complex multi-hop reasoning relationships, the model can deeply capture the deep structure and semantic information of the problem, so as to more accurately understand and process these relationships and reduce the errors and deviations that may occur when processing complex problems. Therefore, this decomposition strategy plays an important role in improving the performance of the model in multi-hop question generation tasks.

[0043] Table 1

[0044] In order to better evaluate the quality of question generation, the embodiment of the present invention also conducts manual evaluation, and the results of manual evaluation are shown in Table 2. By analyzing Table 2, it can be seen that in terms of fluency, although the method of the embodiment of the present invention is slightly inferior to the direct use of GPT-3.5-Turbo for question generation, it still significantly surpasses the other two comparison models. This difference may be mainly attributed to the use of a templated method to convert triples into sentences. Although the use of templates improves the degree of structured generation of multi-hop questions, it also limits the naturalness and diversity of sentences to a certain extent, thereby affecting the overall evaluation of fluency. In terms of answerability, the model of the present invention shows a significant advantage, which is significantly better than the other three baseline models. This advantage is mainly due to the designed question evaluation module, which effectively guarantees the answerability of the generated questions, thereby improving the effectiveness and practicality of the questions. In addition, in terms of the number of hops of multi-hop questions, the model also shows a strong ability. By first generating simple questions, and then rewriting them step by step based on these simple questions, multi-hop questions with increasing hops are generated. This strategy enables the model to finely control the entire multi-hop question generation process, ensuring that the generated questions are logically coherent and accurate in the number of hops, thus meeting the needs of practical applications.

[0045] Table 2

[0046] In order to further explore the impact of the number of context learning samples on model performance, experiments were conducted using 1, 3, and 5 samples, and the experimental results are shown in Table 3. Through analysis, it is found that with the increase in the number of samples, the performance of the model on multiple evaluation indicators shows a significant upward trend.

[0047] Specifically, when the number of samples increased from 1 to 3, a significant improvement in model performance was observed. BLEU-1, BLEU-2, BLEU-3, BLEU-4, ROUGE-L, and METEOR indicators increased by 10.4%, 8.2%, 6.6%, 5.8%, 6.7%, and 5.2%, respectively. When the number of samples further increased from 3 to 5, although the improvement slowed down, the performance of the model in various indicators still maintained a certain improvement.

[0048] Experimental results show the importance of the number of contextual learning examples in stimulating the potential of large language models. By increasing the number of contextual learning examples, large language models can be exposed to more diverse real-world situations and examples, thereby learning and mastering the key information and patterns in the task more deeply. This expansion of learning helps improve the generalization ability of the model, enabling it to make more accurate and reliable predictions when faced with unknown data.

[0049] Table 3

[0050] An embodiment of the present invention further proposes an electronic device, comprising: a memory storing at least one instruction; and a processor executing the instruction stored in the memory to implement the controllable multi-hop question generation method based on a large language model.

[0051] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the controllable multi-hop question generation method based on a large language model is implemented.

[0052] Exemplarily, the instructions stored in the memory may be divided into one or more modules / units, which are stored in a computer-readable storage medium and executed by the processor to complete the controllable multi-hop question generation method based on a large language model of the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the server.

[0053] The electronic device may be a computing device such as a smart phone, a notebook, a PDA, and a cloud server. The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the electronic device may also include more or fewer components, or a combination of certain components, or different components, for example, the electronic device may also include an input / output device, a network access device, a bus, etc.

[0054] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0055] The memory may be an internal storage unit of the server, such as a hard disk or memory of the server. The memory may also be an external storage device of the server, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the server. Furthermore, the memory may include both an internal storage unit of the server and an external storage device. The memory is used to store the computer-readable instructions and other programs and data required by the server. The memory may also be used to temporarily store data that has been output or is to be output.

[0056] It should be noted that the information interaction, execution process and other contents between the above-mentioned module units are based on the same concept as the method embodiment. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0057] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0058] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0059] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0060] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A controllable multi-hop question generation method based on a large language model, characterized in that: include: Input a given context C and the number of hops h required to generate a question into the pre-established large language model; According to the question, supporting evidence is obtained from context C, and according to whether the triple predicates in the supporting evidence are the same, multi-hop question generation is divided into combination multi-hop question generation and comparison multi-hop question generation; The large language model is used to generate combinatorial multi-hop questions and comparative multi-hop questions in sequence to obtain multi-hop questions and their answers.

2. The controllable multi-hop question generation method based on a large language model according to claim 1 is characterized in that: For the combined multi-hop question generation, a large language model is used to extract multiple h-hop reasoning chains in a given context C, and multiple h-hop multi-hop questions and their answers are generated using the large language model based on the extracted reasoning chains.

3. The controllable multi-hop question generation method based on a large language model according to claim 2 is characterized in that: The reasoning chain of extracting multiple h jumps in a given context C using a large language model includes: Use a large language model to extract subject-verb-object (SPO) triples from a given context C; According to the extracted subject-verb-object SPO triples, the term frequency-inverse document frequency TF-IDF algorithm is used to merge similar entities and construct a context graph; Extract all connected subgraphs Gsub containing h+1 nodes from the context graph; Randomly sample a node in the connected subgraph Gsub as the answer node; A spanning tree Gt is extracted with the selected answer node as the root node, wherein the spanning tree Gt contains a total of h+1 nodes, and a reasoning chain for generating an h-hop problem is obtained based on the spanning tree Gt containing a total of h+1 nodes.

4. The controllable multi-hop question generation method based on a large language model according to claim 3 is characterized in that: The step of generating a multi-hop question of multiple h hops and its answer by using a large language model according to the extracted reasoning chain comprises: The nodes in the reasoning chain are traversed in a hierarchical traversal manner. When the root node is traversed for the first time, the reasoning relationship between the root node and its child nodes is converted into a coherent sentence, and a question is asked with the corresponding sentence as the context and the root node as the answer. Then, the non-leaf nodes in the reasoning chain are traversed continuously. Each time a new non-leaf node is traversed, the reasoning relationship between the new non-leaf node and its child nodes is rewritten into a new sentence, and then the new sentence is used to replace the common reality entity in the question after the last rewriting, so as to obtain a multi-hop question with a clear logical structure.

5. The controllable multi-hop question generation method based on a large language model according to claim 3 is characterized in that: The step of merging similar entities using the term frequency-inverse document frequency TF-IDF algorithm according to the extracted subject-verb-object SPO triples to construct a context graph includes: For two entities , first calculate the TF-IDF value of each word in the entity; for the entity Words in , calculate the TF-IDF value according to the following expression: In the formula, Yes word In the entity IDFk is the word frequency in The inverse document frequency of Use the TF-IDF value to construct the vector representation of the entity. The expression is as follows: Based on the vector representation of the entity, the similarity between two entities is calculated using cosine similarity as follows: Where v1·v2 is the vector inner product, |v1| and |v2| are the moduli of vectors v1 and v2 respectively; Set a similarity threshold. When the feature similarity of two entities exceeds the similarity threshold, they are considered to refer to the same entity. All entities referring to the same entity are merged to obtain a context graph.

6. The controllable multi-hop question generation method based on a large language model according to claim 5 is characterized in that: The context graph regards entities in the text as nodes and the relationships between entities as edges, thereby forming a connected graph; each node can be connected to other nodes, so that the relationships between entities can be expressed and organized in multiple ways.

7. The controllable multi-hop question generation method based on a large language model according to claim 1 is characterized in that: For the comparative multi-hop question generation, a large language model is used to extract entities of predefined entity types and attributes of entities, and h entities of the same type are randomly sampled according to the number of hops h of the question to be generated; Perform attribute screening on the randomly sampled h entities to filter out the attributes that exist in all entities; The h entities and their corresponding attributes and attribute values ​​are constructed into entity-attribute triples, which are then input into the large language model to generate corresponding comparative questions. According to the type of question words, they can be divided into Yes / No questions and Wh questions. The answer to Yes / No questions is "Yes" or "No", while Wh questions provide specific details, descriptions, or explanations. When generating comparative multi-hop questions, a Yes / No question is generated first, and then different answers are selected to generate corresponding Wh questions based on whether the attribute values ​​of the entities are the same.

8. A controllable multi-hop question generation system based on a large language model, characterized in that: include: A data input module is used to input a given context C and the number of hops h required to generate a question into a pre-established large language model; A question division module is used to obtain supporting evidence from context C according to the question, and to divide the multi-hop question generation into combinational multi-hop question generation and comparative multi-hop question generation according to whether the triple predicates in the supporting evidence are the same; The question generation module is used to use the large language model to sequentially generate combinational multi-hop questions and comparative multi-hop questions to obtain multi-hop questions and their answers.

9. The controllable multi-hop question generation system based on a large language model according to claim 8, characterized in that: When generating a combined multi-hop question, the question generation module uses the large language model to extract multiple h-hop reasoning chains in a given context C, and generates multiple h-hop multi-hop questions and their answers based on the extracted reasoning chains using the large language model; When generating a comparative multi-hop question, the question generation module uses a large language model to extract entities of predefined entity types and attributes of the entities, and randomly samples h entities from entities of the same type according to the number of hops h of the question to be generated; Perform attribute screening on the randomly sampled h entities to filter out the attributes that exist in all entities; The h entities and their corresponding attributes and attribute values ​​are constructed into entity-attribute triples, which are then input into the large language model to generate corresponding comparative questions. According to the type of question words, they can be divided into Yes / No questions and Wh questions. The answer to Yes / No questions is "Yes" or "No", while Wh questions provide specific details, descriptions, or explanations. When generating comparative multi-hop questions, a Yes / No question is generated first, and then different answers are selected to generate corresponding Wh questions based on whether the attribute values ​​of the entities are the same.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the controllable multi-hop question generation method based on a large language model as described in any one of claims 1 to 7 is implemented.

Images