Controllable multi-hop problem generation method, system and device based on large language model

Abstract

A controllable multi-hop question generation method, system and device based on a large language model, the method comprising inputting a given context C and the number of hops h that need to generate questions into a pre-established large language model; obtaining supporting evidence from the context C according to the questions, and dividing the multi-hop question generation into combined multi-hop question generation and comparative multi-hop question generation according to whether the triple predicates in the supporting evidence are the same; generating combined multi-hop questions and comparative multi-hop questions in turn using the large language model to obtain multi-hop questions and their answers. The controllable multi-hop question generation method of the present application has achieved significant improvement in performance, can deeply capture the deep structure and semantic information of the question, thereby more accurately understanding and processing these relationships, and reducing errors and biases that may occur when processing complex questions. The present application can finely control the entire multi-hop question generation process to ensure that the generated questions are logically coherent and accurate in the number of hops.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a controllable multi-hop question generation method, system and device based on a large language model. Background Art

[0002] Controllable multi-hop question generation refers to the process of generating questions that meet the input parameter requirements by aggregating and reasoning multiple discrete pieces of information in the text material based on given text materials and parameters. Compared with single-hop question generation, multi-hop question generation is more challenging. It requires the model to understand and integrate multiple pieces of information in the context to generate questions that require multi-step reasoning. Controllable multi-hop question generation adds control over the generation process based on multi-hop question generation. That is, it requires the questions generated by the model to not only require multi-hop reasoning, but also meet specific parameter requirements, such as question type and number of hops. Compared with single-hop question answering datasets, multi-hop question answering datasets are generally longer in context and therefore contain richer information. The model needs to eliminate the interference of noisy information and filter out relevant information fragments from long text to generate questions.

[0003] Existing question generation methods primarily include rule-based and neural network-based approaches. Rule-based methods typically rely on manually designed transformation rules to convert a given text into a corresponding question. These methods typically utilize templates extracted from a training set to create questions for corresponding facts in a test set. These methods are generally well-suited for use within a specific domain. Rule-based question generation methods allow for fine control over the question generation process using minimal data. However, their strong reliance on handcrafted transformation and generation rules results in a significant lack of diversity in the generated questions and a lack of scalability across diverse domains. With the rise of data-driven learning methods and the availability of large-scale datasets, neural network-based question generation methods have gradually become mainstream. Neural network-based question generation models are primarily categorized into sequence-to-sequence (Seq2Seq) deep learning model architectures and graph neural network-based models. Most neural network-based question generation models follow the Seq2Seq framework, which first transforms the input context text and optional target answer into an intermediate representation via an encoder, and then uses a decoder to generate questions from this intermediate representation. To address the poor performance of Seq2Seq-based models in addressing long-range dependencies, some research has focused on question generation models based on graph neural networks. This approach typically converts unstructured natural text into a highly structured graph (such as a dependency graph). Graph neural networks then leverage the powerful structural modeling capabilities of graphs to represent long texts and guide question generation. This approach alleviates the ineffectiveness of Seq2Seq-based approaches for long texts. Due to the long context length of multi-hop question-answering datasets, long-distance dependencies can arise in long texts, and a robust solution remains unresolved. Summary of the Invention

[0004] The purpose of the present invention is to address the problems in the above-mentioned prior art and provide a controllable multi-hop question generation method, system and device based on a large language model, which can generate multi-hop questions with a specified number of hops with high quality.

[0005] In order to achieve the above object, the present invention has the following technical solutions:

[0006] In a first aspect, a controllable multi-hop question generation method based on a large language model is provided, comprising:

[0007] Input the given context C and the number of hops h required to generate the question into the pre-established large language model;

[0008] According to the question, supporting evidence is obtained from the context C, and according to whether the triple predicates in the supporting evidence are the same, multi-hop question generation is divided into combinational multi-hop question generation and comparative multi-hop question generation;

[0009] A large language model is used to sequentially generate combinatorial multi-hop questions and comparative multi-hop questions to obtain multi-hop questions and their answers.

[0010] As a preferred solution, for the generation of the combined multi-hop question, a large language model is used to extract multiple h-hop reasoning chains in a given context C, and the large language model is used to generate multiple h-hop multi-hop questions and their answers based on the extracted reasoning chains.

[0011] As a preferred solution, using a large language model to extract multiple h-hop reasoning chains in a given context C includes:

[0012] Use a large language model to extract subject-verb-object (SPO) triples from a given context C;

[0013] Based on the extracted subject-verb-object (SPO) triples, similar entities are merged using the term frequency-inverse document frequency (TF-IDF) algorithm to construct a context graph.

[0014] Extract all connected subgraphs Gsub containing h+1 nodes from the context graph;

[0015] Randomly sample a node in the connected subgraph Gsub as the answer node;

[0016] A spanning tree Gt is extracted with the selected answer node as a root node, wherein the spanning tree Gt contains a total of h+1 nodes. Based on the spanning tree Gt containing a total of h+1 nodes, a reasoning chain for generating an h-hop problem is obtained.

[0017] As a preferred solution, the step of generating multiple h-hop multi-hop questions and their answers using a large language model based on the extracted reasoning chain includes:

[0018] The nodes in the reasoning chain are traversed using a level-order traversal approach. When the root node is traversed for the first time, the reasoning relationship between the root node and its child nodes is converted into a coherent sentence, and a question is asked with the corresponding sentence as the context and the root node as the answer. The non-leaf nodes in the reasoning chain are then traversed. Each time a new non-leaf node is traversed, the reasoning relationship between the new non-leaf node and its child nodes is rewritten into a new sentence. The new sentence is then used to replace the common reality entity in the question after the last rewriting, resulting in a multi-hop question with a clear logical structure.

[0019] As a preferred solution, the step of merging similar entities using the term frequency-inverse document frequency (TF-IDF) algorithm based on the extracted subject-verb-object (SPO) triples to construct a context graph includes:

[0020] For two entities , first calculate the TF-IDF value of each word in the entity; for the entity Words in , calculate the TF-IDF value according to the following expression:

[0021]

[0022] Where, is a word In the entity The word frequency in IDFk is the word The inverse document frequency of

[0023] The TF-IDF value is used to construct the vector representation of the entity. The expression is as follows:

[0024]

[0025] Based on the vector representation of the entity, the similarity between two entities is calculated using cosine similarity as follows:

[0026]

[0027] Where v1·v2 is the vector inner product, |v1| and |v2| are the moduli of vectors v1 and v2 respectively;

[0028] Set a similarity threshold. When the feature similarity of two entities exceeds the similarity threshold, they are considered to refer to the same entity.

[0029] Merge all entities that refer to the same entity to obtain a context graph.

[0030] As a preferred solution, the context graph regards entities in the text as nodes and the relationships between entities as edges, thereby forming a connected graph; each node can be connected to other nodes, so that the relationships between entities can be expressed and organized in multiple ways.

[0031] As a preferred solution, for the generation of comparative multi-hop questions, a large language model is used to extract entities of predefined entity types and their attributes, and h entities are randomly sampled from entities of the same type according to the number of hops h to generate the question.

[0032] Perform attribute screening on the randomly sampled h entities to filter out the attributes that exist in all entities;

[0033] H entities and their corresponding attributes and attribute values ​​are constructed into entity-attribute triples and input into the large language model to generate corresponding comparative questions. Based on the type of question word, they can be divided into two categories: Yes / No questions and Wh questions. Yes / No questions can answer "Yes" or "No", while Wh questions provide specific details, descriptions, or explanations. When generating comparative multi-hop questions, a Yes / No question is first generated, and then different answers are selected based on whether the attribute values ​​of the entities are the same to generate the corresponding Wh questions.

[0034] In a second aspect, a controllable multi-hop question generation system based on a large language model is provided, comprising:

[0035] The data input module is used to input a given context C and the number of hops h required to generate questions into the pre-established large language model;

[0036] The question division module is used to obtain supporting evidence from the context C according to the question, and divide the multi-hop question generation into combinational multi-hop question generation and comparative multi-hop question generation according to whether the triple predicates in the supporting evidence are the same;

[0037] The question generation module is used to use the large language model to sequentially generate combinational multi-hop questions and comparative multi-hop questions to obtain multi-hop questions and their answers.

[0038] As a preferred solution, when generating a combined multi-hop question, the question generation module uses a large language model to extract multiple h-hop reasoning chains from a given context C, and then uses the large language model to generate multiple h-hop multi-hop questions and their answers based on the extracted reasoning chains.

[0039] When generating comparative multi-hop questions, the question generation module uses a large language model to extract entities of predefined entity types and their attributes, and randomly samples h entities from the same type according to the number of hops h of the question to be generated;

[0040] Perform attribute screening on the randomly sampled h entities to filter out the attributes that exist in all entities;

[0041] H entities and their corresponding attributes and attribute values ​​are constructed into entity-attribute triples and input into the large language model to generate corresponding comparative questions. Based on the type of question word, they can be divided into two categories: Yes / No questions and Wh questions. Yes / No questions can answer "Yes" or "No", while Wh questions provide specific details, descriptions, or explanations. When generating comparative multi-hop questions, a Yes / No question is first generated, and then different answers are selected based on whether the attribute values ​​of the entities are the same to generate the corresponding Wh questions.

[0042] In a third aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the computer program implements the controllable multi-hop question generation method based on a large language model.

[0043] Compared with the prior art, the present application has at least the following beneficial effects:

[0044] In view of the long context length of the multi-hop question and answer dataset, which leads to long-distance dependency in long text, the present application provides a multi-hop question generation method based on a large language model with controllable number of hops. The present application method does not rely on artificially annotated supporting evidence. The multi-hop question generation is divided into combined multi-hop question generation and comparative multi-hop question generation. A large language model is used to generate combined multi-hop questions and comparative multi-hop questions in sequence. The combined multi-hop question generation and comparative multi-hop question generation work cooperatively to finally generate multi-hop questions with a specified number of hops. Through experiments, it is proved that the controllable multi-hop question generation method based on a large language model of the present application has achieved significant improvement in performance. It can deeply capture the deep structure and semantic information of the question, so as to more accurately understand and process these relationships, and reduce errors and biases that may occur when processing complex problems. The controllable multi-hop question generation method of the present application can finely control the generation process of the entire multi-hop question, ensure that the generated question is logically coherent and accurate in the number of hops, thereby meeting the needs of practical applications. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and other related drawings can also be obtained by those skilled in the art without creative labor.

[0046] Figure 1 The controllable multi-hop question generation method based on a large language model of the embodiments of the present application is shown in the schematic diagram.

[0047] Figure 2 The context graph constructed by the embodiments of the present application;

[0048] Figure 3 The SPO triple extraction from the context by the large language model in the embodiments of the present application is shown in the schematic diagram.

[0049] Figure 4 The entity extraction of the predefined entity type by the large language model in the embodiments of the present application is shown in the schematic diagram.

[0050] Figure 5 The attribute extraction of the entity by the large language model in the embodiments of the present application is shown in the schematic diagram.

[0051] Figure 6 Schematic diagram showing the impact of whether the attribute values ​​of entities are the same on the Wh problem according to an embodiment of the present invention;

[0052] Figure 7 Schematic diagram of prompts generated for Yes / No questions according to an embodiment of the present invention;

[0053] Figure 8 Schematic diagram of prompts generated by the Wh question according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, ordinary technicians in this field can also derive other embodiments without making any creative work.

[0055] See also Figure 1 To address the problem of long-distance dependencies in long texts caused by long context lengths in multi-hop question-answering datasets, an embodiment of the present invention proposes a controllable multi-hop question generation method based on a large language model, including:

[0056] Input the given context C and the number of hops h required to generate the question into the pre-established large language model;

[0057] According to the question, supporting evidence is obtained from the context C, and according to whether the triple predicates in the supporting evidence are the same, multi-hop question generation is divided into combinational multi-hop question generation and comparative multi-hop question generation;

[0058] A large language model is used to sequentially generate combinatorial multi-hop questions and comparative multi-hop questions to obtain multi-hop questions and their answers.

[0059] In one possible implementation, the model used in this embodiment of the present invention primarily consists of two modules: a combinatorial multi-hop question generation module and a comparative question generation module. These two modules work together to generate a multi-hop question with a specified number of hops. During the question generation process, the combinatorial multi-hop question is generated first, followed by the comparative multi-hop question.

[0060] (1) Combinatorial multi-hop question generation: First, extract SPO triples from the context, then construct a context graph about these triples through the coreference resolution technique, then extract the reasoning chain from the graph as the basis for generating multi-hop questions, and finally use the large language model to gradually rewrite the reasoning chain to generate a multi-hop question that meets the given number of hops. (2) Comparative multi-hop question generation: First, extract entities of the same type and their attributes, then filter out attributes that exist in all entities, and finally construct these entities, attributes, and attribute values ​​into entity-attribute triples, which are input into the large language model to generate the corresponding comparative questions.

[0061] In the step of obtaining supporting evidence from context C according to the question, the supporting evidence is information related to answering the question.

[0062] Furthermore, for generating a combined multi-hop problem, the embodiment of the present invention includes the following steps:

[0063] 1.1 Reasoning Chain Extraction

[0064] With advances in computer technology, research on large language models in the field of open information extraction has made significant progress. Open information extraction aims to extract triples of information, such as entities and relationships, from unstructured text without pre-defining entity types and relationship categories. Because large language models are exposed to text from various fields and types during pre-training, they accumulate rich linguistic knowledge and deep semantic information. This enables large language models to deeply understand and extract complex relationships implicit in text across different fields and diverse scenarios, without requiring separate training for each specific field.

[0065] The embodiment of the present invention uses a large language model to extract SPO triples from the context, and constructs prompts such as Figure 3 After successfully extracting the contextual SPO triples, we further use the TF-IDF algorithm to merge similar entities. Specifically, suppose there are two entities, namely Before calculating the feature similarity, we need to calculate the TF-IDF value of each word in the entity. Words in , its TF-IDF value is calculated as follows:

[0066]

[0067] in, is a word In the entity The word frequency in IDFk is the word The inverse document frequency of .

[0068] Then use the TF-IDF value to construct the vector representation of the entity. The specific formula is:

[0069]

[0070] Next, the cosine similarity is used to calculate the similarity between two entities. Cosine similarity measures the degree of similarity between two vectors by calculating the angle between them, and the formula is:

[0071]

[0072] In the formula, v1·v2 is the inner product of the vectors, and |v1| and |v2| are the norms of vectors v1 and v2, respectively.

[0073] Finally, the embodiment of the present application sets the similarity threshold to 0.4, that is, when the similarity of the features of two entities exceeds 0.4, they are considered to refer to the same entity. After merging all the co-reference entities, the context-based context graph G is obtained as shown in Figure 2 The construction of the context graph helps to solve the long-distance dependency problem in long text. In long text, there are complex relationships and dependencies between entities, which often span multiple sentences or paragraphs. By constructing a context graph, the entities in the text are treated as nodes, and the relationships between the entities are treated as edges, forming a connected graph. Each node can be connected to other nodes, so that the relationships between entities can be expressed and organized in multiple ways. This graphical representation method helps to extract important information in long text, making the complex relationships in the text more clear and understandable.

[0074] After the context graph is constructed, the reasoning chain needs to be extracted from the graph:

[0075] First, all connected subgraphs Gsub containing h+1 nodes are extracted from the context graph, which constitute the candidate set of reasoning chains. Next, in order to generate specific reasoning chains, a node is randomly sampled from the connected subgraph Gsub as the answer node. Then a spanning tree Gt containing h+1 nodes is extracted with the selected answer node as the root node. Through this method, reasoning chains for generating h-hop problems are constructed. These reasoning chains not only reflect the deep relationships between entities, but also clearly show the reasoning process through the tree structure.

[0076] 1.2 Step-by-step rewriting

[0077] After extracting the reasoning chain, the next step is to generate multi-hop questions based on these reasoning chains. In order to improve the explainability of multi-hop question generation, a step-by-step rewriting method is used to generate multi-hop questions with a specific number of hops.

[0078] Specifically, the nodes in the inference chain are traversed in a sequence traversal manner. When the root node is first traversed, the inference relationship between the root node and its child nodes is converted into a coherent sentence, and a question is proposed with the sentence as the context and the root node as the answer. Subsequently, the non-leaf nodes in the inference chain are continuously traversed, and when a new non-leaf node is traversed each time, the inference relationship between the non-leaf node and its child nodes is rewritten into a new sentence, and then the co-occurring entity (i.e., the bridge entity) in the question in the last rewriting is replaced by the new sentence. Through this method, the embodiments of the present application can not only generate multi-hop questions with clear logical structure, but also ensure the explainability and coherence of the questions.

[0079] Further, in the multi-hop question reasoning type, in addition to the combination type, there is also a comparison type, and the question of this type usually involves comparison of entity attributes. For the generation of comparison type multi-hop questions, the embodiments of the present application include the following steps:

[0080] First, the entities of a predefined entity type and their attributes are extracted with the aid of a large language model, and the specific entity and its attribute prompts are shown in Figure 4 and Figure 5 After the extraction of entity attributes is completed, h entities are randomly sampled from the same type of entities according to the input number of hops h. Then, attribute filtering is performed on the sampled entities to ensure that the filtered attributes exist in these entities. This step is crucial because if an attribute does not exist in one of the entities, the attribute information of the entity cannot be accurately obtained from the context, which will result in a question that cannot be answered. The h entities and their corresponding attributes, attribute values collectively constitute the core elements for generating comparison type multi-hop questions.

[0081] Through analysis of cases in the dataset, it is found that comparison type multi-hop questions can be divided into Yes / No questions and Wh questions according to the type of interrogative word. The answer to a Yes / No question is usually simply "Yes" or "No", while a Wh question requires the answerer to provide specific details, descriptions or explanations. Such questions usually start with special interrogative words such as "What", "Where", "When", "How many", etc. Further, it can be observed that whether the attribute values of the entities are the same has a significant impact on the answer to a Wh question. Specifically, as shown in Figure 6As shown in , when the attribute values ​​to be compared are the same, questions tend to be asked with the attribute values ​​of the entity as the answer; when the attribute values ​​are different, questions tend to be asked with the entity itself as the answer. This difference reflects the importance of attribute value identity in the generation of Wh questions. However, for Yes / No questions, whether the attribute values ​​are the same or not does not affect them. Therefore, when generating comparative questions, a Yes / No question is generated first; then, depending on whether the attribute values ​​of the entities are the same, different answers are selected to generate the corresponding Wh questions. The prompts generated for Yes / No questions and Wh questions are as follows: Figure 7 and Figure 8 shown.

[0082] Furthermore, examples learned in context have a significant impact on the quality of generated questions. This is because examples provide contextual information for model learning, and the quality and quantity of this information directly determine what the model can understand and learn. First, high-quality examples help the model better capture the inherent regularities and patterns of the data, thereby improving model performance. These examples are often representative and reflect the true distribution of the data, allowing the model to learn effective features and representations during training. Second, diverse examples can enhance the model's generalization ability. When a model encounters new, unseen data, it will be more likely to understand and process this data if it can recall similar examples encountered during training. Therefore, diverse examples enable the model to better adapt to a variety of scenarios and situations.

[0083] Another embodiment of the present invention further provides a controllable multi-hop question generation system based on a large language model, comprising:

[0084] The data input module is used to input a given context C and the number of hops h required to generate questions into the pre-established large language model;

[0085] The question division module is used to obtain supporting evidence from the context C according to the question, and divide the multi-hop question generation into combinational multi-hop question generation and comparative multi-hop question generation according to whether the triple predicates in the supporting evidence are the same;

[0086] The question generation module is used to use the large language model to sequentially generate combinational multi-hop questions and comparative multi-hop questions to obtain multi-hop questions and their answers.

[0087] In one possible implementation, when generating a combined multi-hop question, the question generation module uses a large language model to extract multiple h-hop reasoning chains from a given context C, and generates multiple h-hop multi-hop questions and their answers based on the extracted reasoning chains using the large language model.

[0088] When generating comparative multi-hop questions, the question generation module uses a large language model to extract entities of predefined entity types and their attributes, and randomly samples h entities from the same type according to the number of hops h of the question to be generated;

[0089] Perform attribute screening on the randomly sampled h entities to filter out the attributes that exist in all entities;

[0090] H entities and their corresponding attributes and attribute values ​​are constructed into entity-attribute triples and input into the large language model to generate corresponding comparative questions. Based on the type of question word, they can be divided into two categories: Yes / No questions and Wh questions. Yes / No questions can answer "Yes" or "No", while Wh questions provide specific details, descriptions, or explanations. When generating comparative multi-hop questions, a Yes / No question is first generated, and then different answers are selected based on whether the attribute values ​​of the entities are the same to generate the corresponding Wh questions.

[0091] The following comparative experiments analyze the effect of the controllable multi-hop question generation method based on a large language model in an embodiment of the present invention.

[0092] The present invention conducts experiments on the multi-hop question-answering dataset R4C, focusing on generating unanswered questions. To effectively evaluate the model's performance, entities most similar to labeled answers are selected as answer nodes to guide the generation of multi-hop questions. Specifically, the TF-IDF algorithm is used to calculate the feature similarity between entities in the reasoning chain and the labeled answers in the dataset. The most similar entity node is then selected as the predicted answer. Furthermore, a context often has multiple reasoning chains, which means that multiple questions can be generated. To this end, the entity relationship pairs in the reasoning chains are concatenated and compared with the supporting evidence provided in the dataset. The highest-scoring reasoning chain is then selected using the BLEU-1 metric to generate the corresponding multi-hop question. For comparative questions, since labeled questions in the data are typically either Yes / No or Wh, corresponding questions are further selected based on the answer type for evaluation, enhancing the relevance and accuracy of the evaluation. By decomposing the multi-hop question generation task into a series of carefully designed intermediate subtasks, the present invention achieves significant performance improvements, as shown in Table 1, and significantly outperforms ReasoningCircuits in comparative experiments. By gradually and orderly decomposing complex multi-hop reasoning relationships, the model can deeply capture the deep structure and semantic information of the problem, thereby more accurately understanding and processing these relationships and reducing the errors and biases that may occur when processing complex problems. Therefore, this decomposition strategy plays an important role in improving the model's performance on multi-hop question generation tasks.

[0093] Table 1

[0094]

[0095] To better evaluate the quality of question generation, the present invention also conducted a manual evaluation. The results are shown in Table 2. Analysis of Table 2 shows that, while the present invention's method is slightly inferior to directly applying GPT-3.5-Turbo for question generation in terms of fluency, it still significantly outperforms the other two comparison models. This difference may be primarily due to the use of a template-based approach to convert triples into sentences. While the use of templates improves the structured nature of multi-hop question generation, it also limits the naturalness and diversity of sentences, which in turn affects the overall fluency assessment. In terms of answerability, the present invention's model demonstrates significant advantages, significantly outperforming the other three baseline models. This advantage is primarily due to the designed question evaluation module, which effectively ensures the answerability of generated questions, thereby improving their effectiveness and practicality. Furthermore, the present model demonstrates strong capabilities in controlling the number of hops in multi-hop questions. By first generating simple questions and then rewriting them step by step, it generates multi-hop questions with increasing hops. This strategy enables the model to finely control the entire multi-hop question generation process, ensuring that the generated questions are logically coherent and the number of hops is accurate, thus meeting the needs of practical applications.

[0096] Table 2

[0097]

[0098] To further explore the impact of the number of contextual learning examples on model performance, experiments were conducted using 1, 3, and 5 examples, and the experimental results are shown in Table 3. The analysis found that as the number of examples increased, the performance of the model on multiple evaluation indicators showed a significant upward trend.

[0099] Specifically, when the number of examples increased from 1 to 3, a significant improvement in model performance was observed. BLEU-1, BLEU-2, BLEU-3, BLEU-4, ROUGE-L, and METEOR metrics increased by 10.4%, 8.2%, 6.6%, 5.8%, 6.7%, and 5.2%, respectively. When the number of examples further increased from 3 to 5, although the rate of improvement slowed, the model's performance across all metrics maintained some improvement.

[0100] Experimental results demonstrate the importance of the number of contextual learning examples in unlocking the potential of large language models. By increasing the number of contextual learning examples, large language models are exposed to a wider variety of real-world scenarios and examples, enabling them to more deeply learn and grasp key information and patterns in the task. This expanded learning approach helps improve the model's generalization capabilities, enabling it to make more accurate and reliable predictions when faced with unknown data.

[0101] Table 3

[0102]

[0103] An embodiment of the present invention further proposes an electronic device, comprising: a memory storing at least one instruction; and a processor executing the instruction stored in the memory to implement the controllable multi-hop question generation method based on a large language model.

[0104] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the controllable multi-hop question generation method based on a large language model is implemented.

[0105] Exemplarily, the instructions stored in the memory can be divided into one or more modules / units, which are stored in a computer-readable storage medium and executed by the processor to implement the controllable multi-hop question generation method based on a large language model described in the present invention. The one or more modules / units can be a series of computer-readable instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program in the server.

[0106] The electronic device may be a computing device such as a smartphone, laptop, PDA, or cloud server. The electronic device may include, but is not limited to, a processor and memory. Those skilled in the art will appreciate that the electronic device may include more or fewer components, or a combination of certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, and the like.

[0107] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0108] The memory may be an internal storage unit of the server, such as a hard disk or memory of the server. The memory may also be an external storage device of the server, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the server. Furthermore, the memory may include both an internal storage unit of the server and an external storage device. The memory is used to store the computer-readable instructions and other programs and data required by the server. The memory may also be used to temporarily store data that has been output or is about to be output.

[0109] It should be noted that the information interaction, execution process, etc. between the above-mentioned module units are based on the same concept as the method embodiment. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0110] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0111] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0112] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0113] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A controllable multi-hop question generation method based on a large language model, characterized in that: include: Input the given context C and the number of hops h required to generate the question into the pre-established large language model; According to the question, supporting evidence is obtained from the context C, and according to whether the triple predicates in the supporting evidence are the same, multi-hop question generation is divided into combinational multi-hop question generation and comparative multi-hop question generation; Use the large language model to sequentially generate combinatorial multi-hop questions and comparative multi-hop questions to obtain multi-hop questions and their answers; For the generation of the combined multi-hop question, a large language model is used to extract multiple h-hop reasoning chains from a given context C, and based on the extracted reasoning chains, the large language model is used to generate multiple h-hop multi-hop questions and their answers; The step of generating a multi-hop question with multiple h hops and its answer using a large language model according to the extracted reasoning chain includes: The nodes in the reasoning chain are traversed using a layer-by-layer traversal approach. When the root node is first reached, the reasoning relationship between the root node and its child nodes is converted into a coherent sentence. A question is then posed using the corresponding sentence as the context and the root node as the answer. The non-leaf nodes in the reasoning chain are then traversed. Each time a new non-leaf node is reached, the reasoning relationship between the new non-leaf node and its child nodes is rewritten into a new sentence. The new sentence is then used to replace the co-existing entity in the question after the previous rewriting, resulting in a multi-hop question with a clear logical structure. For the comparative multi-hop question generation, a large language model is used to extract entities of predefined entity types and their attributes, and h entities of the same type are randomly sampled according to the number of hops h of the question to be generated. Perform attribute screening on the randomly sampled h entities to filter out the attributes that exist in all entities; H entities and their corresponding attributes and attribute values ​​are constructed into entity-attribute triples and input into a large language model to generate corresponding comparative questions. Based on the question word type, the questions are divided into two categories: Yes / No questions and Wh questions. Yes / No questions have a "Yes" or "No" answer, while Wh questions provide specific details, descriptions, or explanations. When generating comparative multi-hop questions, a Yes / No question is first generated, and then, depending on whether the attribute values ​​of the entities are the same, different answers are selected to generate corresponding Wh questions.

2. The controllable multi-hop question generation method based on a large language model according to claim 1 is characterized in that: The reasoning chain of extracting multiple h jumps in a given context C using a large language model includes: Use a large language model to extract subject-verb-object (SPO) triples from a given context C; Based on the extracted subject-verb-object (SPO) triples, similar entities are merged using the term frequency-inverse document frequency (TF-IDF) algorithm to construct a context graph. Extract all connected subgraphs G containing h+1 nodes from the context graph sub ; In the connected subgraph G sub Randomly sample a node as the answer node; A spanning tree Gt is extracted with the selected answer node as a root node, wherein the spanning tree Gt contains a total of h+1 nodes. Based on the spanning tree Gt containing a total of h+1 nodes, a reasoning chain for generating an h-hop problem is obtained.

3. The controllable multi-hop question generation method based on a large language model according to claim 2 is characterized in that: The step of merging similar entities using the term frequency-inverse document frequency (TF-IDF) algorithm based on the extracted subject-verb-object (SPO) triples to construct a context graph includes: For two entities , first calculate the TF-IDF value of each word in the entity; for the entity Words in , calculate the TF-IDF value according to the following expression: Where, is a word In the entity The word frequency in is a word The inverse document frequency of The TF-IDF value is used to construct the vector representation of the entity. The expression is as follows: Based on the vector representation of the entity, the similarity between two entities is calculated using cosine similarity as follows: Where v1·v2 is the vector inner product, |v1| and |v2| are the moduli of vectors v1 and v2 respectively; Set a similarity threshold. When the feature similarity of two entities exceeds the similarity threshold, they are considered to refer to the same entity. Merge all entities that refer to the same entity to obtain a context graph.

4. The controllable multi-hop question generation method based on a large language model according to claim 3 is characterized in that: The context graph regards entities in the text as nodes and the relationships between entities as edges, thereby forming a connected graph; each node can be connected to other nodes, so that the relationships between entities can be expressed and organized in multiple ways.

5. A controllable multi-hop question generation system based on a large language model, characterized by: Applying the controllable multi-hop question generation method based on a large language model as described in any one of claims 1 to 4, comprising: The data input module is used to input a given context C and the number of hops h required to generate questions into the pre-established large language model; The question division module is used to obtain supporting evidence from the context C according to the question, and divide the multi-hop question generation into combinational multi-hop question generation and comparative multi-hop question generation according to whether the triple predicates in the supporting evidence are the same; The question generation module is used to use the large language model to sequentially generate combinational multi-hop questions and comparative multi-hop questions to obtain multi-hop questions and their answers.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the controllable multi-hop question generation method based on a large language model as claimed in any one of claims 1 to 4 is implemented.

Images