Method and device for generating operation and maintenance knowledge test questions based on thinking tree

Abstract

The invention provides an operation and maintenance knowledge test question generation method and device based on a thinking tree, and relates to the technical field of question and answer evaluation. The method comprises the following steps: preprocessing an operation and maintenance knowledge document to obtain multiple segments of knowledge texts; aiming at each section of knowledge text, constructing a thinking tree taking the knowledge text as a root node; the thinking tree comprises three layers of child nodes; the first layer of child nodes are questions related to the knowledge text, the second layer of child nodes are answers corresponding to the questions, and the third layer of child nodes are interference items corresponding to the questions; and obtaining a plurality of operation and maintenance knowledge test questions based on the constructed thinking tree. The task for generating the operation and maintenance knowledge test questions is decomposed into a plurality of simple sub-tasks based on the thinking tree construction process, so that the process of automatically generating a plurality of operation and maintenance knowledge test questions based on the operation and maintenance knowledge document is realized, the labor cost can be greatly reduced, and the generation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of question-answering evaluation, and in particular to a method and device for generating operation and maintenance knowledge test questions based on a thinking tree. Background Art

[0002] In daily operation and maintenance work, there are a large number of operation and maintenance knowledge documents (such as operation manuals, development documents, etc.), and they occupy an important position. However, general large language models often have deficiencies in new knowledge or domain knowledge, and the retrieval enhancement generation system can make up for this shortcoming. By answering questions related to operation and maintenance knowledge, it assists operation and maintenance work and becomes a powerful tool for improving operation and maintenance efficiency. In order to evaluate the ability of the retrieval enhancement generation system in operation and maintenance scenarios, it is necessary to design special operation and maintenance knowledge test questions.

[0003] In the prior art, operation and maintenance knowledge test questions are designed manually. However, since the operation and maintenance field covers multiple sub-fields, the design and verification of test questions requires the participation of many field experts, resulting in high costs. Summary of the invention

[0004] The embodiment of the present invention provides a method and device for generating operation and maintenance knowledge test questions based on a mind tree, so as to reduce the dependence of operation and maintenance knowledge test questions on manual work and reduce costs.

[0005] In a first aspect, an embodiment of the present invention provides a method for generating operation and maintenance knowledge test questions based on a thinking tree, comprising: preprocessing an operation and maintenance knowledge document to obtain multiple knowledge texts; for each knowledge text, constructing a thinking tree with the knowledge text as the root node; the thinking tree comprises three layers of sub-nodes; the first layer of sub-nodes are questions related to the knowledge text, the second layer of sub-nodes are answers corresponding to the questions, and the third layer of sub-nodes are interference items corresponding to the questions; based on the constructed thinking tree, multiple operation and maintenance knowledge test questions are obtained.

[0006] In some embodiments, the construction of a thinking tree with the knowledge text as the root node includes: calling a large language model to generate questions according to the knowledge text, obtaining multiple candidate questions, and determining the question based on the multiple candidate questions to construct the first-layer child nodes; calling the large language model multiple times to generate answers according to the knowledge text and the question, obtaining multiple candidate answers, and determining the answer based on the multiple candidate answers to construct the second-layer child nodes; calling the large language model to generate interference items according to the knowledge text, the question and the answer, obtaining multiple candidate interference items, and determining the interference items based on the multiple candidate interference items to construct the third-layer child nodes.

[0007] In some embodiments, determining the question based on the multiple candidate questions includes: for each candidate question, determining a first score of the candidate question based on the knowledge text and the candidate question; and determining the question from the multiple candidate questions based on the first score.

[0008] As a possible implementation manner, determining the first score of the candidate question based on the knowledge text and the candidate question includes: constructing a first scoring prompt word for the knowledge text and the candidate question, and obtaining a first relevance score, completeness score, and clarity score for the candidate question by calling the large language model; and determining the first score of the candidate question based on the first relevance score, completeness score, and clarity score of the candidate question.

[0009] In some embodiments, determining the answer based on the multiple candidate answers includes: determining the frequency of occurrence of each of the candidate answers; and determining the answer based on the frequency of occurrence of each of the candidate answers.

[0010] In some embodiments, determining the interference items based on the multiple candidate interference items includes: for each of the candidate interference items, determining a second score of the candidate interference item based on the knowledge text, the question, the answer and the candidate interference item; and determining the interference item from the multiple candidate interference items based on the second score.

[0011] As a possible implementation manner, the determining the second score of the candidate interference item based on the knowledge text, the question, the answer and the candidate interference item includes: constructing a second scoring prompt word based on the knowledge text, the question, the answer and the candidate interference item, and obtaining a second relevance score, a distinguishability score and a distraction score of the candidate interference item by calling the large language model; and determining the second score of the candidate interference item based on the second relevance score, the distinguishability score and the distraction score of the candidate interference item.

[0012] In some embodiments, obtaining a plurality of operation and maintenance knowledge test questions based on the constructed thinking tree includes: determining a plurality of initial operation and maintenance knowledge test questions based on the constructed thinking tree; for each of the initial operation and maintenance knowledge test questions, determining a quality score of the initial operation and maintenance knowledge test question based on a first score of the question, a second score of the interference item, and a third score of the answer in the initial operation and maintenance knowledge test question; and determining the plurality of operation and maintenance knowledge test questions from the plurality of initial operation and maintenance knowledge test questions based on the quality scores.

[0013] As a possible implementation method, the third score of the answer in the initial operation and maintenance knowledge test question is obtained in the following manner: constructing a third prompt word based on the knowledge text, question and answer corresponding to the initial operation and maintenance knowledge test question, and obtaining the accuracy score, conciseness score and adaptability score of the answer in the initial operation and maintenance knowledge test question by calling the large language model; determining the third score of the answer in the initial operation and maintenance knowledge test question based on the accuracy score, conciseness score and adaptability score of the answer in the initial operation and maintenance knowledge test question.

[0014] In the second aspect, an embodiment of the present invention provides an operation and maintenance knowledge test question generation device based on a thinking tree, comprising: a preprocessing module, used to preprocess the operation and maintenance knowledge document to obtain multiple knowledge texts; a construction module, used to construct a thinking tree with the knowledge text as the root node for each knowledge text; the thinking tree includes three layers of sub-nodes; the first layer of sub-nodes are questions related to the knowledge text, the second layer of sub-nodes are answers corresponding to the questions, and the third layer of sub-nodes are interference items corresponding to the questions; an acquisition module, used to obtain multiple operation and maintenance knowledge test questions based on the constructed thinking tree.

[0015] According to the method and device for generating operation and maintenance knowledge test questions based on a mind tree provided by an embodiment of the present invention, the operation and maintenance knowledge document is preprocessed to obtain multiple knowledge texts; for each knowledge text, a mind tree with the knowledge text as the root node is constructed; the mind tree includes three layers of sub-nodes; the first layer of sub-nodes are questions related to the knowledge text, the second layer of sub-nodes are answers corresponding to the questions, and the third layer of sub-nodes are interference items corresponding to the questions; based on the constructed mind tree, multiple operation and maintenance knowledge test questions are obtained. The present invention decomposes the task of generating operation and maintenance knowledge test questions into multiple simple sub-tasks based on the process of constructing a mind tree, so as to realize the process of automatically generating multiple operation and maintenance knowledge test questions based on the operation and maintenance knowledge document, thereby greatly reducing labor costs and improving generation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 One of the flow charts of the method for generating operation and maintenance knowledge test questions based on a mind tree provided in an embodiment of the present invention; Figure 2 A second flow chart of a method for generating operation and maintenance knowledge test questions based on a mind tree according to an embodiment of the present invention; Figure 3 A third flowchart of a method for generating operation and maintenance knowledge test questions based on a mind tree according to an embodiment of the present invention; Figure 4 A fourth flowchart of a method for generating operation and maintenance knowledge test questions based on a mind tree according to an embodiment of the present invention; Figure 5 A fifth flowchart of a method for generating operation and maintenance knowledge test questions based on a mind tree provided in an embodiment of the present invention; Figure 6 Schematic diagram of the structure of the thinking tree in the embodiment of the present invention; Figure 7 A schematic diagram of the structure of a device for generating operation and maintenance knowledge test questions based on a thinking tree provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] With the rapid development of information technology, the scale and complexity of IT (Information Technology) systems are also increasing, making intelligent operation and maintenance gradually become a new trend in the field of operation and maintenance. Intelligent operation and maintenance introduces artificial intelligence technology to help enterprises or operation and maintenance personnel efficiently complete IT operation and maintenance tasks, thereby ensuring the stable and orderly operation of IT systems. In recent years, the rise of large language models has brought new possibilities for intelligent operation and maintenance. As a representative achievement in the field of natural language processing and artificial intelligence, large language models have been widely used in tasks such as text classification, question-answering systems, and language translation due to their powerful language understanding and generation capabilities. However, large language models also have significant limitations, such as a lack of understanding of the latest information in a specific field, because the training data of large language models is usually based on a wide range of general corpora. To solve this problem, the retrieval enhancement generation system can significantly improve the quality and accuracy of question-answering results by retrieving external knowledge bases combined with large language model generation technology.

[0020] In daily operation and maintenance work, there are a large number of operation and maintenance knowledge documents (such as operation manuals, development documents, etc.), and they occupy an important position. However, general large language models often have deficiencies in new knowledge or domain knowledge, and the retrieval enhancement generation system can make up for this shortcoming. By answering questions related to operation and maintenance knowledge, it assists operation and maintenance work and becomes a powerful tool to improve operation and maintenance efficiency. In order to evaluate the ability of the retrieval enhancement generation system in the operation and maintenance scenario, it is necessary to design special operation and maintenance knowledge test questions, but there are mainly the following three problems: 1) High labor costs. The operation and maintenance field covers multiple sub-fields, and the design and verification of test questions require the participation of many domain experts, making it difficult to control costs. 2) Regular updates are required. The operation and maintenance documents are updated frequently, so the test data set needs to be updated regularly to prevent the information from being outdated. 3) The complexity boundary is difficult to define. Different individuals and organizations have different understandings of the ability boundaries of the retrieval enhancement generation system, which makes it difficult to clarify the reasonable standards for the complexity of the questions.

[0021] At present, there are two technical means to generate operation and maintenance knowledge test questions, namely the method based on the step-by-step model and the method based on the end-to-end model. Among them, the method based on the step-by-step model. This method generally breaks down the question generation process into different subtasks, and then builds training samples and labels for each subtask, and fine-tunes the pre-trained language model or the large language model. Finally, according to the designed generation process, each subtask model is called in turn to generate different parts of the question, and these parts are merged into a complete knowledge test question. The method based on the end-to-end model usually relies on the powerful generation capability of the large language model, and directly generates complete test questions through prompt word engineering, including the question stem, answers or options.

[0022] However, the main defects of the step-by-step model-based method are: first, each subtask requires sample annotation, resulting in high labor costs. Each subtask requires the construction of an independent data set, and the construction of these data sets usually relies on the participation of domain experts. This not only requires a lot of human resources, but also takes a lot of time, making the annotation process very expensive and inefficient; second, the effectiveness of each subtask depends on the diversity of samples, but diversity is difficult to guarantee. Samples can usually be constructed manually, and this construction process itself is difficult and costly. In addition, even if you try to use a generative model to generate samples, the diversity and quality of the generated results are difficult to fully meet expectations, further limiting the performance improvement of the model; third, there are multiple subtask models, resulting in high maintenance costs in the later stage. Each model needs to be optimized, updated and debugged separately. When the number of models is large, this maintenance work will become extremely difficult and time-consuming.

[0023] The main drawbacks of the end-to-end model-based approach are: on the one hand, generating questions is a relatively complex task, and the generation effect of large models is difficult to guarantee. In order to improve the generation quality, it is usually necessary to design clear and specific prompts for the model, including task descriptions, step breakdowns, constraints, etc., to guide the model to generate expected results. However, it may still be limited by the lack of professional capabilities of large models in specific fields, making the generated results unable to fully meet actual needs; on the other hand, in order to simplify the difficulty of the generation task, existing methods often generate question-and-answer questions, which increases the difficulty of subsequent evaluation.

[0024] In order to solve the above problems, an embodiment of the present invention provides a method and device for generating operation and maintenance knowledge test questions based on a thinking tree.

[0025] Figure 1 This is a flow chart of a method for generating operation and maintenance knowledge test questions based on a mind tree according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps.

[0026] Step 101, pre-processing the operation and maintenance knowledge document to obtain multiple knowledge texts.

[0027] Among them, operation and maintenance knowledge documents can include official operation manuals, enterprise-owned operation and maintenance documents, development documents, etc. Operation and maintenance knowledge documents usually have clear hierarchical structures and knowledge points are highly related to the document structure. Therefore, knowledge texts can be extracted based on the structure of operation and maintenance knowledge documents.

[0028] As an example, each paragraph in the operation and maintenance knowledge document can be regarded as a piece of knowledge text, each subsection in the operation and maintenance knowledge document can be regarded as a piece of knowledge text, and each chapter in the operation and maintenance knowledge document can be regarded as a piece of knowledge text.

[0029] Step 102, for each knowledge text, construct a thinking tree with the knowledge text as the root node; the thinking tree includes three layers of sub-nodes; the first layer of sub-nodes are questions related to the knowledge text, the second layer of sub-nodes are answers corresponding to the questions, and the third layer of sub-nodes are interference items corresponding to the questions.

[0030] That is to say, the task is decomposed by constructing a thinking tree, that is, the task is split into question generation subtasks, answer generation subtasks and interference item generation subtasks, and the growth of the thinking tree is controlled by the execution of subtasks to realize the construction of the thinking tree.

[0031] In some embodiments, each piece of knowledge text will construct a thinking tree, and each thinking tree is equivalent to one or more operation and maintenance knowledge test questions generated based on a piece of knowledge text. Each test question contains questions, answers and interference items, and each test question comes from the corresponding knowledge text.

[0032] In some embodiments, the process of constructing a thinking tree with a knowledge text as a root node may include: presetting initialization parameters of the thinking tree, such as the initial number and the reserved number of child nodes in each layer; setting prompt words based on the knowledge text and the initial number n1 of the first-layer child nodes, generating questions through a large language model, and generating n1 candidate questions, that is, n1 first-layer candidate child nodes; based on the relevance between the n1 candidate questions and the knowledge text, and the reserved number m1 of the first-layer child nodes, determining m1 questions from the n1 candidate questions, and using the m1 questions as m1 first-layer child nodes; for any first-layer child node A among the m1 first-layer child nodes, setting prompt words based on the question corresponding to the first-layer child node A, the knowledge text, and the initial number n2 of the second-layer child nodes corresponding to the first-layer child node A, and generating answers through a large language model, Obtain n2 candidate answers, and based on the adaptability of each candidate answer to the question and the retained number m2 of second-layer subnodes corresponding to the first-layer subnode A, determine m2 answers from the n2 candidate answers, and use the m2 answers as the m2 second-layer subnodes of the first-layer subnode A; for any second-layer subnode B among the m2 second-layer subnodes, based on the answer corresponding to the second-layer subnode B, the question corresponding to the answer, the knowledge text, and the initial number n3 of third-layer subnodes corresponding to the second-layer subnode B, generate interference items through a large language model to obtain n3 candidate interference items, and based on the interference between each interference item and the answer and the question, and the retained number m3 of third-layer subnodes corresponding to the second-layer subnode B, determine m3 interference items from the n3 candidate interference items, and use the m3 interference items as the m3 third-layer subnodes of the second-layer subnode B.

[0033] Step 103, based on the constructed thinking tree, a plurality of operation and maintenance knowledge test questions are obtained.

[0034] It can be understood that the sub-nodes of different levels of each constructed mind tree can form one or more operation and maintenance knowledge test questions, and each operation and maintenance knowledge test question includes a question, one or more answers, and one or more interference items. It should be noted that the operation and maintenance knowledge test questions generated in the embodiment of the present invention are multiple-choice questions.

[0035] In some embodiments, multiple operation and maintenance knowledge test questions can be determined directly based on the three-layer sub-nodes in each thinking tree, or the operation and maintenance knowledge test questions determined by the three-layer sub-nodes in each thinking tree can be screened to obtain the final operation and maintenance knowledge test questions.

[0036] As an example, if each thinking tree contains 1 root node, 5 first-level child nodes, 5 second-level child nodes, and 15 third-level child nodes, each first-level child node corresponds to 1 second-level child node, and each second-level child node corresponds to 3 third-level child nodes, 5 questions can be determined based on the 5 first-level child nodes, and the answer corresponding to each question can be determined based on the second-level child nodes corresponding to each first-level child node. Based on the 3 third-level child nodes corresponding to each second-level child node, the 3 interference items corresponding to each answer can be determined. Therefore, 5 operation and maintenance knowledge test questions can be obtained, and each operation and maintenance test question contains a question, an answer, and three interference items.

[0037] According to the method for generating operation and maintenance knowledge test questions based on a mind tree provided by an embodiment of the present invention, the operation and maintenance knowledge document is preprocessed to obtain multiple knowledge texts; for each knowledge text, a mind tree with the knowledge text as the root node is constructed; the mind tree includes three layers of sub-nodes; the first layer of sub-nodes are questions related to the knowledge text, the second layer of sub-nodes are answers corresponding to the questions, and the third layer of sub-nodes are interference items corresponding to the questions; based on the constructed mind tree, multiple operation and maintenance knowledge test questions are obtained. The present invention decomposes the task of generating operation and maintenance knowledge test questions into multiple simple sub-tasks based on the process of constructing a mind tree, so as to realize the process of automatically generating multiple operation and maintenance knowledge test questions based on the operation and maintenance knowledge document, thereby greatly reducing labor costs and improving generation efficiency.

[0038] Figure 2 The second flowchart of the method for generating operation and maintenance knowledge test questions based on the mind tree provided in the embodiment of the present invention. Figure 2 As shown, based on the above embodiments, Figure 1 The implementation process of constructing a thinking tree with the knowledge text as the root node in step 102 may include the following steps.

[0039] Step 201, calling a large language model to generate questions according to the knowledge text, obtaining multiple candidate questions, and determining questions based on the multiple candidate questions to construct the first layer of sub-nodes.

[0040] In some embodiments, question generation prompt words can be constructed based on knowledge text and a preset number of questions N. By calling a large language model, N candidate questions are obtained, and the N candidate questions are determined as N questions, with each question as a first-level child node, that is, N first-level child nodes are obtained.

[0041] It should be noted that each piece of knowledge text can generate N test questions, each test question corresponds to a question, and the size of N can be set based on actual needs.

[0042] In some embodiments, a question generation prompt word can be constructed based on the knowledge text and the preset initial number N1 of first-layer subnodes, and N1 candidate questions can be obtained by calling the large language model, and the relevance score of each candidate question can be determined based on the knowledge text and the N1 candidate questions through the large language model, and M1 questions can be determined from the N1 candidate questions based on the relevance score of each candidate question, and each question can be used as a first-layer subnode, that is, M1 first-layer subnodes can be obtained. Among them, M1 is the preset reserved number of first-layer subnodes.

[0043] As an example, since the generated questions need to be answered from the knowledge text, the question generation prompt words constructed based on the knowledge text and the preset initial number N1 of first-level child nodes are as follows: "You are an expert in the {domain} field. Please generate {N1} <questions> for this paragraph of text based on the following <knowledge text>. The specific requirements are (1) the generated questions must be able to get answers from the given <knowledge text>; (2) different questions are separated by \n". Where domain represents a specific operation and maintenance sub-field.

[0044] Step 202, the large language model is called multiple times to generate answers according to the knowledge text and the question, multiple candidate answers are obtained, and the answer is determined based on the multiple candidate answers to construct the second-level sub-nodes.

[0045] In some embodiments, for each question, an answer can be constructed based on the knowledge text and the question to generate prompt words. By calling the large language model multiple times, multiple candidate answers are obtained, and the degree of match between each candidate answer and the question is determined based on the large language model. The final answer is determined from the multiple candidate answers, and the obtained answer is used as the second-layer sub-node of the first-layer sub-node corresponding to the question.

[0046] It should be noted that each question may have one or more answers, such as single-choice questions or multiple-choice questions. In other words, each first-layer sub-node may have one or more second-layer sub-nodes.

[0047] In some embodiments, multiple candidate answers may be obtained by calling a large language model multiple times, each call generates a candidate answer, and based on the multiple candidate answers, the implementation process of determining the answer may include: determining the frequency of occurrence of each candidate answer; and determining the answer based on the frequency of occurrence of each candidate answer. As an example, the candidate answer with the highest frequency of occurrence may be determined as the answer corresponding to the question, or a frequency threshold may be pre-set. If the highest frequency of occurrence also exceeds the frequency threshold, the candidate answer with the highest frequency of occurrence may be determined as the answer corresponding to the question; if the highest frequency of occurrence does not exceed the frequency threshold, the large language model may be called multiple times to determine the answer corresponding to the question, or the first-level sub-node corresponding to the question may be directly deleted.

[0048] As an example, a template for generating prompt words for an answer based on the knowledge text and the question can be as follows: "You are an expert in the {domain} field. Please answer the given <question> based on the <knowledge text> given below. The specific requirements are: (1) The answer must be strictly based on the <knowledge text>. If the answer cannot be obtained from the <knowledge text>, please output "cannot answer"; (2) The answer must be short and concise, suitable as an option for a multiple-choice question; (3) The answer does not need to include any content that is not relevant to the question." If the candidate answer that appears the most times is "cannot answer", the child node corresponding to the question is deleted.

[0049] Step 203, calling the large language model to generate interference items according to the knowledge text, questions and answers, obtaining multiple candidate interference items, and determining interference items based on the multiple candidate interference items to construct the third-level sub-nodes.

[0050] It can be understood that in order to form test questions, interference items corresponding to generated questions and answers are also needed to serve as other options besides the correct answers in other questions.

[0051] In some embodiments, for each answer, a distractor generation prompt word can be constructed based on the knowledge text, the answer, its corresponding question, and a preset number of distractors m. By calling the large language model multiple times, m candidate distractors are obtained. The m candidate distractors are directly used as m distractors, and each distractor is used as a third-layer child node to obtain m third-layer child nodes of the second-layer child node corresponding to the answer.

[0052] It should be noted that the answer corresponding to each question can be one or more, and the distractors corresponding to each answer can also include one or more. The number of distractors corresponding to each answer can be determined based on actual needs. As an example, each test question can include a question, an answer, and three distractors.

[0053] In some embodiments, for each answer, an interference item generation prompt word can be constructed based on the knowledge text, the answer, its corresponding question, and a preset initial number N3 of third sub-nodes corresponding to each second sub-node. By calling the large language model multiple times, N3 candidate interference items are obtained, and the degree of interference between each candidate interference item and the answer and its corresponding question is determined based on the large language model. Based on the interference degree, M3 interference items are determined from the N3 candidate interference items, and the M3 interference items are used as the M3 third-layer sub-nodes of the second-layer sub-node corresponding to the answer.

[0054] As an example, based on the knowledge text, the answer, the corresponding question, and the preset initial number N3 of third subnodes corresponding to each second subnode, a template for generating distractor prompts is constructed as follows: "You are an expert in the {domain} field. Please generate {N3} distractors based on the following <knowledge text>, <question>, and <answer>. The specific requirements are (1) the generated distractors can be regarded as incorrect answers generated based on the given <knowledge text> and <question>; (2) the format of the distractors is consistent with that of the answer; (3) different distractors are separated by \n". Wherein, domain represents a specific operation and maintenance subfield.

[0055] According to the method for generating operation and maintenance knowledge test questions based on the thinking tree of an embodiment of the present invention, a large language model is called according to the knowledge text to generate questions, multiple candidate questions are obtained, and questions are determined based on the multiple candidate questions to construct the first layer of child nodes; the large language model is called multiple times according to the knowledge text and the question to generate answers, multiple candidate answers are obtained, and answers are determined based on the multiple candidate answers to construct the second layer of child nodes; the large language model is called according to the knowledge text, the question and the answer to generate interference items, multiple candidate interference items are obtained, and interference items are determined based on the multiple candidate interference items to construct the third layer of child nodes. The present invention can not only realize the automatic generation of questions but also improve the generation quality by calling the large language model to construct the thinking tree layer by layer.

[0056] Figure 3 The third flow chart of the method for generating operation and maintenance knowledge test questions based on the mind tree provided in the embodiment of the present invention. Figure 3 As shown, based on the above embodiment, based on multiple candidate questions, the implementation process of determining the question may include the following steps.

[0057] Step 301, for each candidate question, determine the first score of the candidate question according to the knowledge text and the candidate question.

[0058] It can be understood that the qualities of multiple candidate questions generated by the large model are different. In order to ensure the quality of the constructed first-layer sub-nodes, the qualities of multiple candidate questions can be evaluated first, and then one or more questions with better quality can be selected as the first-layer sub-nodes.

[0059] In some embodiments, a large language model may be called according to the knowledge text and the candidate question, so that the large language model outputs a first score of the candidate question. The first score may be a score determined based on the relevance between the knowledge text and the candidate question, a score determined based on the clarity of the question, or a score determined based on whether the knowledge text can correctly answer the given question.

[0060] As a possible implementation method, in order to improve the accuracy of the candidate question quality evaluation, the evaluation dimensions may include: relevance (indicating the semantic relevance between the candidate question and the knowledge text), completeness (indicating whether the knowledge text can correctly and completely answer the candidate question) and clarity (indicating the clarity of the candidate question description). Determining the first score of the candidate question based on the knowledge text and the candidate question includes: constructing a first scoring prompt word based on the knowledge text and the candidate question, and obtaining the first relevance score, completeness score and clarity score of the candidate question by calling the large language model; determining the first score of the candidate question based on the first relevance score, completeness score and clarity score of the candidate question.

[0061] Among them, the implementation process of determining the first score of the candidate question based on the first relevance score, integrity score and clarity score of the candidate question may include: the first relevance score, integrity score and clarity score of the candidate question may be averaged and the calculation result may be determined as the first score of the candidate question; or the first relevance score, integrity score and clarity score of the candidate question may be weightedly summed based on preset weights, and the weighted sum result may be determined as the first score of the candidate question.

[0062] As an example, a template for constructing the first scoring prompt word based on the knowledge text and candidate questions can be as follows: "You are an expert in the {domain} field. Please score the given <question> based on the following <knowledge text>. The specific requirements are: (1) The first relevance score indicates the semantic relevance between the candidate question and the knowledge text, with a score range of 1-10 points; (2) The completeness score indicates whether the candidate question can be correctly answered based entirely on the knowledge text, with a score range of 1-10 points; (3) The clarity score indicates the clarity of the candidate question description, with a score range of 1-10 points; (4) Return in json format." Where domain represents a specific operation and maintenance sub-field.

[0063] Step 302: determine a question from a plurality of candidate questions based on the first score.

[0064] In some embodiments, multiple candidate questions can be arranged in descending order according to the first scores, and based on a preset number M1 of first-level sub-nodes for questions, the first M1 candidate questions in the sorting results can be summarized and determined as multiple questions corresponding to the knowledge text, to obtain M1 first-level sub-nodes.

[0065] According to the method for generating operation and maintenance knowledge test questions based on the mind tree of an embodiment of the present invention, for each candidate question, a first score of the candidate question is determined according to the knowledge text and the candidate question; based on the first score, the question is determined from multiple candidate questions. In other words, in the process of constructing the mind tree, the quality of the candidate questions generated by the large language model is evaluated and scored to accurately control the generation of the mind tree and improve the quality of the generated results.

[0066] Figure 4 The fourth flowchart of the method for generating operation and maintenance knowledge test questions based on the mind tree provided in the embodiment of the present invention. Figure 4 As shown, based on the above embodiment, based on multiple candidate interference items, the implementation process of determining the interference item may include the following steps.

[0067] Step 401, for each candidate distractor, determine a second score of the candidate distractor based on the knowledge text, the question, the answer and the candidate distractor.

[0068] It can be understood that the qualities of multiple candidate interference items generated by the large model are different. In order to ensure the quality of the constructed third-layer sub-nodes, the qualities of multiple candidate interference items can be evaluated first, and then one or more interference items with better quality can be selected as the third-layer sub-nodes.

[0069] In some embodiments, a large language model may be called based on the knowledge text, the question, the answer, and the candidate distractors, so that the large language model outputs a second score for the candidate distractors. The second score may be a score determined based on the correlation between the candidate distractors and the question, a score based on the degree of similarity between the candidate distractors and the answer, or a score determined based on the degree of misleadingness of the candidate distractors to the correct answer to the question.

[0070] As a possible implementation method, in order to improve the accuracy of the quality evaluation of candidate distractors, the evaluation dimensions may include: relevance (the relevance between the candidate distractors and the question), discriminability (indicating the degree of similarity between the candidate distractors and the answer) and distraction (indicating the degree of misleading of the candidate distractors to the correct answer to the question). Based on the knowledge text, question, answer and candidate distractors, determining the second score of the candidate distractors includes: constructing a second scoring prompt word based on the knowledge text, question, answer and candidate distractors, and obtaining the second relevance score, discriminability score and distraction score of the candidate distractors by calling the large language model; determining the second score of the candidate distractors based on the second relevance score, discriminability score and distraction score of the candidate distractors.

[0071] Among them, based on the second relevance score, distinguishability score and interference score of the candidate interference item, the implementation process of determining the second score of the candidate interference item may include: the second relevance score, distinguishability score and interference score of the candidate interference item may be averaged and the calculation result may be determined as the second score of the candidate interference item; or the second relevance score, distinguishability score and interference score of the candidate interference item may be weightedly summed based on preset weights, and the weighted sum result may be determined as the second score of the candidate interference item.

[0072] As an example, a template for constructing the second scoring prompt based on knowledge text, questions, answers, and candidate distractors can be as follows: "You are an expert in the {domain} field. Please score the given <distractor> based on the following <knowledge text><question><correct answer>. The specific requirements are: (1) The second relevance score indicates the degree of relevance between the candidate distractor and the question, with a score range of 1-10 points; (2) The discrimination score indicates the degree of similarity between the candidate distractor and the answer, with a score range of 1-10 points; (3) The distraction score indicates the degree of misleading of the candidate distractor to the correct answer to the question, with a score range of 1-10 points; (4) Return in json format." Among them, domain represents a specific operation and maintenance sub-field.

[0073] Step 402: determine an interference item from a plurality of candidate interference items based on the second score.

[0074] Since the higher the second score, the higher the quality of the candidate interference items, the multiple candidate interference items can be arranged in descending order according to the second scores. Based on the preset number M3 of interference items corresponding to each answer, the first M3 candidate interference items in the sorting result are used as the M3 interference items of the answer. If the second-layer sub-node of the answer corresponding to the candidate interference item is the second-layer sub-node A, then M3 first-layer sub-nodes corresponding to the second-layer sub-node A are obtained.

[0075] According to the method for generating operation and maintenance knowledge test questions based on the mind tree of an embodiment of the present invention, for each candidate interference item, the second score of the candidate interference item is determined based on the knowledge text, the question, the answer and the candidate interference item; based on the second score, the interference item is determined from multiple candidate interference items. In other words, in the process of constructing the mind tree, the quality of the candidate interference items generated by the large language model is evaluated and scored to accurately control the generation of the mind tree and improve the quality of the generated results.

[0076] In order to further improve the quality of operation and maintenance knowledge test questions, the present invention also provides the following embodiments.

[0077] Figure 5 The fifth flow chart of the method for generating operation and maintenance knowledge test questions based on the mind tree provided in the embodiment of the present invention. Figure 5 As shown, based on the above embodiments, Figure 1 The implementation process of step 103 in may include the following steps.

[0078] Step 501, based on the constructed mindset tree, determine a plurality of initial operation and maintenance knowledge test questions.

[0079] Each piece of knowledge text corresponds to a thinking tree. Each thinking tree has three layers of sub-nodes. The operation and maintenance knowledge test questions directly determined based on the thinking tree are the initial operation and maintenance knowledge test questions.

[0080] As an example, Figure 6 As shown, for each thinking tree, if the number of first-layer sub-nodes of the thinking tree is 1, then the thinking tree can determine an initial operation and maintenance knowledge test question, that is, the question corresponding to the first-layer sub-node, the answer to the second-layer sub-node, and the interference item of the third-layer sub-node constitute the first initial operation and maintenance knowledge test question; if the number of first-layer sub-nodes of the thinking tree is N, then the thinking tree can determine N initial operation and maintenance knowledge test questions, for example, for each first-layer sub-node, the question corresponding to the first-layer sub-node, the answer to the second-layer sub-node corresponding to the first-layer sub-node, and the interference item of the third-layer sub-node corresponding to the second-layer sub-node constitute the first initial operation and maintenance knowledge test question, and finally N initial operation and maintenance knowledge test questions are obtained.

[0081] Step 502: for each initial operation and maintenance knowledge test question, determine the quality score of the initial operation and maintenance knowledge test question based on the first score of the question, the second score of the distractor, and the third score of the answer in the initial operation and maintenance knowledge test question.

[0082] In order to improve the quality of the operation and maintenance knowledge test questions, the quality of the initial operation and maintenance knowledge test questions may be evaluated, and based on the evaluation results, a plurality of high-quality operation and maintenance knowledge test questions may be determined.

[0083] Among them, the first score of the question in the initial operation and maintenance knowledge test can be the first score for quality evaluation of the candidate question in the above embodiment, and the second score of the interference item can be the second score for quality evaluation of the candidate interference item in the above embodiment.

[0084] In some embodiments, the third score of the answer to the initial operation and maintenance knowledge test question can be obtained in the following manner: construct a third prompt word based on the knowledge text, question and answer corresponding to the initial operation and maintenance knowledge test question, and obtain the accuracy score, conciseness score and adaptability score of the answer to the initial operation and maintenance knowledge test question by calling the large language model; determine the third score of the answer to the initial operation and maintenance knowledge test question based on the accuracy score, conciseness score and adaptability score of the answer to the initial operation and maintenance knowledge test question.

[0085] The following is an example of a template for constructing the third prompt word based on the knowledge text, question, and answer corresponding to the initial operation and maintenance knowledge test question: "You are an expert in the {domain} field. Please score the given <answer> based on the following <knowledge text> <question>. The specific requirements are: (1) The accuracy score indicates the accuracy of the answer, with a score range of 1-10 points; (2) The conciseness score indicates the conciseness of the answer, with a score range of 1-10 points; (3) The adaptability score indicates the adaptability of the answer as an option in the multiple-choice question, with a score range of 1-10 points; (4) Return in json format." Domain represents a specific operation and maintenance sub-field.

[0086] As an example, the implementation process of determining the third score of the answer in the initial operation and maintenance knowledge test question based on the accuracy score, conciseness score and adaptability score of the answer in the initial operation and maintenance knowledge test question may include: averaging the accuracy score, conciseness score and adaptability score of the answer in the initial operation and maintenance knowledge test question, and using the calculation result as the third score of the answer in the initial operation and maintenance knowledge test question; or based on preset weights, weighted summing the accuracy score, conciseness score and adaptability score of the answer in the initial operation and maintenance knowledge test question, and using the weighted summing result as the third score of the answer in the initial operation and maintenance knowledge test question.

[0087] In some embodiments, based on the first score of the question in the initial operation and maintenance knowledge test question, the second score of the distractor, and the third score of the answer, the process of determining the quality score of the initial operation and maintenance knowledge test question may include: based on a preset weight, weighted summing the first score of the question in the initial operation and maintenance knowledge test question, the second score of the distractor, and the third score of the answer, and determining the weighted summing result as the quality score of the initial operation and maintenance knowledge test question. The above calculation process is shown in formula (1).

[0088] (1); in, is the weight of the question; is the weight of the answer; is the weight of the interference term, ; It is a set of interference items for the initial operation and maintenance knowledge test questions; is the first score of the question; is the second score of the answer; The third score for each distractor.

[0089] Step 503: Determine a plurality of operation and maintenance knowledge test questions from a plurality of initial operation and maintenance knowledge test questions based on the quality scores.

[0090] In some embodiments, a quality score threshold may be preset, and multiple initial operation and maintenance knowledge test questions whose quality scores are greater than or equal to the quality score threshold among the multiple initial operation and maintenance knowledge test questions are determined as multiple operation and maintenance knowledge test questions.

[0091] In some embodiments, multiple initial operation and maintenance knowledge test questions can be arranged in descending order according to the quality score, and based on a preset number S, the first S initial operation and maintenance knowledge test questions in the sorting result are determined as S operation and maintenance knowledge test questions.

[0092] In some embodiments, after obtaining a plurality of operation and maintenance knowledge test questions, a comparison may be performed based on the questions in each operation and maintenance knowledge test question to remove repeated questions with the same questions.

[0093] According to the method for generating operation and maintenance knowledge test questions based on a mind tree according to an embodiment of the present invention, based on the constructed mind tree, multiple initial operation and maintenance knowledge test questions are determined; for each initial operation and maintenance knowledge test question, based on the first score of the question in the initial operation and maintenance knowledge test question, the second score of the interference item and the third score of the answer, the quality score of the initial operation and maintenance knowledge test question is determined; based on the quality score, multiple operation and maintenance knowledge test questions are determined from the multiple initial operation and maintenance knowledge test questions. The present invention screens out high-quality operation and maintenance knowledge test questions by scoring the quality of each question to improve the quality of the obtained questions.

[0094] In order to implement the above embodiment, the embodiment of the present invention also provides a device for generating operation and maintenance knowledge test questions of a thinking tree.

[0095] Figure 7 A schematic diagram of the structure of a device for generating operation and maintenance knowledge test questions based on a mind tree according to an embodiment of the present invention. Figure 7As shown, the device includes a preprocessing module 710, a construction module 720 and an acquisition module 730. The preprocessing module 710 is used to preprocess the operation and maintenance knowledge document to obtain multiple knowledge texts; the construction module 720 is used to construct a thinking tree with the knowledge text as the root node for each knowledge text; the thinking tree includes three layers of sub-nodes; the first layer of sub-nodes are questions related to the knowledge text, the second layer of sub-nodes are answers corresponding to the questions, and the third layer of sub-nodes are interference items corresponding to the questions; the acquisition module 730 is used to obtain multiple operation and maintenance knowledge test questions based on the constructed thinking tree.

[0096] In some embodiments, the construction module 720 is specifically used to: call the large language model to generate questions based on the knowledge text, obtain multiple candidate questions, and determine the questions based on the multiple candidate questions to construct the first layer of sub-nodes; call the large language model multiple times to generate answers based on the knowledge text and the questions, obtain multiple candidate answers, and determine the answers based on the multiple candidate answers to construct the second layer of sub-nodes; call the large language model to generate interference items based on the knowledge text, questions and answers, obtain multiple candidate interference items, and determine the interference items based on the multiple candidate interference items to construct the third layer of sub-nodes.

[0097] In some embodiments, the construction module 720 is further used to: for each candidate question, determine a first score of the candidate question according to the knowledge text and the candidate question; and determine a question from a plurality of candidate questions based on the first score.

[0098] As a possible implementation method, the construction module 720 is also used to: construct a first scoring prompt word based on the knowledge text and the candidate question, and obtain a first relevance score, completeness score and clarity score of the candidate question by calling the large language model; determine the first score of the candidate question based on the first relevance score, completeness score and clarity score of the candidate question.

[0099] In some embodiments, the construction module 720 is further used to: determine the frequency of occurrence of each candidate answer; and determine the answer based on the frequency of occurrence of each candidate answer.

[0100] In some embodiments, the construction module 720 is further used to: for each candidate distractor, determine a second score of the candidate distractor based on the knowledge text, question, answer and candidate distractor; and determine a distractor from multiple candidate distractors based on the second score.

[0101] As a possible implementation method, the construction module 720 is also used to: construct a second scoring prompt word based on the knowledge text, questions, answers and candidate interference items, and obtain the second relevance score, distinguishability score and interference score of the candidate interference items by calling the large language model; determine the second score of the candidate interference item based on the second relevance score, distinguishability score and interference score of the candidate interference item.

[0102] In some embodiments, the acquisition module 730 is also used to: determine multiple initial operation and maintenance knowledge test questions based on the constructed thinking tree; for each initial operation and maintenance knowledge test question, determine the quality score of the initial operation and maintenance knowledge test question based on the first score of the question, the second score of the interference item and the third score of the answer in the initial operation and maintenance knowledge test question; based on the quality score, determine multiple operation and maintenance knowledge test questions from multiple initial operation and maintenance knowledge test questions.

[0103] In some embodiments, the device also includes a determination module 740, which is specifically used to: construct a third prompt word based on the knowledge text, question and answer corresponding to the initial operation and maintenance knowledge test question, and obtain the accuracy score, conciseness score and adaptability score of the answer in the initial operation and maintenance knowledge test question by calling the large language model; determine the third score of the answer in the initial operation and maintenance knowledge test question based on the accuracy score, conciseness score and adaptability score of the answer in the initial operation and maintenance knowledge test question.

[0104] According to the operation and maintenance knowledge test question generation device based on the mind tree of the embodiment of the present invention, the operation and maintenance knowledge document is preprocessed to obtain multiple knowledge texts; for each knowledge text, a mind tree with the knowledge text as the root node is constructed; the mind tree includes three layers of sub-nodes; the first layer of sub-nodes are questions related to the knowledge text, the second layer of sub-nodes are answers corresponding to the questions, and the third layer of sub-nodes are interference items corresponding to the questions; based on the constructed mind tree, multiple operation and maintenance knowledge test questions are obtained. The present invention decomposes the task of generating operation and maintenance knowledge test questions into multiple simple sub-tasks based on the process of constructing the mind tree, so as to realize the process of automatically generating multiple operation and maintenance knowledge test questions based on the operation and maintenance knowledge document, thereby greatly reducing labor costs and improving generation efficiency.

[0105] It should be noted that the description of the method for generating operation and maintenance knowledge test questions based on a thinking tree in the above embodiment is also applicable to the device for generating operation and maintenance knowledge test questions based on a thinking tree in this embodiment, and will not be repeated here.

[0106] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating operation and maintenance knowledge test questions based on a mind tree, characterized in that: include: Pre-process the operation and maintenance knowledge documents to obtain multiple knowledge texts; For each of the knowledge texts, a thinking tree is constructed with the knowledge text as the root node; the thinking tree includes three layers of sub-nodes; the first layer of sub-nodes are questions related to the knowledge text, the second layer of sub-nodes are answers corresponding to the questions, and the third layer of sub-nodes are interference items corresponding to the questions; Based on the constructed thinking tree, multiple operation and maintenance knowledge test questions are obtained.

2. The method according to claim 1, characterized in that: The step of constructing a thinking tree with the knowledge text as a root node includes: Calling a large language model to generate questions according to the knowledge text to obtain multiple candidate questions, and determining the question based on the multiple candidate questions to construct the first-layer sub-node; The large language model is called multiple times to generate answers according to the knowledge text and the question to obtain multiple candidate answers, and the answer is determined based on the multiple candidate answers to construct the second-layer sub-node; The large language model is called to generate interference items according to the knowledge text, the question and the answer, a plurality of candidate interference items are obtained, and the interference items are determined based on the plurality of candidate interference items to construct the third-layer sub-nodes.

3. The method according to claim 2, characterized in that The step of determining the question based on the plurality of candidate questions comprises: For each of the candidate questions, determining a first score of the candidate question according to the knowledge text and the candidate question; The question is determined from among the plurality of candidate questions based on the first score.

4. The method according to claim 3, characterized in that: The step of determining a first score of the candidate question according to the knowledge text and the candidate question includes: Constructing a first scoring prompt word based on the knowledge text and the candidate question, and obtaining a first relevance score, a completeness score, and a clarity score of the candidate question by calling the large language model; A first score for the candidate question is determined based on the first relevance score, the completeness score, and the clarity score of the candidate question.

5. The method according to claim 2, characterized in that: The step of determining the answer based on the plurality of candidate answers comprises: determining the frequency with which each of the candidate answers occurs; The answer is determined based on the frequency of occurrence of each of the candidate answers.

6. The method according to claim 2, characterized in that The determining the interference item based on the multiple candidate interference items includes: For each of the candidate distractors, determining a second score of the candidate distractor based on the knowledge text, the question, the answer, and the candidate distractor; The interferer is determined from among the plurality of candidate interferers based on the second score.

7. The method according to claim 6, characterized in that The step of determining a second score of the candidate distractor based on the knowledge text, the question, the answer, and the candidate distractor comprises: Constructing a second scoring prompt word based on the knowledge text, the question, the answer and the candidate interference item, and obtaining a second relevance score, a discrimination score and an interference score of the candidate interference item by calling the large language model; A second score for the candidate distractor is determined based on the second relevance score, the discriminability score, and the distractor score of the candidate distractor.

8. The method according to any one of claims 1 to 7, characterized in that Based on the constructed mindset tree, multiple operation and maintenance knowledge test questions are obtained, including: Based on the constructed mindset tree, determine a plurality of initial operation and maintenance knowledge test questions; For each of the initial operation and maintenance knowledge test questions, determine the quality score of the initial operation and maintenance knowledge test question based on the first score of the question, the second score of the distractor, and the third score of the answer in the initial operation and maintenance knowledge test question; Based on the quality scores, the plurality of operation and maintenance knowledge test topics are determined from the plurality of initial operation and maintenance knowledge test topics.

9. The method according to claim 8, characterized in that The third score of the answer to the initial operation and maintenance knowledge test question is obtained in the following way: Constructing a third prompt word based on the knowledge text, question and answer corresponding to the initial operation and maintenance knowledge test question, and obtaining the accuracy score, simplicity score and adaptability score of the answer in the initial operation and maintenance knowledge test question by calling the large language model; Based on the accuracy score, simplicity score and adaptability score of the answer in the initial operation and maintenance knowledge test question, a third score of the answer in the initial operation and maintenance knowledge test question is determined.

10. A device for generating operation and maintenance knowledge test questions based on a mind tree, characterized in that: include: The preprocessing module is used to preprocess the operation and maintenance knowledge documents to obtain multiple knowledge texts; A construction module is used to construct a thinking tree with the knowledge text as the root node for each knowledge text; the thinking tree includes three layers of sub-nodes; the first layer of sub-nodes are questions related to the knowledge text, the second layer of sub-nodes are answers corresponding to the questions, and the third layer of sub-nodes are interference items corresponding to the questions; The acquisition module is used to obtain multiple operation and maintenance knowledge test questions based on the constructed thinking tree.

Images