Question generation device, question generation system, and question generation method

Abstract

To provide question generation means that can more accurately evaluate understanding of a learner and the performance of a large-scale language model.SOLUTION: A question generation device includes: an input unit that acquires context data formed of text information; a question generation unit that processes the context data by using a first large-scale language model to generate a set of first multiple-choice questions for the context data; an evaluation unit that evaluates the set of first multiple-choice questions on the basis of a predetermined cognition difficulty evaluation criterion to determine a difficulty evaluation value indicating cognition difficulty for the set of first multiple-choice questions; and a filtering unit that selects a subset of first multiple-choice questions in which the difficulty evaluation value satisfies a predetermined cognition difficulty threshold, from the set of first multiple-choice questions, and outputs the selected subset.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a question generation device, a question generation system, and a question generation method. [Background technology]

[0002] Multiple choice questions (MCQs) are a widely used assessment tool in education, and in recent years have also been used to quantify the performance of large-scale language models (LLMs).

[0003] A multiple-choice question typically consists of a description that describes the problem situation or scenario, a question that poses a problem related to the description, a correct answer that is the correct response to the question, and several incorrect answer options. Automating the creation of multiple-choice questions is expected to significantly reduce human resources, time, cost, and effort.

[0004] As a means for automating the creation of multiple-choice questions, for example, there is research by Doughty et al. (Non-Patent Document 1). Non-Patent Document 1 states, "Educators continually need to develop and maintain effective, up-to-date assessment tools. In computing education, there is growing research on the use of large-scale language models (LLMs) in generating and engaging coding exercises, but the use of LLMs in generating multiple-choice questions (MCQs) about programming has not been widely investigated. We analyzed the ability of GPT-4 to generate MCQs tailored to specific learning objectives (LOs) for Python programming classes in higher education. Specifically, we developed a system using LLMs (GPT-4) to generate MCQs from high-level course context and module-level LOs. We evaluated 651 MCQs generated by LLMs and 449 human-created MCQs tailored to 246 LOs from six Python classes. We confirmed that GPT-4 can generate MCQs with clear language, a single correct answer, and high-quality false answers. We also observed that the generated MCQs appear to be well aligned with the LOs. Our research may be useful for educators who wish to utilize state-of-the-art generative models to support their MCQ creation efforts." [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Jacob Doughty, Zipiao Wan, Anishka Bompelli, Jubahed Qayum, Taozhi Wang, Juran Zhang, Yujia Zheng, Aidan Doyle, Pragnya Sridhar, Arav Agarwal, Christopher Bogart, Eric Keylor, Can Kultur, Jaromir Savelka, and Majd Sakr. 2024. A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education. In Australian Computing Education Conference (ACE 2024), January 29--February 02, 2024, Sydney, NSW, Australia. ACM, New York, NY, USA 10 Pages. https: / / doi.org / 10.1145 / 3636243.3636256 Summary of the Invention [Problem to be solved by the invention]

[0006] In general, to accurately assess learners' understanding, it is desirable to create not only low-difficulty questions that simply require students to recall memorized knowledge, but also high-difficulty multiple-choice questions that require students to apply knowledge or analyze a certain concept, thereby requiring deep understanding.

[0007] Non-Patent Document 1 describes a method for automatically generating multiple-choice questions related to programming using an LLM such as GPT-4. However, in the method described in Non-Patent Document 1, the multiple-choice questions are generated in a single step, and no consideration is given to controlling the cognitive difficulty of the multiple-choice questions according to the user's needs. As a result, the multiple-choice questions generated by the technology in Non-Patent Document 1 are low-difficulty questions, similar to the so-called Cloze task, and it may be difficult to accurately evaluate the learner's understanding. Additionally, previous proposals to assess the quality of multiple-choice questions based on, for example, vocabulary difficulty or the number of options, limit their ability to accurately assess the cognitive skills required to answer the question.

[0008] Therefore, the present disclosure aims to provide a question generation means that can more accurately evaluate learner understanding and the performance of large-scale language models by generating multiple-choice questions with cognitive difficulty levels that match the user's needs. [Means for solving the problem]

[0009] In order to solve the above problem, a representative question generation device of the present invention comprises a processor and a memory, and the memory contains processing instructions for causing the processor to function as an input unit that acquires context data consisting of text information, a question generation unit that processes the context data using a first large-scale language model to generate a first set of multiple-choice questions for the context data, an evaluation unit that evaluates the first set of multiple-choice questions based on predetermined cognitive difficulty evaluation criteria to determine a difficulty evaluation value indicating a cognitive difficulty for the first set of multiple-choice questions, and a filtering unit that selects and outputs a subset of first multiple-choice questions from the first set of multiple-choice questions, whose difficulty evaluation value satisfies a predetermined cognitive difficulty threshold. [Effects of the Invention]

[0010] According to the present disclosure, it is possible to provide a question generation means that can more accurately evaluate learner understanding and the performance of large-scale language models by generating multiple-choice questions with cognitive difficulty levels that match the user's needs. Problems, configurations, and effects other than those described above will become apparent from the following description of the preferred embodiment of the invention. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating a computer system for implementing an embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of a question generation system according to an embodiment of the present disclosure. [Figure 3] FIG. 3 is a diagram illustrating a data flow in the question generation device according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating the processing of the summary generation unit according to an embodiment of the present disclosure. [Figure 5] FIG. 5 is a diagram illustrating the processing of the question generation unit according to an embodiment of the present disclosure. [Figure 6] FIG. 6 is a diagram illustrating the processing of the evaluation unit according to the embodiment of the present disclosure. [Figure 7] FIG. 7 is a diagram illustrating processing by the filtering unit according to an embodiment of the present disclosure. [Figure 8] FIG. 8 is a diagram illustrating a specific example of processing by the question generation device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are designated by the same reference numerals. Furthermore, although terms such as "first," "second," and "third" may be used to describe various elements or components in this disclosure, it will be understood that these elements or components should not be limited by these terms. These terms are used only to distinguish one element or component from another. Thus, a first element or component discussed below could also be referred to as a second element or component without departing from the teachings of the inventive concept.

[0013] Next, referring to FIG. 1, a computer system 100 for implementing embodiments of the present disclosure will be described. The mechanisms and devices of various embodiments disclosed herein may be applied to any suitable computing system. The main components of the computer system 100 include one or more processors 102, memory 104, a terminal interface 112, a storage interface 113, an I / O (input / output) device interface 114, and a network interface 115. These components may be interconnected via a memory bus 106, an I / O bus 108, a bus interface unit 109, and an I / O bus interface unit 110.

[0014] Computer system 100 may include one or more general-purpose programmable central processing units (CPUs) 102A and 102B, collectively referred to as processors 102. In some embodiments, computer system 100 may include multiple processors, while in other embodiments, computer system 100 may be a single CPU system. Each processor 102 executes instructions stored in memory 104 and may include an on-board cache. In some embodiments, computer system 100 may include a graphics processing unit (GPU) in addition to processors 102. The use of a GPU can accelerate the processing of machine learning models, etc., used in question generation application 150, described below.

[0015] In one embodiment, memory 104 may include random-access semiconductor memory, storage devices, or storage media (either volatile or non-volatile) for storing data and programs. Memory 104 may store all or part of the programs, modules, and data structures that implement the functions described herein. For example, memory 104 may store a question generation application 150. In one embodiment, question generation application 150 may include instructions or descriptions that execute the functions described below on processor 102.

[0016] In some embodiments, question generation application 150 may be implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and / or other physical hardware devices instead of or in addition to a processor-based system. In some embodiments, question generation application 150 may include data other than instructions or descriptions. In some embodiments, cameras, sensors, or other data input devices (not shown) may be provided to communicate directly with bus interface unit 109, processor 102, or other hardware of computer system 100.

[0017] Computer system 100 may include a bus interface unit 109 that facilitates communication between processor 102, memory 104, display system 124, and I / O bus interface unit 110. I / O bus interface unit 110 may couple to an I / O bus 108 for transferring data to and from various I / O units. I / O bus interface unit 110 may communicate via I / O bus 108 with multiple I / O interface units 112, 113, 114, and 115, also known as I / O processors (IOPs) or I / O adapters (IOAs).

[0018] Display system 124 may include a display controller, a display memory, or both. The display controller may provide video, audio, or both data to display device 126. Computer system 100 may also include one or more sensors or other devices configured to collect data and provide the data to processor 102.

[0019] For example, computer system 100 may include biometric sensors that collect heart rate data, stress level data, etc., environmental sensors that collect humidity data, temperature data, pressure data, etc., and motion sensors that collect acceleration data, movement data, etc. Other types of sensors may also be used. Display system 124 may be connected to a display device 126, such as a standalone display screen, a television, a tablet, or a handheld device.

[0020] The I / O interface unit provides functionality for communicating with various storage or I / O devices. For example, the terminal interface unit 112 may be attached to user I / O devices 116, such as user output devices such as a video display, a television with speakers, and user input devices such as a keyboard, a mouse, a keypad, a touchpad, a trackball, buttons, a light pen, or other pointing device. A user may use a user interface to enter input data or instructions into the user I / O devices 116 and the computer system 100, and receive output data from the computer system 100, by operating the user input devices. The user interface may be displayed on a display, played through speakers, or printed via a printer via the user I / O devices 116, for example.

[0021] Storage interface 113 allows attachment of one or more disk drives or direct access storage device 117 (typically a magnetic disk drive storage device, but may also be an array of disk drives or other storage devices configured to appear as a single disk drive). In some embodiments, storage device 117 may be implemented as any secondary storage device. The contents of memory 104 may be stored in storage device 117 and retrieved as needed from storage device 117. I / O device interface 114 may provide an interface to other I / O devices, such as printers, fax machines, etc. Network interface 115 may provide a communications path that allows computer system 100 and other devices to communicate with each other. This communications path may be, for example, network 130.

[0022] In some embodiments, computer system 100 may be a device that receives requests from other computer systems (clients) without a direct user interface, such as a multi-user mainframe computer system, a single-user system, or a server computer. In other embodiments, computer system 100 may be a desktop computer, a portable computer, a laptop, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable electronic device.

[0023] 2 is a diagram illustrating an example of a configuration of a question generation system 200 according to an embodiment of the present disclosure. The question generation system 200 is a system for generating and outputting multiple-choice questions with a cognitive difficulty level tailored to a user's needs. As shown in FIG. 2, the question generation system 200 mainly includes a question generation device 210, a communication network 250, and a user terminal 260. The question generation device 210 and the user terminal 260 may be connected to each other via the communication network 250.

[0024] The question generation device 210 is a device for generating and outputting multiple-choice questions with cognitive difficulty levels tailored to the user's needs, and as shown in FIG. 2, mainly includes a memory 220, a storage unit 230, a processor 244, and an input / output unit 246. In one embodiment, the question generator 210 may be implemented by the computer system 100 shown in FIG.

[0025] The memory 220 may be a memory for storing a question generation application 150 for implementing the functionality of a question generator according to an embodiment of the present disclosure. The question generation application 150 may include processing instructions for implementing the functionality of software modules such as an input unit 222, a summary generator 224, a question generator 226, an evaluator 228, and a filter 229, as shown in FIG.

[0026] The input unit 222 is a functional unit for inputting various information used by the question generation device 210. In one embodiment, the input unit 222 may input context data consisting of text information and information on difficulty level distribution conditions from the user terminal 260 or the like. The context data here is text that serves as the basis for the multiple-choice questions to be generated, and may be text extracted from, for example, academic papers, books, articles, magazines, etc., and is not particularly limited as long as it is text information. The difficulty level distribution conditions are information that specifies the desired ratio of low difficulty questions to high difficulty questions in the set of multiple-choice questions to be generated. The input unit 222 may store the input context data and difficulty distribution conditions in the context DB 236 in the storage unit 230 . The details of the functions of the input unit 222 will be described later, and therefore will not be described here.

[0027] The summary generation unit 224 is a functional unit for generating summary information indicating key points extracted from the context data by processing the context data input by the input unit 222 using a large-scale language model. As will be described later, using the summary information for the context data can facilitate the generation of high-difficulty questions. The details of the functions of the summary generation unit 224 will be described later, and therefore will not be described here.

[0028] The question generation unit 226 is a functional unit for generating a set of multiple-choice questions for the context data (for example, a first set of multiple-choice questions and a second set of multiple-choice questions) by processing, with a large-scale language model, the context data input by the input unit 222 or the summary information generated by the summary generation unit 224. The question generation unit 226 may store the generated set of multiple-choice questions in a question DB 238 included in the storage unit 230. The details of the function of the question generator 226 will be described later, and therefore will not be described here.

[0029] The evaluation unit 228 is a functional unit for determining a difficulty evaluation value indicating the cognitive difficulty of the set of multiple-choice questions generated by the question generation unit 226 by evaluating the set of multiple-choice questions based on a predetermined cognitive difficulty evaluation criterion. The difficulty evaluation value here is information that quantitatively indicates the degree of difficulty of identifying the correct answer to each multiple-choice question. In one embodiment, the cognitive difficulty evaluation criterion used to evaluate the difficulty evaluation value may be, for example, a criterion based on the so-called Bloom's Taxonomy. In this case, the difficulty evaluation value may be expressed as a numerical value within a range of 0 to 6, for example. In one embodiment, the evaluation unit 228 inputs a subset of multiple-choice questions selected by the filtering unit 229 (described later) into the large-scale language model, and determines a performance score that quantitatively indicates the performance of the large-scale language model based on the accuracy rate of the answers of the large-scale language model to the subset of multiple-choice questions. This makes it possible to evaluate the performance of the large-scale language model. The details of the function of the evaluation unit 228 will be described later, and therefore will not be described here.

[0030] The filtering unit 229 is a functional unit for selecting and outputting a subset of first multiple-choice questions whose difficulty assessment values ​​satisfy a predetermined cognitive difficulty threshold from the set of multiple-choice questions generated by the question generation unit 226. The cognitive difficulty threshold here may be a value that specifies a desired difficulty level, and may be freely set by the user of the user terminal 260, for example. In one embodiment, the filtering unit 229 may generate a subset of multiple-choice questions by excluding multiple-choice questions that do not satisfy a cognitive difficulty threshold from the set of multiple-choice questions. If the ratio of low difficulty questions to high difficulty questions in this subset of multiple-choice questions satisfies the difficulty distribution condition input by the input unit 222, the filtering unit 229 may output the subset of multiple-choice questions to the user terminal 260. On the other hand, if the ratio of low difficulty questions to high difficulty questions in the subset of multiple-choice questions does not satisfy the difficulty distribution condition input by the input unit 222, the question generator 226 may generate additional multiple-choice questions. The details of the function of the filtering unit 229 will be described later, and therefore will not be described here.

[0031] The storage unit 230 is a storage area that accommodates a database (hereinafter, "DB") for storing various information according to an embodiment of the present disclosure, and may include a context DB 236 and a question DB 238 as shown in FIG.

[0032] The context DB 236 is a database for storing input data (context data and difficulty distribution conditions) used in the present disclosure. The question DB 238 is a database for storing multiple-choice questions generated by the question generator 226 .

[0033] The processor 244 is a processing unit for executing the processing instructions stored by the memory 220 that define the functionality of each functional unit of the question generation application 150 .

[0034] The input / output unit 246 is a functional unit for accepting information (e.g., context data and difficulty distribution conditions) input to the question generation device 210 and outputting information (such as multiple-choice questions) generated by the question generation device 210. In one embodiment, the input / output unit 246 may include, for example, a keyboard, a mouse, a display that displays a GUI (Graphical User Interface), and the like. In one embodiment, the input / output unit 246 may provide the user terminal 260 with a GUI that inputs and outputs various types of information.

[0035] The communications network 250 may include, for example, a local area network (LAN), a wide area network (WAN), a satellite network, a cable network, a WiFi network, or any combination thereof.

[0036] The user terminal 260 is a terminal device that can be used by a user of the question generation device 210. By using the user terminal 260, the user can input context data and information on difficulty distribution conditions to the question generation device 210 and check multiple-choice questions output from the question generation device 210. As an example, the user terminal 260 may include, but is not limited to, a smartphone, smartwatch, tablet, or personal computer of a user who subscribes to a question generation service provided by the question generation system 200. For ease of explanation, FIG. 2 illustrates an example of a configuration including one user terminal 260, but the number of user terminals 260 is not limited, and a configuration including multiple user terminals 260 is also possible.

[0037] According to the question generation system 200 of the present disclosure described above, multiple-choice questions with cognitive difficulty levels tailored to the user's needs can be generated, making it possible to more accurately evaluate the learner's understanding and the performance of large-scale language models.

[0038] Next, with reference to FIG. 3, a data flow in the question generation device 210 according to an embodiment of the present disclosure will be described.

[0039] FIG. 3 is a diagram showing the flow of data in the question generation device 210 according to an embodiment of the present disclosure.

[0040] First, the input unit 222 acquires context data 302 from the user terminal 260 (not shown in FIG. 3 ). As described above, this context data 302 is text that will be the basis for the multiple-choice questions to be generated. For example, the context data 302 may be text extracted from academic papers, books, articles, magazines, etc., and is not particularly limited as long as it is text information. The input unit 222 may also acquire information on difficulty distribution conditions 304 from the user terminal 260. The difficulty distribution conditions 304 are information that specifies the desired ratio of low difficulty questions to high difficulty questions in the set of multiple-choice questions to be generated. For example, in one embodiment, the difficulty distribution conditions 304 may be a ratio of low difficulty questions to high difficulty questions of "7:3." The input unit 222 transfers the acquired context data 302 to the summary generation unit 224 and / or the question generation unit 226, and transfers the acquired difficulty level distribution conditions 304 to the filtering unit 229. The input unit 222 may also store the context data 302 and the difficulty level distribution conditions 304 in the context DB 236 shown in FIG. 2.

[0041] Next, the summary generation unit 224 processes the context data 302 received from the input unit 222 using a large-scale language model to generate summary information 306 indicating key points extracted from the context data 302, and transfers the generated summary information 306 to the question generation unit 226. If the summary generation unit 224 determines that the context data 302 satisfies a predetermined length criterion, it may divide the context data 302 into multiple context data portions having a predetermined length, and process each of the context data portions using a large-scale language model to generate partial summary information indicating the extracted main points for each of the context data portions.

[0042] The length criteria here are criteria used to identify long context data 302, and may be set, for example, to the number of words (10,000 words or more), the number of characters (30,000 characters or more), the number of pages (20 pages or more), etc. In this way, by dividing long context data 302 into multiple sections and generating summary information for each section individually, it becomes possible to generate questions that require understanding of multiple parts of the context data 302.

[0043] Next, the question generator 226 processes the context data 302 received from the input unit 222 and the summary information 306 received from the summary generator 224 using a large-scale language model to generate a set of multiple-choice questions 308 for the context data and summary information 306. More specifically, the question generation unit 226 may generate low-difficulty questions for the context data 302 by processing the context data 302 with a large-scale language model, and may generate high-difficulty questions for the context data 302 by processing the summary information 306 with a large-scale language model, and may configure the generated low-difficulty questions and high-difficulty questions as a set 308 of multiple-choice questions. The question generator 226 transfers the generated set of multiple-choice questions 308 to the evaluation unit 228 and the filtering unit 229. The question generator 226 may also store the generated set of multiple-choice questions 308 in a question DB 238 shown in FIG.

[0044] According to research by the inventors of the present disclosure, direct processing of the context data 302 with a large-scale language model has been found to generate low-level questions, such as Cloze tasks, that can be answered using information extracted directly from the context data 302. However, the quality of high-level questions, which require a higher cognitive level, may be insufficient. On the other hand, the inventors of the present disclosure have discovered that using a large-scale language model to generate summary information 306 indicating key points extracted from the context data 302 and then using the large-scale language model to generate multiple-choice questions for the summary information 306, can result in higher-level questions of higher quality that require a comprehensive understanding of the context data 302. Therefore, one aspect of the present disclosure relates to generating low-level questions by directly processing the context data 302 with a large-scale language model, and generating high-level questions based on the summary information 306 indicating key points extracted from the context data 302. This allows the user to obtain a set of multiple-choice questions with a desired difficulty distribution.

[0045] Next, the evaluation unit 228 determines a difficulty evaluation value 310 indicating a cognitive difficulty for the set of multiple-choice questions by evaluating the set of multiple-choice questions received from the question generation unit 226 based on a predetermined cognitive difficulty evaluation criterion. Here, the evaluation unit 228 may determine a difficulty evaluation value 310 for each of the multiple-choice questions included in the set of multiple-choice questions. As described above, the difficulty rating 310 here is information that quantitatively indicates the degree of difficulty of identifying the correct answer to a multiple-choice question. In one embodiment, the cognitive difficulty rating criterion used to evaluate the difficulty rating may be, for example, a criterion based on the so-called Bloom's Taxonomy. In this case, the difficulty rating may be expressed as a number ranging from 0 to 6, for example.

[0046] Next, the filtering unit 229 selects and outputs a subset 312 of multiple-choice questions whose difficulty assessment values ​​310 received from the evaluation unit 228 satisfy a predetermined cognitive difficulty threshold from the set 308 of multiple-choice questions received from the question generation unit 226. As described above, the cognitive difficulty threshold here may be a value that specifies a desired difficulty level, and may be freely set by the user of the user terminal 260, for example. In one embodiment, the filtering unit 229 may generate a subset 312 of multiple-choice questions by excluding multiple-choice questions that do not satisfy a cognitive difficulty threshold from the set 308 of multiple-choice questions. If the ratio of low difficulty questions to high difficulty questions in this subset 312 of multiple-choice questions satisfies the difficulty distribution condition input by the input unit 222, the filtering unit 229 may output the subset 312 of multiple-choice questions to the user terminal 260. On the other hand, if the ratio of low difficulty questions to high difficulty questions in the subset of multiple-choice questions does not satisfy the difficulty distribution condition input by the input unit 222, the question generator 226 may generate additional multiple-choice questions.

[0047] In one embodiment, after the multiple-choice question subset 312 is generated by the filtering unit 229, the evaluation unit 228 may present this multiple-choice question subset 312 to a human learner or a large-scale language model, and determine a score that quantitatively indicates the performance of the human learner or the large-scale language model based on the correct answer rate for this multiple-choice question subset 312. This score may be the correct answer rate for the multiple-choice question subset 312 itself, or may be a value obtained by performing a predetermined calculation on the correct answer rate. This makes it possible to evaluate the understanding of the human learner and the large-scale language model.

[0048] According to the question generation device 210 of the present disclosure described above, multiple-choice questions with cognitive difficulty levels tailored to the user's needs can be generated, making it possible to more accurately evaluate the learner's understanding and the performance of a large-scale language model.

[0049] Next, the summary generation unit according to an embodiment of the present disclosure will be described with reference to FIG.

[0050] 4 is a diagram illustrating processing by the summary generation unit 224 according to an embodiment of the present disclosure. As described above, the summary generation unit 224 according to an embodiment of the present disclosure is a functional unit for generating summary information 306 indicating key points extracted from the context data 302 by processing the context data 302 using a large-scale language model. FIG. 4 illustrates an example of processing by the summary generation unit 224.

[0051] 4, a prompt 410 requesting the generation of summary information 306 for the above-described context data 302 is input to the summary generation unit 224. For example, as shown in FIG. 4, this prompt 410 may include a sentence of the context data 302 and a sentence such as "Please extract the main points of the following sentence" that prompts the generation of summary information 306 for the sentence.

[0052] 4 is input to the summary generation unit 224, the summary generation unit 224 processes the received prompt 410 using a predetermined large-scale language model to generate summary information 306. Here, the summary generation unit 224 may use, for example, GPT-4 as the large-scale language model, and is not particularly limited as long as it is a large-scale language model that can extract the main points of a sentence.

[0053] As an example, when the following context data 302 is input to the summary generation unit 224: "Cold fusion is a phenomenon in which a nuclear fusion reaction of hydrogen atoms occurs at low temperatures, from room temperature to approximately 1,000 degrees Celsius. There are several hypotheses about the mechanism by which a fusion reaction occurs, such as the tunneling effect or muons contained in cosmic rays. This article deals with a nuclear fusion reaction that is claimed to have occurred at low temperatures and was visible to the naked eye, and on a scale that could be used as a practical energy source. After the sensational announcement about cold fusion in 1989, it was called "the greatest scientific scandal of the 20th century" due to its low reproducibility, but in recent years, in response to the need for decarbonization, it is expected to be used in industry.", the summary generation unit 224 may generate the following summary information 306: "Cold fusion is a phenomenon in which a nuclear fusion reaction of hydrogen atoms occurs at low temperatures, and there are several hypotheses about the mechanism by which a fusion reaction occurs, such as the tunneling effect or muons contained in cosmic rays. This article deals with a nuclear fusion reaction that is claimed to have occurred at low temperatures and was visible to the naked eye, and on a scale that could be used as a practical energy source. After the sensational announcement about cold fusion in 1989, it was called "the greatest scientific scandal of the 20th century" due to its low reproducibility, but in recent years, it is expected to be used in industry due to the need for decarbonization." As described above, the generation of high-level questions can be facilitated by using summary information 306 that extracts key points from context data 302.

[0054] In some cases, the context data 302 may contain long sentences. For example, if the context data 302 is a book, academic paper, or the like, it may be hundreds of pages long. Therefore, one aspect of the question generation means according to an embodiment of the present disclosure relates to dividing the long context data 302 into sections of a fixed size and generating summary information for each section. More specifically, if the summary generation unit 224 determines that the context data 302 satisfies a predetermined length criterion, it divides the context data 302 into multiple context data portions of a predetermined length and processes each of the context data portions using a large-scale language model to generate partial summary information indicating extracted key points for each context data portion.

[0055] The length criteria here are criteria used to identify context data 302 containing long sentences that are desirable to divide, and may be set, for example, to the number of words (10,000 words or more), the number of characters (30,000 characters or more), the number of pages (20 pages or more), etc. In one embodiment, the summary generator 224 may divide the context data 302 into context data portions of a predetermined size, such as 1,000 words, 10,000 characters, or 2 pages, and then individually generate partial summary information that is summary information for each context data portion. In this way, by dividing long sentences of context data 302 into fixed-size context data portions and generating summary information for each context data portion individually, it becomes possible to generate high-difficulty questions that require understanding of many key points contained in the context data 302, even if the sentences in the context data 302 are long.

[0056] Next, the question generation unit according to the embodiment of the present disclosure will be described with reference to FIG.

[0057] 5 is a diagram illustrating processing by the question generation unit 226 according to an embodiment of the present disclosure. As described above, the question generation unit 226 according to an embodiment of the present disclosure is a functional unit for generating a set of multiple-choice questions by processing the context data 302 or the summary information 306 generated from the context data 302 using a large-scale language model. FIG. 5 illustrates an example of processing by the question generation unit 226.

[0058] 5, the question generator 226 receives a prompt 510 requesting the generation of a multiple-choice question for the context data 302 or summary information 306. The prompt 510 may include, for example, a sentence from the context data 302 or summary information 306 and a sentence prompting the generation of a multiple-choice question for the sentence, as shown in FIG. 5. In one embodiment, the prompt 510 may include an example that specifies the desired structure of the multiple-choice question. This allows the structure of the multiple-choice question to be specified by the question generator 226.

[0059] 5 is input to the question generation unit 226, the question generation unit 226 processes the received prompt 510 with a predetermined large-scale language model to generate a set of multiple-choice questions 308. Here, the question generation unit 226 may use, for example, GPT-4 as the large-scale language model, and is not particularly limited as long as it is a large-scale language model capable of generating multiple-choice questions. It should be noted that the large-scale language model used to generate the set of multiple-choice questions 308 may be the same as or a different large-scale language model used by the summary generator 224 to generate the summary information 306.

[0060] The set of multiple-choice questions 308 generated by the question generator 226 may be structured as defined in prompts 510, for example, as shown in FIG. For ease of explanation, FIG. 5 shows the set of multiple-choice questions 308 including one multiple-choice question, but the present disclosure is not limited thereto, and the question generator 226 may generate the set of multiple-choice questions 308 including any number of multiple-choice questions.

[0061] Furthermore, as described above, if the context data 302 satisfies the length criteria and is divided into multiple context data portions by the summary generator 224, and partial summary information is generated for each of the context data portions, the question generator 226 may generate the set of multiple-choice questions 308 by processing each piece of partial summary information using a large-scale language model. This makes it possible to generate high-level questions that individually require understanding of multiple key points contained in the context data 302 when the sentences in the context data 302 are long.

[0062] Furthermore, in some embodiments, if the context data 302 satisfies a length criterion and is divided into multiple context data portions by the summary generator 224, and partial summary information is generated for each of the context data portions, the question generator 226 may identify multiple context data portions (first and second context data portions) that satisfy a predetermined relevance criterion, and aggregate the partial summary information (first and second partial summary information) generated for each of the identified context data portions to generate aggregated partial summary information that combines the respective partial summary information. Thereafter, the question generator 226 may generate a set of multiple-choice questions 308 by processing the aggregated partial summary information thus generated using a large-scale language model.

[0063] The relevance criteria are used to identify multiple context data portions that have similar semantic content. In some embodiments, the multiple context data portions that satisfy the relevance criteria may be identified using natural language processing techniques such as Term Frequency-Inverse Document Frequency (TF-IDF), cosine similarity, paragraph vectors, Jaccard similarity, latent semantic analysis, Bidirectional Encoder Representations from Transformers (BERT), FastText, Semantic Textual Similarity, Siamese Networks, etc.

[0064] In this way, by generating a set of multiple-choice questions 308 from aggregated partial summary information that aggregates partial summary information from multiple context data parts with similar semantic content, it is possible to generate high-difficulty questions that require a comprehensive understanding of multiple related topics in the context data 302.

[0065] In addition, the question generation unit 226 may generate low-difficulty questions obtained by processing the context data 302 with a large-scale language model and high-difficulty questions obtained by processing summary information 306, which extracts key points from the context data 302, with a large-scale language model, and may use the generated low-difficulty questions and high-difficulty questions as a set 308 of multiple-choice questions. This allows the user to obtain a set of multiple-choice questions with a desired difficulty distribution.

[0066] Next, the evaluation unit according to the embodiment of the present disclosure will be described with reference to FIG.

[0067] Fig. 6 is a diagram showing processing by the evaluation unit 228 according to an embodiment of the present disclosure. As described above, the evaluation unit 228 according to an embodiment of the present disclosure is a functional unit for determining a difficulty evaluation value indicating the cognitive difficulty of the set of multiple-choice questions by evaluating the set of multiple-choice questions 308 generated by the question generation unit 226 described with reference to Fig. 5 based on a predetermined cognitive difficulty evaluation criterion. Fig. 6 shows an example of processing by the evaluation unit 228.

[0068] 6, the evaluator 228 receives a prompt 610 requesting an evaluation of the set of multiple-choice questions 308 generated by the question generator 226. The prompt 610 may include, for example, as shown in FIG. 6, the set of multiple-choice questions 308 and a sentence defining the cognitive difficulty evaluation criteria used in evaluating the set of multiple-choice questions 308. The cognitive difficulty assessment criterion here is a criterion used to assess the cognitive difficulty of the set of multiple-choice questions 308, and may be an assessment criterion based on the so-called Bloom's Taxonomy. More specifically, the cognitive difficulty assessment criterion is: For questions where the correct answer to the question is incorrect, a difficulty rating of "0" is given. For questions that require you to memorize and recall certain knowledge, assign a difficulty rating of "1"; Questions that require interpretation of memorized knowledge are given a difficulty rating of "2," For questions that require applying memorized knowledge to a given problem, a difficulty rating of "3" is given. A question that requires a complex problem to be broken down into several elements and understood is given a difficulty rating of "4", A difficulty rating of "5" is given to questions that require critical evaluation and judgment of information or ideas. It may be specified that a difficulty rating of "6" be given to questions that require combining multiple elements to create a new overall picture.

[0069] 6 is input to the evaluation unit 228, the evaluation unit 228 processes the received prompt 610 using a predetermined large-scale language model to evaluate the set of multiple-choice questions 308 based on the defined cognitive difficulty evaluation criteria, and generates a difficulty evaluation value 310 that indicates the cognitive difficulty of the set of multiple-choice questions 308. As described above, the difficulty evaluation value 310 is information that expresses the cognitive difficulty of each question included in the set of multiple-choice questions 308 as a numerical value between "0 to 6" defined in the cognitive difficulty evaluation criteria.

[0070] In this way, assessing the cognitive difficulty of the set of multiple-choice questions 308 using the cognitive difficulty assessment criteria described above provides a more quantitative understanding of the cognitive ability required to solve the questions than traditional assessment criteria such as vocabulary difficulty or number of options. Furthermore, as described below, filtering the set of multiple-choice questions 308 to which cognitive difficulty has been assigned in this way can identify a subset of multiple-choice questions that can more accurately assess learner understanding and the performance of large-scale language models.

[0071] In one embodiment, after the multiple-choice question subset 312 is generated by the filtering unit 229, the evaluation unit 228 may present this multiple-choice question subset 312 to a human learner or a large-scale language model, and determine a score that quantitatively indicates the performance of the human learner or the large-scale language model based on the correct answer rate for this multiple-choice question subset 312. This score may be the correct answer rate for the multiple-choice question subset 312 itself, or may be a value obtained by performing a predetermined calculation on the correct answer rate. This makes it possible to evaluate the understanding of the human learner and the large-scale language model.

[0072] Next, a filtering unit according to an embodiment of the present disclosure will be described with reference to FIG.

[0073] Fig. 7 is a diagram showing processing by the filtering unit 229 according to an embodiment of the present disclosure. As described above, the filtering unit 229 according to an embodiment of the present disclosure is a functional unit for selecting and outputting a subset of multiple-choice questions whose difficulty assessment values ​​satisfy a predetermined cognitive difficulty threshold from the set 308 of multiple-choice questions generated by the question generation unit 226 described with reference to Fig. 5. Fig. 7 shows an example of processing by the filtering unit 229.

[0074] First, as shown in FIG. 7, in step S702, the filtering unit 229 determines whether the difficulty assessment value (Score_1, Score_2, ... Score_n) assigned to each multiple-choice question (MCQ_1, MCQ_2, ... MCQ_n) included in the set of multiple-choice questions 302 satisfies a first cognitive difficulty threshold. The first cognitive difficulty threshold here is a threshold for identifying low-quality multiple-choice questions, and may be set to, for example, "a difficulty assessment value of 1 or more." This makes it possible to exclude low-quality questions, in which the correct answer to the question is incorrect, from the set of multiple-choice questions 302. The filtering unit 229 generates a subset of multiple-choice questions from the set of multiple-choice questions 302 by eliminating multiple-choice questions that do not meet the first cognitive difficulty threshold.

[0075] Next, in step S704, the filtering unit 229 classifies, from the subset of multiple-choice questions generated in step S702, multiple-choice questions that satisfy the second cognitive difficulty threshold as high-difficulty questions and multiple-choice questions that do not satisfy the second cognitive difficulty threshold as low-difficulty questions, thereby generating a classified subset of multiple-choice questions 312. The second cognitive difficulty threshold here is a threshold for distinguishing low-difficulty questions that are relatively easy to answer from high-difficulty questions that are more difficult to answer, and may be set to, for example, a "difficulty evaluation value: 4 or more." Note that this second cognitive difficulty threshold may be freely set by, for example, the user of the question generation device 210. This makes it possible to provide multiple-choice questions with cognitive difficulty levels tailored to the user's needs.

[0076] Next, in step S706, the filtering unit 229 determines whether or not the subset 312 of classified multiple-choice questions in the subset of multiple-choice questions satisfies the difficulty distribution condition 304, based on the classified questions generated in step S704 and the difficulty distribution condition 304 input by the user to the input unit 222. As described above, the difficulty distribution condition 304 here is information that specifies the desired ratio of low difficulty questions to high difficulty questions, and may be, for example, a ratio of "7:3" between low difficulty questions and high difficulty questions. As an example, if the difficulty distribution condition 304 is "7:3" and the classified subset 312 of multiple-choice questions includes eight low-difficulty questions and two high-difficulty questions, the filtering unit 229 determines that the difficulty distribution condition 304 is not satisfied because there are not enough high-difficulty questions. On the other hand, if the difficulty distribution condition 304 is "7:3" and the classified subset 312 of multiple-choice questions includes seven low-difficulty questions and three high-difficulty questions, the filtering unit 229 determines that the difficulty distribution condition 304 is satisfied. If the classified subset of multiple-choice questions 312 satisfies the difficulty distribution condition 304, the subset of multiple-choice questions is output to the user and the process ends. On the other hand, if the subset of multiple-choice questions 312 does not satisfy the difficulty distribution condition 304, the process proceeds to step S708.

[0077] In step S708, the filtering unit 229 checks whether the counter variable n, which indicates the number of times the processes in steps S702 to S706 have been performed on the subset of multiple-choice questions, is greater than or equal to the threshold value n max Here, it is determined whether the number of times is less than or equal to the threshold value n max is a value that specifies the number of times the question generation process may be repeated to satisfy the difficulty distribution condition 304, and may be set by the user. For example, the number threshold n max In principle, the number of times threshold n max The higher , the more likely it is to obtain a subset of multiple-choice questions that meets the difficulty distribution condition 304, but the more computing resources are required.

[0078] A counter variable n indicating the number of times steps S702 to S706 have been performed on a subset of multiple-choice questions is set to a threshold value n that specifies the upper limit of the number of times the processes have been performed. max In the following cases, the counter variable n is incremented (n=n+1), the question generator 226 (not shown in FIG. 7) generates an additional set of multiple-choice questions (a second set of multiple-choice questions) for the context data or summary information, and after the evaluator 228 determines the difficulty assessment value, the processes of steps S702 to S706 described above are repeated for the additional set of multiple-choice questions. Thereafter, if a subset of multiple-choice questions (i.e., a set of multiple-choice questions including the first subset of multiple-choice questions and the second subset of multiple-choice questions) satisfies the difficulty distribution condition 304, these multiple-choice questions are output. On the other hand, the counter variable n indicating the number of times the processes of steps S702 to S706 have been performed on the subset of multiple-choice questions is equal to or exceeds the threshold value n maxIn the above cases, this process ends, and a notification is output to the user that a set of multiple-choice questions that meets the difficulty distribution condition 304 could not be generated. In this case, the generated multiple-choice questions may be output together.

[0079] According to the processing of the filtering unit 229 described above, it is possible to generate and provide multiple-choice questions with cognitive difficulty levels that meet the user's needs by eliminating low-quality multiple-choice questions from the set 302 of multiple-choice questions to which difficulty evaluation values ​​have been assigned and repeating the generation of questions until the difficulty distribution conditions specified by the user are satisfied.

[0080] Next, a specific example of processing by the question generation device according to an embodiment of the present disclosure will be described with reference to FIG.

[0081] FIG. 8 is a diagram illustrating a specific example of processing by the question generation device according to an embodiment of the present disclosure.

[0082] First, the summary generation unit 224 receives from the input unit 222 (not shown in Figure 8) the following information: "Cold fusion is a phenomenon in which a nuclear fusion reaction of hydrogen atoms occurs in the low temperature range from room temperature to about 1,000 degrees Celsius. There are several hypotheses as to the mechanism by which a fusion reaction occurs, such as the tunneling effect or muons contained in cosmic rays. This article deals with a nuclear fusion reaction that is said to have occurred at a low temperature, visible to the naked eye, and on a scale that could be used as a practical energy source. After the sensational announcement of cold fusion in 1989, it was called "the greatest scientific scandal of the 20th century" due to its low reproducibility, but in response to the need for decarbonization in recent years, there are hopes that it will be used in industry." " is generated, and by processing it with a large-scale language model, summary information 306 is generated, which reads, "There are multiple hypotheses about cold fusion, which is said to occur when hydrogen atoms fuse at low temperatures, and after its sensational announcement in 1989, doubts were raised due to issues with reproducibility, but in recent years, expectations for industrial use have been rising due to the need for decarbonization."

[0083] Next, the question generator 226 processes the prompt 510 requesting the generation of questions for the context data 302 and the summary information 306 using a large-scale language model to generate a set of multiple-choice questions 308. More specifically, the question generator 226 processes the prompt requesting the generation of questions for the context data 302 using a large-scale language model to generate the following questions as low-difficulty questions: Question 1: What is the principle behind the cold fusion reaction? Select from: A. Tunnel effect B. Caused by cosmic rays C. Phenomena that occur at high temperatures D. Caused by a chemical reaction Correct answer: A. Tunnel effect Explanation of correct answer: According to the article, one of the hypotheses proposed as the principle behind cold fusion is the tunneling effect. Furthermore, the question generator 226 processes a prompt requesting the generation of a question for the summary information 306 using a large-scale language model, thereby generating the following question as a high-difficulty question: Question 2: Based on the above, please explain why you think expectations for decarbonization will increase if the issue of reproducibility is resolved. Select from: A. Because the need for decarbonization has decreased B. A new source of energy has been discovered. C. Because expectations for industrial use are unrealistic D. It could be a practical energy source as long as reproducibility is not an issue. Correct answer: D. It could be a practical energy source as long as reproducibility is not an issue. Explanation of correct answer: According to the article, once the reproducibility issue is resolved, expectations for industrial use will increase due to the recent need for decarbonization. If the reproducibility issue is resolved, cold fusion may become a practical energy source, so expectations are likely to rise.

[0084] Next, the evaluation unit 228 generates a prompt 610 requesting an evaluation of the set of multiple-choice questions 302 received from the question generation unit 226, and processes the prompt 610 using a large-scale language model to generate information on difficulty ratings 310 indicating the cognitive difficulty of each of the multiple-choice questions included in the set of multiple-choice questions 308. As an example, the evaluation unit 228 may determine a difficulty rating of "1" for the above-mentioned question 1, and a difficulty rating of "3" for the above-mentioned question 2. In this case, the large-scale language model may output information indicating why a particular question was given a particular difficulty rating 310. For example, for question 2 above, the large-scale language model may output the following explanatory text: "This question requires you to understand the given information and reason about the impact of reproducibility issues if they are resolved. The application stage tests your skills in using memorized knowledge and applying it to new situations."

[0085] 7 based on the assigned difficulty assessment value 310 (Score_1, Score_2, ... Score_n) for each multiple-choice question (MCQ_1, MCQ_2, ... MCQ_n) included in the set of multiple-choice questions 302, and selects and outputs a subset of multiple-choice questions whose difficulty assessment value satisfies a predetermined cognitive difficulty threshold from the set of multiple-choice questions 302. For example, the filtering unit 229 may output the above-mentioned questions 1 and 2 as the subset of multiple-choice questions, as they satisfy the cognitive difficulty threshold.

[0086] The question generation device 210 described above can generate multiple-choice questions with cognitive difficulty levels tailored to the user's needs, making it possible to more accurately evaluate the learner's understanding and the performance of a large-scale language model.

[0087] As described above, one aspect of the question generation means according to the embodiment of the present disclosure relates to generating multiple-choice questions with cognitive difficulty levels tailored to a user's needs. By processing the context data with a large-scale language model, low-difficulty questions for the context data can be generated, and by processing summary information extracted from the main points of the context data with the large-scale language model, high-difficulty questions for the context data can be generated.

[0088] By using a large-scale language model to generate multiple-choice questions for summary information indicating key points extracted from context data, it is possible to obtain higher-level cognitively challenging questions that require a comprehensive understanding of the context data, compared to when the context data is directly input into the large-scale language model.

[0089] Furthermore, when the context data includes long sentences (e.g., when the context data meets length criteria such as 10 pages or more or 30,000 characters or more), the summary generation unit according to an embodiment of the present disclosure may divide the context data into multiple context data portions each having a predetermined length and process each of the context data portions using a large-scale language model to generate partial summary information indicating extracted key points for each of the context data portions.The question generation unit may then process each partial summary information using the large-scale language model to generate multiple-choice questions.

[0090] Here, the question generator may process each piece of partial summary information individually to generate a multiple-choice question, or may aggregate multiple pieces of partial summary information to generate a multiple-choice question. By processing each piece of partial summary information individually to generate a multiple-choice question, it becomes possible to generate a high-difficulty question that individually requires understanding of multiple topics included in the context data. On the other hand, by aggregating multiple pieces of partial summary information (e.g., partial summary information of context data that meets a predetermined relevance criterion) to generate a multiple-choice question, it becomes possible to generate a high-difficulty question that requires a comprehensive understanding of multiple related topics.

[0091] Furthermore, by using an evaluation unit according to an embodiment of the present disclosure to evaluate the multiple-choice questions generated by the question generation unit based on a predetermined cognitive difficulty evaluation criterion, and using a filtering unit according to an embodiment of the present disclosure to select multiple-choice questions that satisfy a predetermined cognitive difficulty threshold, a subset of multiple-choice questions that has an appropriate balance of low-difficulty questions and high-difficulty questions and satisfies a difficulty distribution condition specified by a user can be obtained. By presenting the multiple-choice questions obtained in this manner to a human learner or a large-scale language model, it is possible to evaluate the understanding of the human learner and the large-scale language model.

[0092] According to the question generation means according to the embodiment of the present disclosure described above, it is possible to provide a question generation means that can more accurately evaluate the understanding of a learner and the performance of a large-scale language model.

[0093] As described above, the question generation means according to the embodiment of the present disclosure relates to the following aspects.

[0094] (Aspect 1) A question generation device, a processor and a memory, The memory includes: an input unit for acquiring context data consisting of text information; a question generator that processes the context data through a first large-scale language model to generate a first set of multiple-choice questions for the context data; an evaluation unit that evaluates the first set of multiple-choice questions based on a predetermined cognitive difficulty evaluation criterion to determine a difficulty evaluation value indicating a cognitive difficulty of the first set of multiple-choice questions; a filtering unit for selecting and outputting a subset of first multiple-choice questions from the first set of multiple-choice questions, the subset of first multiple-choice questions having difficulty assessment values ​​that satisfy a predetermined cognitive difficulty threshold; a processing instruction for causing the processor to function as a query generator;

[0095] (Aspect 2) The input unit obtaining, in addition to the context data, a difficulty distribution condition that defines a proportion of low-difficulty questions in the first set of multiple-choice questions whose difficulty assessment values ​​meet a first cognitive difficulty threshold and high-difficulty questions in the first set of multiple-choice questions whose difficulty assessment values ​​meet a second cognitive difficulty threshold; 2. The question generation device according to claim 1,

[0096] (Aspect 3) The memory includes: and further comprising processing instructions for causing the processor to function as a summary generator that processes the context data using a second large-scale language model to generate summary information indicating key points extracted from the context data. 3. The question generation device according to aspect 2.

[0097] (Aspect 4) The question generation unit generating the low difficulty questions for the context data by processing the context data with the first large-scale language model; generating the high-level questions for the context data by processing the summary information with the first large-scale language model; 4. The question generation device according to aspect 3.

[0098] (Aspect 5) The filtering unit generating the first subset of multiple-choice questions by eliminating multiple-choice questions from the first set of multiple-choice questions that do not meet the first cognitive difficulty threshold; determining whether the proportion of the low difficulty questions and the high difficulty questions included in the first subset of multiple-choice questions satisfies the difficulty distribution condition; outputting the first subset of multiple-choice questions if the ratio of the low difficulty questions and the high difficulty questions included in the first subset of multiple-choice questions satisfies the difficulty distribution condition; If the ratio of the low difficulty questions and the high difficulty questions included in the first subset of multiple-choice questions does not satisfy the difficulty distribution condition, The question generation unit generating a second set of multiple-choice questions for the context data by processing the context data or the summary information with the first large-scale language model; The filtering unit generating a second subset of multiple-choice questions by eliminating multiple-choice questions from the second set of multiple-choice questions that do not meet the first cognitive difficulty threshold; determining whether the ratio of the low difficulty questions and the high difficulty questions included in the first subset of multiple-choice questions and the second subset of multiple-choice questions satisfies the difficulty distribution condition; outputting the first subset of multiple-choice questions and the second subset of multiple-choice questions when the ratio of the low difficulty questions and the high difficulty questions included in the first subset of multiple-choice questions and the second subset of multiple-choice questions satisfies the difficulty distribution condition; 5. The question generation device according to claim 4,

[0099] (Aspect 6) The summary generation unit If it is determined that the context data satisfies a predetermined length criterion, dividing the context data into a plurality of context data portions each having a predetermined length, and processing each of the context data portions with the second large-scale language model to generate partial summary information indicating extracted key points for each of the context data portions; The question generation unit identifying a first context data portion and a second context data portion among the plurality of context data portions that satisfy a predetermined relevance criterion; generating the high-difficulty question for the context data by processing, in the first large-scale language model, aggregated partial summary information that aggregates first partial summary information indicating main points extracted from the identified first context data portion and second partial summary information indicating main points extracted from the identified second context data portion; 6. The question generation device according to aspect 4 or 5.

[0100] (Aspect 7) The evaluation unit inputting a subset of the first multiple-choice questions into a third large-scale language model; determining a performance score that quantitatively indicates a performance of the third large-scale language model based on a correct answer rate of the answers of the third large-scale language model to the subset of the first multiple-choice questions; 7. The question generation device according to any one of aspects 1 to 6.

[0101] (Aspect 8) The evaluation unit For each question in the first set of multiple-choice questions: determining a first difficulty evaluation value for a question in which the correct answer to the question sentence is incorrect; determining a second difficulty rating for questions requiring memorization and recall of certain knowledge; determining a third difficulty rating for questions requiring interpretation of memorized knowledge; determining a fourth difficulty rating for questions that require applying memorized knowledge to a predetermined problem to solve the problem; A fifth difficulty rating is given to questions that require the student to break down a complex problem into several elements and understand the structure. A sixth difficulty rating is assigned to questions that require critical evaluation and judgment of information or ideas. A seventh difficulty rating is assigned to questions that require students to integrate multiple elements to create a new overall picture. 8. The question generation device according to any one of aspects 1 to 7.

[0102] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0103] 150 Question Generation Applications 200 Question Generation System 210 Question generator 220 memory 222 Input section 224 Summary generator 226 Question generation part 228 Evaluation Department 229 Filtering Section 230 Storage section 236 Context DB 238 Question DB 244 processors 246 Input / output section 250 Communication Network 260 User Terminals

Claims

1. A question generation device, comprising: a processor and a memory, The memory includes: an input unit for acquiring context data consisting of text information; a question generator that processes the context data through a first large-scale language model to generate a first set of multiple-choice questions for the context data; an evaluation unit that evaluates the first set of multiple-choice questions based on a predetermined cognitive difficulty evaluation criterion to determine a difficulty evaluation value indicating a cognitive difficulty of the first set of multiple-choice questions; a filtering unit for selecting and outputting a subset of first multiple-choice questions from the first set of multiple-choice questions, the subset of first multiple-choice questions having difficulty assessment values ​​that satisfy a predetermined cognitive difficulty threshold; a processing instruction for causing the processor to function as a query generator;

2. The input unit acquiring, in addition to the context data, a difficulty distribution condition that defines a ratio of low difficulty questions in the first set of multiple-choice questions whose difficulty assessment values ​​meet a first cognitive difficulty threshold and high difficulty questions in the first set of multiple-choice questions whose difficulty assessment values ​​meet a second cognitive difficulty threshold; 2. The question generation device according to claim 1, wherein:

3. The memory includes: and further comprising processing instructions for causing the processor to function as a summary generator that processes the context data with a second large-scale language model to generate summary information indicating key points extracted from the context data.

3. The question generation device according to claim 2.

4. The question generation unit generating the low difficulty questions for the context data by processing the context data with the first large-scale language model; generating the high-level questions for the context data by processing the summary information with the first large-scale language model; 4. The question generation device according to claim 3, wherein:

5. The filtering unit generating the first subset of multiple-choice questions by eliminating multiple-choice questions from the first set of multiple-choice questions that do not meet the first cognitive difficulty threshold; determining whether the proportion of the low difficulty questions and the high difficulty questions included in the first subset of multiple-choice questions satisfies the difficulty distribution condition; outputting the first subset of multiple-choice questions if the ratio of the low difficulty questions and the high difficulty questions included in the first subset of multiple-choice questions satisfies the difficulty distribution condition; If the ratio of the low difficulty questions and the high difficulty questions included in the first subset of multiple-choice questions does not satisfy the difficulty distribution condition, The question generation unit generating a second set of multiple-choice questions for the context data by processing the context data or the summary information with the first large-scale language model; The filtering unit generating a second subset of multiple-choice questions by eliminating multiple-choice questions from the second set of multiple-choice questions that do not meet the first cognitive difficulty threshold; determining whether the ratio of the low difficulty questions and the high difficulty questions included in the first subset of multiple-choice questions and the second subset of multiple-choice questions satisfies the difficulty distribution condition; outputting the first subset of multiple-choice questions and the second subset of multiple-choice questions when the ratio of the low difficulty questions and the high difficulty questions included in the first subset of multiple-choice questions and the second subset of multiple-choice questions satisfies the difficulty distribution condition; 5. The question generation device according to claim 4.

6. The summary generation unit If it is determined that the context data satisfies a predetermined length criterion, dividing the context data into a plurality of context data portions each having a predetermined length, and processing each of the context data portions with the second large-scale language model to generate partial summary information indicating extracted key points for each of the context data portions; The question generation unit identifying a first context data portion and a second context data portion among the plurality of context data portions that satisfy a predetermined relevance criterion; generating the high-difficulty question for the context data by processing, in the first large-scale language model, aggregated partial summary information obtained by aggregating first partial summary information indicating main points extracted from the identified first context data portion and second partial summary information indicating main points extracted from the identified second context data portion; 5. The question generation device according to claim 4.

7. The evaluation unit inputting the subset of first multiple-choice questions into a third large-scale language model; determining a performance score that quantitatively indicates a performance of the third large-scale language model based on a correct answer rate of the third large-scale language model to the subset of the first multiple-choice questions; 2. The question generation device according to claim 1, wherein:

8. The evaluation unit For each question in the first set of multiple-choice questions: determining a first difficulty evaluation value for a question in which the correct answer to the question sentence is incorrect; determining a second difficulty rating for questions requiring memorization and recall of certain knowledge; determining a third difficulty rating for questions requiring interpretation of memorized knowledge; determining a fourth difficulty evaluation value for a question that requires the user to apply memorized knowledge to a predetermined problem and solve it; A fifth difficulty evaluation value is determined for questions that require the user to break down a complex problem into several elements and understand the structure; assigning a sixth difficulty rating to questions that require critical evaluation and judgment of information or ideas; A seventh difficulty rating is determined for questions that require integrating multiple elements to create a new overall picture.

2. The question generation device according to claim 1, wherein:

9. A question generation method implemented in a question generation device, comprising: The question generation device a processor and a memory, The question generation method includes: The processing instructions stored in the memory obtaining context data comprising text information; generating a first set of multiple-choice questions for the context data by processing the context data with a first large-scale language model; determining a difficulty rating value indicative of cognitive difficulty for the first set of multiple-choice questions by rating the first set of multiple-choice questions based on a predetermined cognitive difficulty rating criterion; selecting and outputting a subset of first multiple-choice questions from the first set of multiple-choice questions, the subset of first multiple-choice questions having difficulty assessment values ​​that meet a predetermined cognitive difficulty threshold; A question generation method characterized by causing the processor to execute the following.

10. a question generator; A question generation system connected to a user terminal via a communication network, The question generation device a processor and a memory, The memory includes: an input unit that acquires context data consisting of text information from the user terminal; a question generator that processes the context data through a first large-scale language model to generate a first set of multiple-choice questions for the context data; an evaluation unit that evaluates the first set of multiple-choice questions based on a predetermined cognitive difficulty evaluation criterion to determine a difficulty evaluation value indicating a cognitive difficulty of the first set of multiple-choice questions; a filtering unit that selects, from the first set of multiple-choice questions, a subset of first multiple-choice questions whose difficulty assessment values ​​satisfy a predetermined cognitive difficulty threshold, and outputs the subset to the user terminal; a processing instruction for causing the processor to function as a query generator;

Images