A computer-aided system based on machine learning and comprising at least one processor, giving distractor suggestions for multiple-choice question generation and working method thereof

Abstract

The invention relates to a computer-aided system based on machine learning and comprising at least one processor, giving distractor suggestions for multiple-choice question generation, and to the working method thereof.

Description

[0001] A COMPUTER-AIDED SYSTEM BASED ON MACHINE LEARNING AND COMPRISING AT LEAST ONE PROCESSOR, GIVING DISTRACTOR SUGGESTIONS FOR MULTIPLE-CHOICE QUESTION GENERATION AND WORKING METHOD THEREOF

[0002] Technical Field of the Invention

[0003] The invention relates to a machine learning-based, computer-aided system with at least one processor that offers distractor suggestions for multiple-choice question generation, and to the working method thereof.

[0004] State of the Art

[0005] Examination systems used to measure and evaluate people's abilities in general, especially in the field of education, commonly include multiple-choice questions. In multiple-choice questions, there are essentially three main components consisting of the question text, the correct answer option, and the distractor options. The respondent is expected to choose the correct answer from all provided options / alternatives based on the question text. To improve the quality of the question, it is crucial that the distractors are contextually aligned with the text and / or question, are plausible, and can potentially cause confusion or dilemmas among the alternatives for the respondent.

[0006] In the state of the art, generating distractors for multiple-choice questions from a text is a challenging and time-consuming process for question preparers. Although the automatic question generation methods and systems used practically today have reached a satisfactory level, they remain superficial in producing distractors / false alternatives to multiple-choice questions. For example, some classical methods generate distractors by using keywords from the input text. Some systems train word pairs from classical natural language processing (NLP) methods and knowledge bases (e.g., WordNet) to determine which distractors are better, but these methods require a knowledge base for a specific language.

[0007] Patent document no. US2017330079 in the prior art relates to an automatic distractor generation system that performs disambiguation processes. In the system of the invention in the aforementioned document, by creating the text in the digital environment as input, along with concept dependency graph based on the input text from the subject domain and the same subject domain from the reference text, concept and keywords within the system, along with the relationships between the nodes, a distractor extraction process is carried out. The system of the invention in the aforementioned patent document (US2017330079) also has the ability to generate and filter technical multiple-choice questions. There is also a question prioritization module in said system. The aforementioned module uses meta information along with the questions and decides which question should be processed first and output produced. There is a need for a knowledge base within the system for the use of the technique in said system. The system also has parts such as concept and keyword extraction from this knowledge base and the tools within, disambiguation output from keywords, and question filtering (for example, eliminating a question whose answer is a pronoun). The system described in the aforementioned invention uses classical and traditional natural language processing (NLP) methods, in contrast to modern NLP models (e.g., Transformer, allMLP) and techniques. Furthermore, said system has disadvantages such as the necessity of the technique to contain a knowledge base and the difficulty of language-specific processing. In addition to the aforementioned disadvantages, the method used in the system of the invention is focused on words and cannot both process and produce output for a series of words or word groups as a higher-level. When all these reasons are considered in detail, the outputs of the aforementioned system remain superficial in terms of distractor and therefore question quality (e.g., distractors can be easily distinguished).

[0008] Patent document no. US10817790B2 in the state of the art is about a system that creates an automatic distractor by determining the relationships between reference keywords and concepts. In said system, to create automatic distractors, steps such as providing a subject area related to the input text, including a question or questionanswer pair, building a reference corpus, and extracting and constructing relational graphs are required. The system of the invention in the aforementioned patent application can work on classical and relational graphics and only on keywords and specified concepts. In other words, there is a growing need for modern large language models and a system that can fully process the context giving distractor suggestions for the production of multiple-choice questions. In the present art, there is an application called "assessmentQ" supported by high-dose machine learning. Said application includes automatic production of a distractor for the generation of multiple-choice questions. In said application, ready-made machine learning tools (ChatGPT, etc.) are used without algorithmic development on the language model side; in other words, there is a need for 3rd party applications or an external source. Since ready-made machine learning tools are used in said application, it is not possible to use models that are trained specifically for the user's needs and under specified conditions.

[0009] A study [1] conducted by Zhaopeng Qiu, Xian Wu and Wei Fan in the state of the art, discusses the automatic generation of a distractor for multiple-choice questions in standardized tests. In the system proposed in said study, the question is needed as input during the automatic generation of distractor for multiple-choice questions. In this study, the input text (passage) can be considered as a pre-processing for the question and answer, and three representation vectors are deduced from the language model separately, and then it is aimed that the three vectors mentioned interact with each other through the fusion and attention layers. This arrangement creates an extra burden on the computing power. In addition, the system described in said study is a structure that requires training and the results of the training are also shared in the document. In other words, a training-independent structure in which training is offered optionally cannot be presented in the aforementioned study.

[0010] Due to reasons such as the limitations and inadequacies of the systems giving distractor suggestions for the generation of multiple-choice questions in the present art and the working methods of said systems, the aforementioned systems requiring a knowledge bank for a specific language, the outputs produced by the classical and traditional natural language processing (NLP) methods used in the systems remaining superficial, the disadvantages of the systems in the state of the art such as the need for the technique to have a knowledge base and the difficulty of language-specific processing, focusing only on words and not being able to process and produce output for the series of words or word groups in the systems of the present art, the systems in the state of the art being able to work on classical and relational graphics and only on keywords and specified concepts, the increasing need for modern models of large language in the systems of the present art giving distractor suggestions for multiplechoice question generation, as well as for a system giving distractor suggestions for multiple-choice question generation that can fully process the context, the need for 3rd party applications (ChatGPT, etc.) or an external source in systems supported by machine learning, giving distractor suggestions for multiple-choice question generation in the state of the art, needing the question as input during the automatic generation of the distractor for multiple-choice questions in the systems addressing the issue of automatic distractor generation for multiple-choice questions in standardized tests in the present art, and said systems creating an extra burden on the computational power and an training-independent structure in which training being offered optionally is not possible in the aforementioned systems, it was made necessary to introduce a system giving distractor suggestions for multiple-choice question generation, which eliminates all these problems, and a working method for this system.

[0011] Brief Description and Object of the Invention

[0012] The invention describes a machine learning-based, computer-aided system with at least one processor that offers distractor suggestions for multiple-choice question generation, along with its operating method. The system and method do not require a language-specific data bank, and the outputs are not superficial. The invention addresses disadvantages such as the need for a knowledge base and the difficulty of language-specific processing. Instead, it processes sequences of words or word groups, leveraging modern large language models and the context comprehensively. The system eliminates the need for third-party applications (e.g., ChatGPT) or any external sources and removes the necessity of the question as input during automatic distractor generation for multiple-choice questions. Additionally, it prevents extra computational load and allows for a training-independent structure.

[0013] The aim of the invention is to save manpower and reduce costs by automating the distractor determination stage, which is time-consuming and resource-intensive, during the preparation phase of multiple-choice questions in fields such as education, industry (e.g., human resources), and research surveys.

[0014] An objective of the invention is to present a machine learning-based, computer-aided system with at least one processor that suggests distractors for multiple-choice question generation without requiring a language-specific knowledge base, and its working method. A machine learning-based, computer-aided system with at least one processor that suggests distractors for multiple-choice question generation without requiring a language-specific knowledge base, and its working method operates through a two-step algorithm. In the first step, candidate distractors are generated using a Transformer-based large language model trained with the masked-language modeling objective. In the second step, these candidates are filtered using another Transformer-based model trained on the Recognizing Textual Entailment (RTE) problem, resulting in the final set of distractors.

[0015] Another objective of the invention is to provide a machine learning-based, computer- aided system with at least one processor that offers distractor suggestions for multiplechoice question generation, ensuring non-superficial outputs, and its working method. The achievement of ensuring non-superficial outputs via a machine learning-based, computer-aided system with at least one processor that offers distractor suggestions for multiple-choice question generation and its working method is provided at the second stage that comprises filtering candidates that either have the same or similar meaning with the answer potentially invalidating the question or have similar meanings within the candidates where distractors are tested and filtered sequentially to avoid inconsistencies among them by the distractor selection module (300).

[0016] In the invention, a machine learning-based, computer-aided system with at least one processor that offers distractor suggestions for multiple-choice question generation, which eliminates the disadvantages such as the necessity of the technique to contain a knowledge base and the difficulty of performing language-specific processing, and the working method thereof are provided. A machine learning-based, computer-aided system with at least one processor that offers distractor suggestions for multiple-choice question generation, which eliminates the disadvantages such as the necessity of the technique to contain a knowledge base and the difficulty of performing languagespecific processing, and the working method thereof are introduced in the invention by the use of a pretrained language model (PLM) (15), which has completed its pretraining phases and has “learned” its general knowledge through training, and also by not subjecting these models to fine-tuning / further-training processes.

[0017] With the invention, a machine learning-based, computer-aided system with at least one processor that offers distractor suggestions for multiple-choice question generation, which can process a series of words or word group and produce output, in other words, which does not only focus on words, and the working method thereof are introduced. The introduction of a machine learning-based, computer-aided system with at least one processor that offers distractor suggestions for multiple-choice question generation, which can process a series of words or word group and produce output, in other words, which does not only focus on words, and the working method thereof is presented with a new and effective algorithm in the invention, and this algorithm is achieved by using a unique decoding strategy in the masking module (12) during model input generation through the language model used in the first step, which maximizes the joint probability and consequently generates multiple tokens and thus candidate distractors (17) as output. The problem of the systems in the present art only being able to work through classical and relational graphs and on the basis of keywords and defined concepts is solved with the system of the invention and the working method thereof.

[0018] In the invention, a machine learning-based, computer-aided system with at least one processor that offers distractor suggestions for multiple-choice question generation, which satisfies the need for a system making distractor suggestions for multiple-choice question generation that can fully process modern large language models and the context, and the working method thereof are introduced. The introduction of a machine learning-based, computer-aided system with at least one processor that offers distractor suggestions for multiple-choice question generation, which satisfies the need for a system making distractor suggestions for multiple-choice question generation that can fully process modern large language models and the context, and the working method thereof are provided in the invention by a pretrained language model (PLM) (15), a novel decoding algorithm, and a masking module (12).

[0019] In the invention, a machine learning-based, computer-aided system with at least one processor that offers distractor suggestions for multiple-choice question generation, which eliminates the need for the question as input during the automatic generation of the distractor for multiple choice questions, and the working method thereof are provided. A machine learning-based, computer-aided system with at least one processor that offers distractor suggestions for multiple-choice question generation, which eliminates the need for the question as input during the automatic generation of the distractor for multiple choice questions, and the working method thereof are provided in the invention by a system that works by using only the given input text and the answer part in the text and masking the tokens, and that works by creating output from models with these inputs, therefore that does not require the question text. In other words, a machine learning-based, computer-aided system with at least one processor that offers distractor suggestions for multiple-choice question generation, which eliminates the need for the question as input during the automatic generation of the distractor for multiple choice questions, and the working method thereof are provided in the invention by the candidate distractor generation module (200).

[0020] With the invention, a machine learning-based, computer-aided system with at least one processor that offers distractor suggestions for multiple-choice question generation, in which training is offered optionally, and a training-independent structure is made possible, and the working method thereof are provided. A machine learning-based, computer-aided system with at least one processor that offers distractor suggestions for multiple-choice question generation, in which training is offered optionally, and a training-independent structure is made possible, and the working method thereof are provided in the invention by a system structure that makes use of Pretrained Language Models (PLMs) for the first step and models trained for the Natural Language Inference (NLI) problem type for the second step which are already available in the invention and can be used for direct prediction, and also does not require a further training process. In other words, a machine learning-based, computer-aided system with at least one processor that offers distractor suggestions for multiple-choice question generation, in which training is offered optionally, and a training-independent structure is made possible, and the working method thereof are provided in the invention by a candidate distractor generation module (200), a pretrained language model (PLM) (15) and a language model trained on the natural language inference (NLI) problem (24).

[0021] Description of the Drawings

[0022] Figure 1. A schematic representative illustration of a machine learning-based, computer-aided system with at least one processor that offers distractor suggestions for multiple-choice question generation.

[0023] Figure 2. A schematic representative illustration of an exemplary working method flow of the Distractor Generation System.

[0024] Description of the References in Drawings 1. Application interface

[0025] 2. Text and answer

[0026] 3. Final distractors

[0027] 11. Pre-processing module

[0028] 12. Masking module

[0029] 13. Masking configuration module

[0030] 14. Prediction module

[0031] 15. Pretrained language model (PLM)

[0032] 17. Candidate distractors

[0033] 21. Answer-Distractor Comparison Module

[0034] 22. Filtered distractors

[0035] 23. Distractor-Distractor Comparison Module

[0036] 24. Language model trained on Natural Language Inference (NLI) problem

[0037] 30. Processor

[0038] 11. Pre-Processing Module

[0039] 101. Input layer in text-answer format

[0040] 102. Text markup: passage / context

[0041] 103. Text markup: answer

[0042] 200. Candidate distractor generation module

[0043] 201. Masking module (12) context outputs

[0044] 202. Masking module (12) answer output (mask)

[0045] 204. PLM prediction outputs

[0046] 300. Distractor Selection Module

[0047] 302. Answer-Distractor Comparison Module Outputs

[0048] 400. Final distractor suggestions

[0049] 401. Distractor Selection Module Outputs

[0050] Detailed Description of the Invention

[0051] The invention pertains to a machine learning-based, computer-aided system with at least one processor that offers distractor suggestions for multiple-choice question generation, and to the working method thereof. The system of the invention and the working method thereof do not require a specific language data bank, the outputs from said system are not superficial, the disadvantages, such as the requirement for a knowledge base and the challenges of language-specific processing, are eliminated by the system and method of the invention; with the system and method of the invention, a structure that can process a series of words or word group and produces output providing a structure that does not solely focus on words but also fully processes modern large language models along with their context is provided; with the system of the invention, the need for third-party applications (such as ChatGPT) or any external sources is eliminated, as is the requirement for the question as an input during the automatic generation of distractors for multiple-choice questions is eliminated, an extra load on the computational power is prevented and an training-independent structure is made possible.

[0052] A computer-aided system of the invention based on machine learning and comprising at least one processor, giving distractor suggestions for multiple-choice question generation comprises;

[0053] • an application interface (1) enabling the import of inputs in various formats, the export of system outputs in various formats, adjustment of parameter settings that can be determined optionally, and the provision of the entire configuration environment in which the user can interact with the system,

[0054] • a pre-processing module (11 ) performing data cleaning, normalization / text standardization, determination or modification of the appropriate encoding format, tokenization of the text, and conversion of the raw input into input format for the candidate distractor generation module (200) on imported texts,

[0055] • an advanced candidate distractor generation module (200) and a distractor selection module (300) performing the generation of distractors for the answers specified by the users through the interface and the associated multiple-choice questions,

[0056] • a masking module (12) creating the masking process and subsequently model inputs through the tokens according to the settings and values determined in the masking configuration module (13),

[0057] • a masking configuration module (12) used in the task of selecting specific parameters and settings for the masking process and the Masking Module (13),

[0058] • a prediction module (14) used to execute the prediction process using inputs from the pretrained language model (PLM) (15),

[0059] • pretrained language model (PLM) (15) module used in the language model task in the prediction operation carried out through the prediction module (14), • an answer-distractor comparison module (21 ) generating filtered distractors (22) by comparing the set of candidate distractors (17) produced in the first step with the answers through a language model (24) trained on the Natural Language Inference (NLI) problem and passing this through a filtering system,

[0060] • a distractor-distractor comparison module (23) determining the suitability of candidate distractors (17) by comparing them with each other and used in a systematic filtering task,

[0061] • a processor used to run the system in an end-to-end electronic environment and to operate the accompanying algorithms and modules (30),

[0062] • a distractor selection module (300) used in the task of systematically and sequentially filtering out candidate distractors (17) and systematically filtering both distractors that are incompatible with the answer and distractors that are incompatible with each other,

[0063] • a candidate distractor generation module (200) in which the text and answer (2) both are passed through text-cleaning and tokenization processes in the preprocessing module (11 ).

[0064] The candidate distractor generation module (200) mentioned here makes it possible to produce distractors suitable for the context and flow in an amount determined by the user through transformer-based large language models, by advanced decoding and gradual model outputs processing methods and advanced filtering methods for the answers specified by the users through the interface and related multiple-choice questions.

[0065] In a computer-aided system of the invention based on machine learning and comprising at least one processor, giving distractor suggestions for multiple-choice question generation, and the working method thereof; the text is unbundled by importing a text in digital format or specified file formats containing text as input. Afterwards, the text mentioned is included in the pre-processing process and text editing, normalization and cleaning processes are applied. The distractor generation method consists of two phases. In the first phase, the answer word or word group determined by the user and mentioned in the text is masked by applying the word masking technique. After masking, using a bidirectional large language model that has completed the pre-training process, k amount of different words or word groups, which are hyper-parameters, are produced by means of the model for the masked part. (Language generation). The options produced in k amount are subjected to a filtering mechanism in the second phase of the algorithm. In the aforementioned filtering mechanism, using a model previously trained on the textual entailment objective, the conditional relationship between the real answer and the outputs produced in the first phase is first examined, and if the model semantically detects a connection between the two, the first phase outputs are eliminated at this stage. In the next stage, the remaining and uneliminated outputs become potential distractor candidates. The relationship between the potential distractor candidates mentioned (in descending order) is examined, and if a candidate in descending order with a low score is similar to any of the candidates with a higher score, that candidate is also eliminated at this stage. The remaining candidates are false alternatives that do not coincide with the answer determined by the system, but are similar in context.

[0066] To give an example of the functioning system of the inventionfor example, a teacher wanting to prepare multiple-choice questions from a digital text-based source can import the digital text document through the application interface (1), specify the answers, and obtain the distractors generated by the system for those answers in a time-efficient manner. Furthermore, users are offered the use of custom-built machine learning models for different areas of knowledge.

[0067] The working method of a computer-aided system of the invention based on machine learning and comprising at least one processor, giving distractor suggestions for multiple-choice question generation comprises; i. transferring the determined text and answer (a substring in the text) to the candidate distractor generation module (200) via the application interface (1 ) and subjecting the text and answer (2) to text-cleaning and tokenization processes in the pre-processing module (11) within the candidate distractor generation module (200), ii. transferring the outputs received to the masking module (12) and, in the masking module (12), creating the masking process and subsequently model inputs through the tokens according to the settings and values determined in the masking configuration module (13), iii. generating candidate distractors (17) by transferring the generated model inputs to the prediction module (14) and using the pretrained language model module (PLM) (15), iv. transferring the candidate distractors (17), which are the output of the candidate distractor generation module (200), to the distractor selection module (300), v. transferring candidate distractors (17) to the answer-distractor comparison module (21 ), which is the first step and sub-module of the distractor selection module (300), vi. generating filtered distractors (22) with answer-distractor comparison module

[0068] (21) by comparing the set of candidate distractors (17) produced in the first step with the answers through a language model (24) trained on the Natural Language Inference (NLI) problem and passing this through a filtering system, vii. subjecting the distractors to a filtering system by comparing them with each other through filtered distractors (22), distractor-distractor comparison module (23) and once again a language model (24) trained on the natural language interface (NLI) problem, and generating the final distractors (3) and transferring them to the application interface (1).

[0069] Figure 1 describes / discloses an embodiment example of a distractor generation system for multiple-choice questions including a candidate distractor generation module (200), a distractor selection module (300), an application interface (1 ) and a processor (30) connected to it. The application interface (1) allows the user to give inputs through interaction with the processor (30) and transfers these inputs to the candidate distractor generation module (200), which is the next step. The candidate distractor generation module (200) consists of three different sub-modules, these are; preprocessing module (11 ), masking module (12) connected to masking configuration module (13), and prediction module (14) connected to pretrained language model (PLM) (15). The distractor selection module (300) consists of two sub-modules, the answer-distractor comparison module (21 ) and the distractor-distractor comparison module (23), both of which depend on the language model (24) trained on the natural language inference (NLI) problem.

[0070] The application interface (1) is an interface that enables users to interact and enter text-based data and create an answer to the question in the text, and the determined text and answer (2) (a substring in the text) is transferred to the candidate distractor generation module (200) via the application interface (1). The text and answer (2) both are subjected to certain processes such as text-cleaning and tokenization in the pre- processing module (11). Following these processes, the outputs received are transferred to the masking module (12) and, in the masking module (12), the masking process and subsequently model inputs are created through the tokens according to the settings and values determined in the masking configuration module (13). Candidate distractors (17) are generated by transferring the generated model inputs to the prediction module (14) and using the pretrained language model module (PLM) (15), The candidate distractors (17), which are the output of the candidate distractor generation module (200), are transferred to the distractor selection module (300). The candidate distractors (17) are transferred to the answer-distractor comparison module (21 ), which is the first step and sub-module of the distractor selection module (300), Filtered distractors (22) are generated with answer-distractor comparison module (21) by comparing the set of candidate distractors (17) produced in the first step with the answers through a language model (24) trained on the Natural Language Inference (NLI) problem and passing this through a filtering. The distractors are subjected to a filtering system by comparing them with each other through filtered distractors (22), distractor-distractor comparison module (23) and once again a language model (24) trained on the natural language interface (NLI) problem, the final distractors (3) are generated and transferred to the application interface (1 ).

[0071] References

[0072] [1] Automatic distractor generation for multiple choice questions in ... (n.d.-a). https: / / aclanthology.org / 2020.coling-main.189.pdf

Claims

CLAIMS1. A computer-aided system based on machine learning and comprising at least one processor, giving distractor suggestions for multiple-choice question generation, characterized in that it comprises:• an application interface (1) enabling the import of inputs in various formats, the export of system outputs in various formats, adjustment of parameter settings that can be determined optionally, and the provision of the entire configuration environment in which the user can interact with the system,• a pre-processing module (11 ) performing data cleaning, normalization / text standardization, determination or modification of the appropriate encoding format, tokenization of the text, and conversion of the raw input into input format for the candidate distractor generation module (200) on imported texts,• an advanced candidate distractor generation module (200) and a distractor selection module (300) performing the generation of distractors for the answers specified by the users through the interface and the associated multiple-choice questions,• a masking module (12) creating the masking process and subsequently model inputs through the tokens according to the settings and values determined in the masking configuration module (13),• a masking configuration module (12) used in the task of selecting specific parameters and settings for the masking process and the Masking Module (13),• a prediction module (14) used in the task of executing the prediction process using inputs from the pretrained language model (PLM) (15),• pretrained language model (PLM) (15) module used in the language model task in the prediction operation carried out through the prediction module (14),• an answer-distractor comparison module (21) generating filtered distractors (22) by comparing the set of candidate distractors (17) produced in the first step with the answers through a language model (24) trained on the Natural Language Inference (NLI) problem and passing this through a filtering system,• a distractor-distractor comparison module (23) determining the suitability of candidate distractors (17) by comparing them with each other and used in a systematic filtering task,• a processor used to run the system in an end-to-end electronic environment and to operate the accompanying algorithms and modules (30),• a distractor selection module (300) used in the task of systematically and sequentially filtering out candidate distractors (17) and systematically filtering both distractors that are incompatible with the answer and distractors that are incompatible with each other,• a candidate distractor generation module (200) in which the text and answer (2) both are passed through text-cleaning and tokenization processes in the preprocessing module (11).

2. The working method of a computer-aided system based on machine learning and comprising at least one processor, giving distractor suggestions for multiple-choice question generation, characterized in that it comprises the process steps of: i. transferring the determined text and answer (a substring in the text) to the candidate distractor generation module (200) via the application interface (1) and, in the candidate distractor generation module (200), passing the text and answer (2) both through text-cleaning and tokenization processes in the pre-processing module (11 ), ii. transferring the outputs received to the masking module (12) and, in the masking module (12), creating the masking process and subsequently model inputs through the tokens according to the settings and values determined in the masking configuration module (13), iii. generating candidate distractors (17) by transferring the generated model inputs to the prediction module (14) and using the pretrained language model module (PLM) (15), iv. transferring the candidate distractors (17), which are the output of the candidate distractor generation module (200), to the distractor selection module (300),v. transferring candidate distractors (17) to the answer-distractor comparison module (21), which is the first step and sub-module of the distractor selection module (300), vi. generating filtered distractors (22) with answer-distractor comparison module (21 ) by comparing the set of candidate distractors (17) produced in the first step with the answers through a language model (24) trained on the Natural Language Inference (NLI) problem and passing this through a filtering system, vii. subjecting the distractors to a filtering system by comparing them with each other through filtered distractors (22), distractor-distractor comparison module (23) and once again a language model (24) trained on the natural language interface (NLI) problem, and generating the final distractors (3) and transferring them to the application interface (1).

Images