Ai-based customized online exam management solution providing system

The AI-driven online exam system addresses the lack of comprehensive management in current systems by automating question generation, cheating detection, and personalized feedback, improving efficiency and learning outcomes.

KR1020260163554APending Publication Date: 2026-11-17서원직
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
KR1020250058893
Authority / Receiving Office
KR · KR
Patent Type
Applications
Current Assignee / Owner
Filing Date
2025-05-07
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Current online exam systems lack comprehensive exam management capabilities, failing to integrate automatic question generation, automated operations, and personalized feedback for examinees, while primarily focusing on exam supervision and grading functions.

Method used

A system utilizing AI to generate exam questions based on specified conditions, monitor for cheating, generate exam results, and provide customized learning materials, integrating an AI tutor for real-time explanations and feedback.

Benefits of technology

Automates exam processes, reduces administrative workload, enhances operational efficiency, and provides personalized learning support through real-time explanations and feedback.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure PAT00001_ABST
    Figure PAT00001_ABST
Patent Text Reader

Abstract

A system for providing a customized online exam management solution utilizing AI is provided, comprising: an administrator terminal that generates exam questions by specifying exam conditions including subject, scope, exam time, question type, difficulty, and number; a storage unit that stores subject content corresponding to at least one subject in a database; a question generation unit that generates exam questions corresponding to the exam conditions based on a pre-established AI (Artificial Intelligence) when exam conditions are input from the administrator terminal; a transmission unit that transmits the generated exam questions to the administrator terminal; a selection unit that receives a selection of exam questions from the administrator terminal; and an exam paper generation unit that generates an exam paper using the selected exam questions.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to a system for providing a customized online exam management solution utilizing AI, and provides a system that automatically generates exam questions, manages the progress of an examinee's exam, and provides customized learning materials based on the exam results. Background Technology

[0002] Recently, the use of online examination systems in the fields of education and assessment has expanded rapidly, leading to an increased demand for more efficient and reliable examination administration technologies. In particular, as the social demand for remote testing has grown since the COVID-19 pandemic, the digital transformation of various examination administration processes—such as proctoring, question development, and grade analysis—is accelerating. As the education and assessment environment rapidly transitions to digital, the demand for online examination systems is surging; furthermore, the widespread adoption of remote testing has highlighted the need for technological means capable of simultaneously ensuring both the reliability and efficiency of examination operations. While online examinations offer advantages such as overcoming spatial and temporal constraints and facilitating the management of large numbers of candidates, improvements are still required in various operational aspects.

[0003] At this time, methods for analyzing weaknesses based on the correct answer rate or enabling self-scoring based on real-time explanations have been researched and developed. In this regard, prior art Korean Registered Patent No. 10-2723232 (published on October 28, 2024) and Korean Published Patent No. 2023-0160655 (published on November 24, 2023) each disclose a configuration in which, when a question is generated from a question generation terminal, question information including a subject code, correct answer, keyword, and question type is received and stored, and when an answer is recognized, scoring is performed by comparing the answer with the correct answer, and when generating test results including test scores, the result is generated by including the subject code, correct answer, keyword, question type, and whether the answer is correct for at least one question; and a configuration in which test questions and correct answer association keywords are mapped and provided, and real-time instructor explanation streaming is provided, thereby enabling the user to score themselves.

[0004] However, the former does not provide a comprehensive exam management process from the perspective of the exam administrator, but merely offers a configuration that identifies the learner's weaknesses; similarly, the latter only offers a configuration that assists learners in arriving at the correct answers on their own, failing to provide a comprehensive exam management process from the administrator's standpoint. Most currently commercialized online exam solutions are limited to question provision and grading functions, and the utilization of artificial intelligence is concentrated solely on exam supervision features such as cheating detection. Consequently, functions for improving the work efficiency of exam administrators, enhancing learning convenience for examinees, and providing feedback remain insufficient. Therefore, research and development are required for an online exam management system that integrates automatic question generation, automated exam operations, and personalized feedback for examinees based on AI technology. The problem to be solved

[0005] One embodiment of the present invention provides a system for providing a customized online exam management solution utilizing AI, wherein, when exam conditions including subject, scope, exam time, type, difficulty, and number of questions are specified at an administrator terminal, exam questions are automatically generated and provided from a pre-established subject content database based on the specified exam conditions; logs and alarms are provided to the administrator terminal by automatically monitoring whether cheating occurs while ensuring the exam is conducted within the limited exam time at the examinee terminal; grades are generated based on the examinee terminal's answers to create exam results; an incorrect answer note is generated based on the generated exam results; and customized learning materials are provided to help supplement weak areas. However, the technical problem that this embodiment aims to solve is not limited to the technical problem described above, and other technical problems may exist. means of solving the problem

[0006] As a technical means for achieving the aforementioned technical task, one embodiment of the present invention comprises: an administrator terminal that generates test questions by specifying test conditions including a subject, scope, test time, type of question, difficulty level, and number of questions; a storage unit that stores subject content corresponding to at least one subject in a database; a question generation unit that generates test questions corresponding to test conditions based on a pre-established AI (Artificial Intelligence) when test conditions are input from the administrator terminal; a transmission unit that transmits the generated test questions to the administrator terminal; a selection unit that receives a selection of test questions from the administrator terminal; and a test paper generation unit that generates a test paper using the selected test questions. Effects of the invention

[0007] According to any one of the means for solving the problem of the present invention described above, a series of processes including question generation, test supervision, detection of cheating, generation of test results, and provision of incorrect answer notes can be automated using AI, and the repetitive workload of test administrators can be reduced and operational efficiency improved. Additionally, an AI tutor can be provided to test takers to support real-time explanations and Q&A during or after the test, and by presenting incorrect answer notes and customized learning materials based on test results, it can help maximize learning effects. Brief explanation of the drawing

[0008] FIG. 1 is a diagram illustrating a customized online test management solution providing system utilizing AI according to an embodiment of the present invention. Figure 2 is a block diagram illustrating a test management service providing server included in the system of Figure 1. FIGS. 3 and FIGS. 4 are drawings for explaining an embodiment in which a test management service according to an embodiment of the present invention is implemented. FIG. 5 is an operation flowchart illustrating a method for providing a test management service according to an embodiment of the present invention. Specific details for implementing the invention

[0009] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0010] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected" but also cases where they are "electrically connected" with other elements interposed between them. Furthermore, when a part is described as "including" a component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components, and it should be understood that this does not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0011] Terms such as “about,” “substantially,” etc., used throughout the specification, are used to mean at or near the stated value when inherent manufacturing and material tolerances are presented in the stated meaning, and are used to prevent unscrupulous infringers from unfairly exploiting the disclosure in which precise or absolute values ​​are mentioned to aid in understanding the invention. Terms such as “step” or “step of” used throughout the specification of the invention do not mean “step for”.

[0012] In this specification, the term "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more pieces of hardware, and two or more units may be realized by one piece of hardware. Meanwhile, "part" is not limited to software or hardware, and "part" may be configured to reside in an addressable storage medium or configured to run on one or more processors. Accordingly, as an example, "part" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and "parts" may be combined into a smaller number of components and "parts" or further separated into additional components and "parts." In addition, the components and '~parts' may be implemented to play one or more CPUs within the device or secure multimedia card.

[0013] Some of the operations or functions described herein as being performed by a terminal, device, or device may instead be performed by a server connected to said terminal, device, or device. Likewise, some of the operations or functions described as being performed by a server may also be performed by a terminal, device, or device connected to said server.

[0014] In this specification, some of the operations or functions described as mapping or matching with a terminal may be interpreted as meaning mapping or matching the terminal's unique number or personal identification information, which is the terminal's identifying data.

[0015] The present invention will be described in detail below with reference to the attached drawings.

[0016] FIG. 1 is a diagram illustrating a customized online exam management solution providing system utilizing AI according to an embodiment of the present invention. Referring to FIG. 1, the customized online exam management solution providing system utilizing AI (1) may include at least one administrator terminal (100), an exam management service providing server (300), and at least one examinee terminal (400). However, since the customized online exam management solution providing system utilizing AI (1) of FIG. 1 is merely an embodiment of the present invention, the present invention is not to be interpreted as being limited by FIG. 1.

[0017] At this time, each component of FIG. 1 is generally connected through a network (Network, 200). For example, as shown in FIG. 1, at least one administrator terminal (100) can be connected to an exam management service provider server (300) through the network (200). And, the exam management service provider server (300) can be connected to at least one administrator terminal (100) and at least one examinee terminal (400) through the network (200). Also, at least one examinee terminal (400) can be connected to the exam management service provider server (300) through the network (200).

[0018] Here, a network refers to a connection structure capable of exchanging information among individual nodes, such as multiple terminals and servers. Examples of such networks include Local Area Networks (LANs), Wide Area Networks (WANs), the World Wide Web (WWW), wired and wireless data networks, telephone networks, and wired and wireless television networks. Examples of wireless data communication networks include, but are not limited to, 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), 5GPP (5th Generation Partnership Project), 5G NR (New Radio), 6G (6th Generation of Cellular Networks), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth network, NFC (Near-Field Communication) network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, etc.

[0019] In the following, the term "at least one" is defined as a term including both singular and plural forms, and it will be obvious that even if the term "at least one" does not exist, each component may exist in a singular or plural form and may mean singular or plural. Furthermore, whether each component is provided in a singular or plural form may be changed according to the embodiment.

[0020] At least one administrator terminal (100) may be a terminal of an instructor or administrator who is a test creator, who generates test questions by inputting test conditions using a web page, app page, program, or application related to the test management service, outputs the results of monitoring cheating by the examinee terminal (400), and checks the test results.

[0021] Here, at least one administrator terminal (100) may be implemented as a computer capable of connecting to a remote server or terminal via a network. Here, the computer may include, for example, a navigation system, a laptop equipped with a web browser, a desktop, a laptop, etc. At this time, at least one administrator terminal (100) may be implemented as a terminal capable of connecting to a remote server or terminal via a network. At least one manager terminal (100) may include all kinds of handheld-based wireless communication devices, such as navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminal, smartphone, smartpad, tablet PC, etc.

[0022] The exam management service providing server (300) may be a server that provides an exam management service web page, app page, program, or application. Additionally, the exam management service providing server (300) may be a server that generates exam questions corresponding to the exam conditions entered by the administrator terminal (100) when it receives a request to generate exam questions from the administrator terminal (100). Furthermore, the exam management service providing server (300) may be a server that detects cheating by monitoring the examinee who is provided to at least one examinee terminal (400) and captured by the screen of the examinee terminal (400) and the webcam. Additionally, the exam management service providing server (300) may be a server that generates exam results for the examinee terminal (400) and provides incorrect answer notes and customized learning materials in response. Additionally, the exam management service providing server (300) may be a server that provides exam results to the administrator terminal (100).

[0023] Here, the test management service providing server (300) can be implemented as a computer capable of connecting to a remote server or terminal via a network. Here, the computer may include, for example, a navigation system, a laptop equipped with a web browser, a desktop, a laptop, etc.

[0024] At least one examinee terminal (400) may be a terminal of an examinee taking an online exam using a web page, app page, program, or application related to the exam management service. The examinee terminal (400) may be a terminal that prints out the exam paper, receives answers, and prints out the exam results.

[0025] Here, at least one candidate terminal (400) may be implemented as a computer capable of connecting to a remote server or terminal via a network. Here, the computer may include, for example, a navigation system, a laptop equipped with a web browser, a desktop, a laptop, etc. At this time, at least one candidate terminal (400) may be implemented as a terminal capable of connecting to a remote server or terminal via a network. At least one candidate terminal (400) may include all kinds of handheld-based wireless communication devices, such as navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminal, smartphone, smartpad, tablet PC, etc.

[0026] FIG. 2 is a block diagram for explaining a test management service providing server included in the system of FIG. 1, and FIG. 3 and FIG. 4 are drawings for explaining an embodiment in which a test management service according to an embodiment of the present invention is implemented.

[0027] Referring to FIG. 2, the test management service providing server (300) may include a storage unit (310), a question generation unit (320), a transmission unit (330), a selection unit (340), a test paper generation unit (350), a test supervision unit (360), a cheating detection unit (370), an explanation generation unit (380), and a result management unit (390).

[0028] When a test management service providing server (300) or another server (not shown) operating in conjunction with a test management service according to one embodiment of the present invention transmits a test management service application, program, app page, web page, etc. to at least one administrator terminal (100) and at least one candidate terminal (400), the at least one administrator terminal (100) and at least one candidate terminal (400) may install or open the test management service application, program, app page, web page, etc. Additionally, a service program may be operated on at least one administrator terminal (100) and at least one candidate terminal (400) using a script executed in a web browser. Here, a web browser refers to a program that enables the use of web (WWW: World Wide Web) services and receives and displays hypertext described in HTML (Hyper Text Mark-up Language), and includes, for example, Chrome, Microsoft Edge, Safari, Firefox, Whale, UC Browser, etc. In addition, "application" refers to an application on a terminal, and includes, for example, an app running on a mobile terminal (smartphone).

[0029] Referring to FIG. 2, the storage unit (310) can store subject content corresponding to at least one subject in a database. The subject content may include textbooks, reference books, and workbooks for each subject, but is not limited thereto. Here, the database may be constructed as a vector database, thereby enabling an LLM-based generative AI to generate problems or explanations based on it when generating problems or explanations later.

[0030] The problem generation unit (320) can generate test questions corresponding to the test conditions based on a pre-established AI (Artificial Intelligence) when test conditions are input from the administrator terminal (100). The administrator terminal (100) can specify test conditions including subject, scope, test time, type of question, difficulty level, and number. At this time, there may be various methods for generating questions, for example, as shown in FIG. 3 (b), the process may be [subject content analysis → extraction of key points → generation of question sentences → determination of question type → generation of test questions]. At this time, the process for each step may be as shown in Table 1 below.

[0031] order detail technology 1. Course Content Analysis - AI separates PDFs or text into sentences and divides them into meaningful units - Extracts learning points such as key concepts, facts, definitions, and explanations. - Sentence Segmentation - Named Entity Recognition (NER) - Dependency Parsing 2. Extracting Key Points & Generating Question Sentences Select sentences from the extracted sentences that are good for making questions. - Question Generation (QG) models (e.g., fine-tuned models such as T5, BART, GPT-3, KoGPT, etc.) 3. Determining Question Types & Generating Exam Questions - Automatically selects one of Multiple Choice, Short Answer, or True / False based on sentence content and context - Short Answer: Asking about concepts and definitions - Multiple Choice: Delete words or phrases → Automatically generate Distractor (incorrect answer) - True / False: Analyze true / false statements and modify them 4. Generating Answers and Explanations - After extracting the question and answer, explain why it is the correct answer and why other options are incorrect - GPT-type models can generate appropriate explanations - Combination of KoGPT + Retrieval-based training materials

[0032] Also, the models and tools available to implement the technology of Table 1 may be as shown in Table 2.

[0033] step Model / Tool explanation Course Content Analysis SpaCy, KoNLPy, HuggingFace Transformers Korean sentence analysis, syntax comprehension Extract key points BERTSum, TextRank, KeyBERT Summary / Key Concept Extraction Problem sentence generation T5, BART, KoGPT, GPT-4 Automatic question generation from sentences Determining Question Types & Generating Exam Questions Word2Vec, BERT-embedding, FastText Automatic generation of similar words or confusing concepts (Distractor) Generate Answers and Explanations GPT-3.5 / 4, KoGPT Automatic generation of explanations for the correct answer

[0034] In this context, "Distractor" refers to an incorrect answer, meaning that the AI ​​automatically generates potentially confusing incorrect answer choices in addition to the correct answer. To achieve this, the process involves: ① extracting concepts from the key sentence (the sentence based on the correct answer); ② identifying the topic and context; ③ generating incorrect answers using one of the [conceptual confusion type, category confusion type, or random information insertion type]; and ④ evaluating linguistic and semantic similarity. Here, the explanation assumes that the key sentence is, "ARGOS is an AI-based fraud detection solution developed by KT DS." ② When identifying the topic and context, the core concept is identified from the key sentence, and [similar but incorrect information] related to this concept can be found. This requires a foundation based on two key concepts: semantic similarity and context-based filtering.

[0035] At this point, to find similar but incorrect information, the process can be followed as follows: [Concept embedding of the correct answer → Selection of similar candidates → Verification of factual consistency (filtering of incorrect information) → Selection of only incorrect but plausible Distractors]. For the concept embedding of the correct answer, the AI ​​first converts the word of the correct answer in the problem into a vector form. For example, "ARGOS" is converted into a numerical vector such as [0.1, -0.3, 0.05, ...]. This vector numerically represents the meaning of the word; words that appear frequently in the same context tend to have similar vectors, and for this purpose, tools such as Word2Vec, FastText, BERT, KoBERT, and KLUE-BERT can be used. Next, similar candidate selection must be performed, and the AI ​​can extract words that are semantically similar to the correct answer from a pre-trained word / sentence vector space. For example, if the correct answer is "ARGOS," the AI ​​can extract similar words such as "MONITO," "AI-GUARD," "PROCTURE," and "SUPERVIEW." At this time, sorting can be done based on the similarity score (based on cosine similarity).

[0036] Next, incorrect information must be filtered out by verifying factual consistency; however, simply selecting similar words is insufficient, so context-based veracity must be determined. For example, assuming MONITO and ChatGPT were extracted from AI, MONITO is a factual statement because it is a semantically similar word, as it offers similar functionality to ARGOS but is from a different company. However, ChatGPT is a factual statement because, although it uses the semantically similar word "AI," it is not an exam proctoring system, which does not align with the prompt asking to select an exam proctoring system. Consequently, the passage for ChatGPT can be removed. This judgment can be performed using a Knowledge Graph, search-based fact-checking, or a fine-tuned QA model.

[0037] Next, the task is to select only the incorrect but plausible Distractors, leaving only those that are "not the correct answer but could be confusing" from the above process. Items that are incorrect but have some connection to the topic—for example, incorrect answers like "MONITO" or "AI-GUARD" in the ARGOS problem—can be ideal, so these can be kept and the rest removed. At this point, the key criteria for distinguishing incorrect but similar answers judged by the AI ​​may be as shown in Table 3 below, and the pseudocode for implementing this may be as shown in Table 4.

[0038] item explanation Semantic similarity Is it a word that has the same field / function as the correct answer? Topic relevance Are they in the same industry, technology, and role? Surface similarity Are the name structures, spellings, and pronunciations similar? Isn't it the correct answer? Finally, check if it differs from the correct answer.

[0039] from transformers import BertModel, BertTokenizerimport torch# Example: Getting the ARGOS vector tokenizer = BertTokenizer.from_pretrained("bert-base-multilingual-cased")model = BertModel.from_pretrained("bert-base-multilingual-cased")input_ids = tokenizer("ARGOS", return_tensors="pt")["input_ids"]outputs = model(input_ids)argos_vector = outputs.last_hidden_state.mean(dim=1)# Compare similarity with list of candidate wordscandidate_words = ["MONITO", "ChatGPT", "AI-GUARD"]for word in candidate_words: word_ids = tokenizer(word, return_tensors="pt")["input_ids"] word_vec = model(word_ids).last_hidden_state.mean(dim=1) similarity = torch.cosine_similarity(argos_vector, word_vec).item() print(f"{word} similarity: {similarity:.2f}")

[0040] If similar but incorrect information is found, ③ a Distractor is generated. To generate a Distractor of [Concept Confusion Type - Category Confusion Type - Random Information Insertion Type], the method shown in Table 5 below can be used.

[0041] Conceptually confusing type Distractor Example: "ARGOS" → Creation of products with similar names: -MONITO (actually similar system) -AGROS (spelling similarity) -AI-WATCHER (creation based on AI-related words) Category-confusing type Distractor If the question topic is 'AI exam proctoring solutions,' suggest names of other types of AI technologies—e.g., "ChatGPT," "TensorFlow"—in the same AI family but in different contexts. Insertion of irrelevant information Mixing contextually irrelevant names or concepts—e.g., "Google Docs," "Zoom," etc.—to confuse test-takers

[0042] Next, ④ linguistic and semantic similarity is evaluated, using Word2Vec, FastText, KoBERT, etc., to find words that are semantically similar to the correct answer. At this time, the GPT model can generate natural-sounding incorrect answers that match the overall tone of the sentence. For example, if the question is "What is the name of the AI-based cheating detection system developed by KT DS?" and the correct answer is "ARGOS," incorrect answers (Distractors) can be generated, for example, MONITO (another company product with similar functions), AI-GUARD (an AI-related name), and PROCTURE (a synonym for a term related to exam proctoring).

[0043] The transmission unit (330) can transmit the generated test questions to the administrator terminal (100). The administrator terminal (100) can output the generated test questions.

[0044] The selection unit (340) can receive test questions from the administrator terminal (100). Among the generated test questions, the administrator, for example, the question setter, can select the desired test questions. At this time, the generated test questions may be tagged with various data such as subject, difficulty level, and range, making it easier for the question setter to select the desired test questions. Additionally, by outputting the expected correct answer rate and expected average according to the difficulty level for each test question, the average can be matched with last year's or previous exams. For example, if the average for Subject A was 80 points in 2022, 82 points in 2023, and 81 points in 2024, the questions can be configured to aim for an expected average of 80 points this time. This is also referred to as Adaptive Item Generation or Targeted Assessment Design, and for this to work, metadata regarding the correct answer rate and difficulty level for each question must first be established.

[0045] Through this, the predicted accuracy rate is derived by predicting difficulty based on question length, word count, and concept count using past test-taking data (actual accuracy rates). It can also generate the predicted accuracy rate based on LLM-based inference, such as, "It is highly likely that only the top 30% will answer this question correctly." Furthermore, after setting a target average, problems can be combined using Greedy and constraint optimization algorithms. For instance, while combining problems, conditions such as [cumulative predicted average of 70 points ± margin of error], [securing diversity of question types], and [keyword balance / range diversity] can be satisfied. Additionally, prompts can be generated by linking with an LLM-based generative AI that creates problems. For instance, prompts such as "Generate problems with an accuracy rate of 0.3–0.4" can be provided.

[0046] The test paper generation unit (350) can generate a test paper with selected test questions. Once the test questions are generated in this way, the test questions can be placed in the format of the test paper. The test questions can be placed in the format of the test paper according to the school test format in the case of a midterm or final exam, according to the mock exam format in the case of a mock exam, or according to the quiz format in the case of a quiz.

[0047] The test supervisor (360) transmits the generated test paper to at least one registered examinee terminal (400), causes the test paper to be printed on at least one examinee terminal (400), and can terminate the test when the test time has elapsed after the test paper has been printed.

[0048] The cheating detection unit (370) can activate the webcam of at least one candidate terminal (400) to photograph the candidate and record the screen of at least one candidate terminal (400), perform abnormal behavior detection using the candidate and the screen as input, and then transmit the alarm and log of the abnormal behavior detection to the administrator terminal (100). For example, if the candidate looks elsewhere or if a GPT program or KakaoTalk is displayed on the screen, this can be recorded in a log and classified as abnormal behavior. This is because such behavior usually implies cheating.

[0049] Accordingly, the cheating detection unit (370) can receive video collected through a webcam / microphone via streaming, receive the examinee's face, eyes, posture, voice, etc. in real-time streaming, and perform abnormal behavior detection, which can be performed using computer vision-based detection or a deep learning model. In addition, since the judgment is not made based on a single source such as video, but can also be made based on a source such as text or a program displayed on the screen, it can be processed in a multimodal manner. The main detection elements and methods may be as shown in Table 6 below.

[0050] Detection element technology Example detection behavior Face recognition Face Detection (eg, MTCNN, MediaPipe) Disappears when the face is not visible except from the front Eye tracking Eye Tracking, Iris Position The act of constantly looking down / side Tracking the number of people Object Detection (YOLOv5, OpenCV) Cheating when two or more people are detected Screen capture detection JS-based event hooks, OS API hooking Screen capture attempt detection and warning Voice detection Speech Detection / Whisper Detect when someone speaks

[0051] Additionally, abnormal behavior detection can be performed using deep learning models, where CNN, RNN, or Transformer-based behavior analysis models can evaluate real-time frames. For example, body poses can be analyzed using OpenPose, or candidate behavior patterns can be analyzed using LSTM-based time-series analysis. If there is a possibility of cheating, the administrator is notified in real-time, logs (time, video snapshot, detection type) are recorded, and a warning (including pause and force quit options) can be displayed on the candidate's screen.

[0052] The explanation generation unit (380) can map and store the correct answer information that generated the test questions to the test questions when the AI ​​generates the test questions, and when generating the explanation for the test questions, it can present the correct answer information or construct the explanation using LLM (Large Language Model) Generative AI (Generative Artificial Intelligence) based on the correct answer information. When generating the explanation, methods such as the correct answer basis restoration type (Extractive), generative explanation (Generative), and option comparison type (Distractor analysis) can be used. For example, a sentence or paragraph corresponding to the correct answer can be extracted directly from the original text. For example, if the document states, “ARGOS is an AI-based cheating detection system,” that sentence can be used as the explanation.

[0053] Generative explanations involve GPT-based language models constructing explanations in natural language. For instance, if the explanation structure is, "This question asks about the concept of ○○, and the correct answer is △△. This is because ~," the model can be configured to generate an explanation such as, "ARGOS is an AI cheating detection solution developed by KT DS. It is a problem that can be easily solved if you only know the name." Additionally, the choice comparison type constructs explanations by refuting incorrect options one by one in a multiple-choice question. For instance, a method such as, "① is the correct answer. ② is an explanation of Monito, and ③ is incorrect. This is because it is a method that does not use AI," can be utilized.

[0054] purpose Technology / Model explanation Extract the correct answer BERT QA, Extractive QA Extract relevant sentences from the original text Description generation GPT-3.5, KoGPT, T5, BART Automatic generation of natural language explanations Error Analysis Text Classification + GPT Explanation after comparing the basis of Distractor Recommended study materials Grade-based recommendation system Provides similar questions for incorrect answers and key points

[0055] The result management unit (390) can perform grading based on the answers of at least one examinee terminal (400), provide test results to at least one examinee terminal (400), calculate grades based on the test results, and recommend an incorrect answer note and customized learning materials. For example, when a mouse is hovered over a test question, the AI ​​can provide a short hint or automatically save the incorrect question to the incorrect answer note and automatically configure [incorrect question + explanation + link to similar question]. In addition, voice explanations can be provided, for example, a function that reads the explanation using TTS (Text-to-Speech) technology can be added. Also, customized learning materials refer to materials that help prevent the student from making the same mistake again by clearly understanding the concept of the incorrect question and analyzing why it was wrong. An intelligent tutoring system (ITS) can be used for this purpose.

[0056] At this time, a language model (LM)-based framework can be provided to provide knowledge tracing (KT), a core element of ITS, and customized learning feedback. General KT (knowledge tracing) has limitations in that it relies on numerical-based information (e.g., problem ID, concept ID, etc.), fails to sufficiently reflect semantic information, is difficult to apply to new domains (cold start problem), and lacks interpretability. Accordingly, in one embodiment of the present invention, a Language Model-based Knowledge Tracing (LKT) model combining KT and LM is used to understand whether the user knows the content of the actual problem and how they solved it, thereby enabling the provision of practical customized content.

[0057] For example, assuming there are Q1. 2 + 3 = ? → Correct, Q2. 5 + 7 = ? → Incorrect, and Q3. 4 + 6 = ? → Correct, a standard KT predicts whether the user knows the concept based only on the question number and the record of correct or incorrect answers. However, LKT is a method in which the AI ​​understands and predicts not just the number or score, but also the actual text (content) of the problem and the sentence explaining how the user solved it. For example, if the user says, "Q2. 5 + 7 = ?" → "I thought that adding 2 to 5 gives 7" (including the user's explanation), the AI ​​can determine, "Aha, this student knows the concept of addition, but the concept of carrying is still lacking."

[0058] For LKT, a pretrained language model such as KoBERT can be used to include semantic-based information (Text) in the user's response history and estimate the probability of a future correct answer using a [MASK] token prediction method. In this case, estimating the probability of a future correct answer using the [MASK] token prediction method is a method in which a pretrained language model like KoBERT predicts the correct answer to a question as if filling in a blank; by estimating the probability that the user will answer the question correctly, it estimates the user's current state of knowledge. Using this, personalized questions can be recommended, conceptual understanding can be analyzed, and feedback can be provided against predicted incorrect answers. Furthermore, pretraining optimization techniques, such as DAPT (Domain-Adaptive Pre-Training) and TAPT (Task-Adaptive Pre-Training), can be used to fine-tune the system by domain and task to improve accuracy and adaptability.

[0059] Hereinafter, the operation process according to the configuration of the test management service providing server of FIG. 2 described above will be explained in detail with reference to FIG. 3 and FIG. 4. However, it is obvious that the embodiment is merely one of the various embodiments of the present invention and is not limited thereto.

[0060] Referring to FIG. 3, (a) the exam management service providing server (300) builds a database for subject content by subject, and when it receives exam conditions from the administrator terminal (100) as in (b), it generates exam questions corresponding to them. Then, the exam management service providing server (300) conducts exam supervision when the exam is conducted as in (c) and can (d) detect cheating. In addition, the exam management service providing server (300) can generate and provide explanations as in (a) of FIG. 4, and (b) provide incorrect answer notes and customized learning materials along with exam results. Furthermore, (c) it can provide the examinee with access to the same menu as the administrator so that they can create questions themselves and increase or decrease the difficulty level while studying, and (d) it can provide a community to discuss questions and provide correct answer rate statistics so that the difficulty level can be recognized by checking how many people answered correctly or incorrectly.

[0061] As for the test management service provision method shown in FIGS. 2 to 4, any details not described are identical to or can be easily inferred from the description of the test management service provision method shown in FIG. 1, so further explanation will be omitted.

[0062] FIG. 5 is a diagram illustrating the process of data transmission and reception between each component included in the system for providing a customized online test management solution utilizing AI according to one embodiment of the present invention. Hereinafter, an example of the process of data transmission and reception between each component will be described through FIG. 5, but the present invention is not to be interpreted as being limited to such an embodiment, and it is obvious to those skilled in the art that the process of data transmission and reception shown in FIG. 5 may be changed according to various embodiments described above.

[0063] Referring to FIG. 5, the test management service providing server stores subject content corresponding to at least one subject in a database (S5100).

[0064] And, when test conditions are entered from an administrator terminal, the test management service provider generates test questions corresponding to the test conditions based on a pre-established AI (Artificial Intelligence) (S5200).

[0065] Additionally, the test management service provider server transmits the generated test questions to the administrator terminal (S5300) and receives a selection of test questions from the administrator terminal (S5400). Then, the test management service provider server generates a test paper with the selected test questions (S5500).

[0066] The order of the steps described above (S5100~S5500) is merely an example and is not limited thereto. That is, the order of the steps described above (S5100~S5500) may vary, and some of these steps may be executed simultaneously or deleted.

[0067] As for matters not described regarding the test management service provision method of Fig. 5, they are identical to or can be easily inferred from the description of the test management service provision method described in Figs. 1 to 4 above, so further explanation will be omitted.

[0068] The method for providing a test management service according to one embodiment described through FIG. 5 may also be implemented in the form of a recording medium containing computer-executable instructions, such as an application or program module executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, as well as removable and non-removable media. Additionally, a computer-readable medium may include all computer storage media. Computer storage media include both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data.

[0069] The method for providing a test management service according to one embodiment of the present invention described above may be executed by an application basically installed on a terminal (which may include a program included in a platform or operating system, etc., basically installed on the terminal), or by an application (i.e., a program) directly installed by a user on a master terminal through an application providing server, such as an application store server, an application, or a web server related to the service. In this sense, the method for providing a test management service according to one embodiment of the present invention described above may be implemented as an application (i.e., a program) that is basically installed on a terminal or directly installed by a user, and may be recorded on a computer-readable recording medium such as a terminal.

[0070] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0071] The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present invention.

Claims

Claim 1 A system for providing a customized online exam management solution utilizing AI, comprising: an administrator terminal for generating exam questions by specifying exam conditions including subject, scope, exam time, type of question, difficulty, and number of questions; and an exam management service providing server including a storage unit for storing subject content corresponding to at least one subject in a database, a question generation unit for generating exam questions corresponding to the exam conditions based on a pre-established AI (Artificial Intelligence) when the exam conditions are input from the administrator terminal, a transmission unit for transmitting the generated exam questions to the administrator terminal, a selection unit for receiving a selection of exam questions from the administrator terminal, and an exam paper generation unit for generating an exam paper with the selected exam questions. Claim 2 A customized online exam management solution providing system utilizing AI, characterized in that, in claim 1, the exam management service providing server further includes an exam proctor that transmits a generated exam paper to at least one registered examinee terminal, causes the exam paper to be printed on the at least one examinee terminal, and performs a termination process when the exam time elapses after the exam paper is printed. Claim 3 A customized online exam management solution providing system utilizing AI, characterized in that, in claim 1, the exam management service providing server further comprises a cheating detection unit that activates a webcam of at least one examinee terminal to photograph the examinee, records the screen of at least one examinee terminal, performs abnormal behavior detection using the examinee and the screen as input, and then transmits an alarm and log of the abnormal behavior detection to an administrator terminal. Claim 4 A customized online exam management solution providing system utilizing AI, characterized in that, in claim 1, the exam management service providing server further includes an explanation generation unit that, when generating the exam question in the AI, maps and stores the correct answer information that generated the exam question to the exam question, and when generating the explanation for the exam question, presents the correct answer information or constructs the explanation using a Large Language Model (LM) Generative Artificial Intelligence (Generative AI) based on the correct answer information. Claim 5 A customized online exam management solution providing system utilizing AI, characterized in that, in claim 2, the exam management service providing server further includes a result management unit that performs scoring based on the answers of at least one examinee terminal, provides exam results to at least one examinee terminal, calculates grades based on the exam results, and recommends incorrect answer notes and customized learning materials.