Methods and systems for generating scoring instructions for automated scoring
The method and system generate and refine scoring instructions using a large-scale language model to address inconsistent essay scoring, ensuring consistent and adaptable automated evaluation.
Patent Information
- Application Number
- KR1020250060464
- Authority / Receiving Office
- KR · KR
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-09
- Publication Date
- 2026-11-17
AI Technical Summary
Existing automated scoring systems for essay-type responses are limited by the need for manual setting of scoring criteria, lack of adaptability to varied expression styles, and inability to automatically improve criteria, leading to inconsistent evaluations.
A method and system that utilizes a large-scale language model to generate, improve, and determine optimal scoring instructions based on essay-type question-and-answer data, adjusting criteria automatically to ensure consistent and accurate scoring.
Ensures consistent scoring results across varied responses, maintains scoring quality without human intervention, and adapts to different input data types, improving efficiency and scalability.
Smart Images

Figure PAT00001_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a method and system for generating instructions for automatic scoring. Background Technology
[0002] Recently, online learning platforms such as Coursera and K-MOOC are being widely utilized, and the adoption of classes using smart devices and AI-based digital textbooks is also spreading in schools. Due to these changes, education is shifting toward a non-face-to-face model, and there is a growing demand for individualized education and assessment tailored to students' levels and characteristics.
[0003] In particular, for essay-type questions where students articulate their thoughts in writing, there are limitations to automatic machine evaluation due to the wide variety of response lengths, sentence structures, and modes of expression. Furthermore, manually reading and evaluating multiple responses takes a significant amount of time, and inconsistencies in evaluation results may arise as different individuals may apply different grading criteria.
[0004] However, existing automated scoring systems are structured so that humans manually set scoring criteria and evaluate answers based on those standards. This approach suffers from the problem that setting criteria is time-consuming and the established standards are not appropriately applied to all types of answers. Furthermore, there are no procedures in place to automatically adjust or improve criteria even if they are inappropriate.
[0005] Accordingly, there is a need for technology capable of automatically evaluating essay-type responses written in various ways, as well as a need for methods to automatically set or determine scoring criteria appropriate to the situation.
[0006] Therefore, there is a need for a method and system for generating instructions for automated scoring that can fairly evaluate responses according to certain standards even if the content differs, and automatically improve them if the standards are inappropriate. The problem to be solved
[0007] The present invention relates to a method and system for generating instructions for automatic grading capable of automatically grading essay-type answers.
[0008] More specifically, the present invention aims to provide a method and system for generating instructions for automatic scoring that can generate instructions capable of consistently evaluating essay-type responses with various expression styles according to certain criteria.
[0009] Furthermore, the present invention aims to provide a method and system for generating instructions for automatic scoring that can automatically generate instructions suitable for scoring criteria based on user responses and assigned scores, and identify and determine instructions with superior performance.
[0010] Furthermore, the present invention aims to provide a method and system for generating instructions for automatic grading that can improve instructions based on the difference between the grading result and the assigned score, or automatically adjust grading criteria without teacher intervention. means of solving the problem
[0011] To solve the problem described above, the method for generating instructions for automatic scoring according to the present invention may include the steps of: receiving essay-type question-and-answer data configured such that at least one question to which a score is assigned and an essay-type response to said question form a pair; generating a plurality of scoring instructions using said essay-type question-and-answer data; scoring the essay-type response using said plurality of scoring instructions and calculating a scoring result for said essay-type response; improving said plurality of scoring instructions based on said calculated scoring result to generate an improved plurality of scoring instructions; and determining an optimal scoring instruction among said improved plurality of scoring instructions that satisfies a preset criterion.
[0012] Furthermore, in the step of generating the plurality of scoring instructions, the essay-type question-and-answer data may be input into a large-scale language model (LLM), and the large-scale language model may generate the plurality of scoring instructions including scoring criteria related to the essay-type question-and-answer data.
[0013] Furthermore, the scoring criteria related to the essay-type question-and-answer data include scoring criteria for each of the plurality of scoring instructions, and in the step of calculating the scoring result, the essay-type response can be scored based on the scoring criteria for each of the plurality of scoring instructions, and the scoring result for the essay-type response can be calculated.
[0014] Furthermore, the degree of consistency between the above scoring result and the score assigned to the above essay-type question-and-answer data can be calculated, and the above multiple scoring instructions can be improved based on the calculated degree of consistency to generate the above improved multiple scoring instructions.
[0015] Furthermore, the method may further include a step of determining whether there is an error in the scoring instruction by comparing the above-mentioned consistency with a preset threshold value.
[0016] Furthermore, if the degree of consistency is below a preset threshold value as a result of the above judgment, it can be determined that the scoring instruction contains an error.
[0017] Furthermore, it may further include a step of adjusting the scoring criteria of the scoring instructions containing the above error.
[0018] Furthermore, in the step of generating the above-mentioned improved plurality of scoring instructions, the above-mentioned improved plurality of scoring instructions can be generated using the above-mentioned adjusted scoring criteria.
[0019] Furthermore, the step of determining the optimal scoring instructions may further include the step of scoring the essay-type response using the improved plurality of scoring instructions and the step of calculating the scoring result for the essay-type response.
[0020] Furthermore, it may further include a step of calculating the degree of consistency between the above scoring result and the above assigned score.
[0021] Furthermore, by comparing the accuracy of the calculated accuracy, the scoring instruction with the highest accuracy among the improved multiple scoring instructions can be determined as the optimal scoring instruction.
[0022] Furthermore, the method may further include the step of grading at least one specific essay-type response using the optimal grading instructions determined above, and the step of calculating a grading result for the specific essay-type response.
[0023] Furthermore, the method may further include a step of comparing the difference between the improved plurality of scoring instructions and the existing scoring instructions, and a step of storing information including at least one of change information and adjustment details of the scoring criteria based on the comparison result.
[0024] Meanwhile, the instruction generation system for automatic scoring according to the present invention includes an input unit that receives essay-type question-and-answer data configured such that at least one question to which a score is assigned and an essay-type response to the question form a pair; a large-scale language model that generates a plurality of scoring instructions using the essay-type question-and-answer data; and a control unit that scores the essay-type response using the plurality of scoring instructions and calculates a scoring result for the essay-type response. The control unit can improve the plurality of scoring instructions based on the calculated scoring result to generate improved plurality of scoring instructions, and determine an optimal scoring instruction among the improved plurality of scoring instructions that satisfies a preset criterion.
[0025] Meanwhile, a program that can be stored on a computer-readable recording medium according to the present invention is executed by one or more processes in an electronic device and is a program stored on a computer-readable recording medium, wherein the program may include instructions for performing the steps of: receiving essay-type question-and-answer data configured such that at least one question to which a score is assigned and an essay-type response to the question form pairs; generating a plurality of scoring instructions using the essay-type question-and-answer data; scoring the essay-type response using the plurality of scoring instructions and calculating a scoring result for the essay-type response; improving the plurality of scoring instructions based on the calculated scoring result to generate an improved plurality of scoring instructions; and determining an optimal scoring instruction among the improved plurality of scoring instructions that satisfies a preset criterion. Effects of the invention
[0026] As described above, the method and system for generating instructions for automatic scoring according to the present invention can generate a plurality of scoring instructions and determine the optimal instruction among them. Through this, even if essay-type responses to the same question are written in various ways, scores can be calculated by applying the criterion with the highest evaluation accuracy, thereby ensuring consistency in scoring results. Furthermore, by the system independently determining appropriate criteria without evaluator intervention, the overall scoring quality can be stably maintained, and a reliable automatic evaluation environment can be realized.
[0027] In addition, the method and system for generating instructions for automatic scoring according to the present invention can improve instructions by comparing the difference between the scoring result and the assigned score and reflecting it. Through this, the model learns from previous errors and does not repeat them, and instructions with low performance are excluded while only instructions close to the actual scoring criteria remain, so that the system can be structured to improve itself.
[0028] Furthermore, the method and system for generating instructions for automatic scoring according to the present invention can automatically adjust scoring criteria and reflect them in instructions. This eliminates the need for a person to repeatedly modify or review the criteria, and allows for effective response to various types of essay-type evaluation situations. Additionally, since the criteria automatically adapt even if the input data or question type changes, they can be utilized in diverse environments without being fixed to a specific situation.
[0029] Furthermore, the method and system for generating instructions for automatic scoring according to the present invention can re-verify the performance of instructions by re-scoring based on adjusted instructions. Through this, the excellence of instructions can be quantified and measured by repeatedly verifying actual scoring results rather than through simple comparison, and standards can be determined based on objective grounds.
[0030] Furthermore, the method and system for generating instructions for automatic scoring according to the present invention can generate instructions by utilizing essay-type responses and assigned scores together. Through this, the direction of judgment can be learned by referring to actual human scoring results, and as a result, natural scoring criteria similar to the flow of human evaluation can be automatically configured.
[0031] Furthermore, the method and system for generating instructions for automatic scoring according to the present invention can automatically generate scoring instructions. Through this, scoring criteria can be quickly generated or adjusted for various evaluation items without the need for a person to design the criteria directly, thereby significantly improving the efficiency and scalability of the scoring system. Brief explanation of the drawing
[0032] FIGS. 1 and FIGS. 2 are conceptual diagrams for explaining an instruction generation system for automatic scoring according to the present invention. FIG. 3 is a flowchart illustrating a method for generating instructions for automatic scoring according to the present invention. FIGS. 4, FIGS. 5, FIGS. 6, FIGS. 7, FIGS. 8 and FIGS. 9 are conceptual diagrams for explaining a method for generating instructions for automatic scoring according to the present invention. Specific details for implementing the invention
[0033] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components are assigned the same reference number regardless of the drawing symbols, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.
[0034] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.
[0035] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.
[0036] Singular expressions include plural expressions unless the context clearly indicates otherwise.
[0037] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.
[0038] The present invention relates to a method and system for generating instructions for automatic scoring. More specifically, the present invention relates to a method and system for generating automatic scoring instructions that can simultaneously automate essay-type evaluation, improve evaluation reliability, and improve scoring efficiency by enabling the generation of scoring instructions for evaluating essay-type responses without a person having to directly set the scoring instructions, and by automatically determining and improving the optimal scoring instructions.
[0039] Meanwhile, the method and system for generating instructions for automatic scoring according to the present invention can be utilized to automatically evaluate essay-type exams in educational settings and can be applied to various environments such as online learning platforms, AI tutor systems, and edutech services.
[0040] In addition, by automatically generating appropriate criteria based on user response patterns or educational objectives and iteratively improving them, it can also be utilized for purposes such as providing customized feedback, automatically analyzing achievement, and securing fair evaluation standards.
[0041] As such, the present invention is a technology capable of simultaneously achieving the automation of essay-type assessments, the refinement of assessment criteria, and the autonomy of the scoring process based on the ability to generate and evaluate large-scale language models, and can be usefully utilized in various education and assessment service fields.
[0042] Hereinafter, the present invention will be examined in more detail with reference to the attached drawings. FIGS. 1 and 2 are conceptual diagrams for explaining an instruction generation system for automatic scoring according to the present invention, FIG. 3 is a flowchart for explaining an instruction generation method for automatic scoring according to the present invention, and FIGS. 4, 5, 6, 7, 8 and 9 are conceptual diagrams for explaining an instruction generation method for automatic scoring according to the present invention.
[0043] Meanwhile, as illustrated in FIG. 1, the instruction generation system for automatic scoring according to the present invention (hereinafter referred to as the “instruction generation system,” 100) may include at least one of an input unit (110), an output unit (120), a storage unit (130), a control unit (140), and a large-scale language model (150).
[0044] The instruction generation system (100) according to the present invention may include at least one processor and at least one memory containing computer program code. In this case, the memory may perform the role of the storage unit (130) above. In the present invention, the memory and the program code may cooperate with the processor to perform a series of processes described below.
[0045] Although not illustrated, the instruction generation system (100) according to the present invention may include one or more processors, and such processors may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., digital signal processor, tensor processing unit (TPU), graphics processing unit (GPU), neural network processing unit (NPU), application integrated circuit, application semiconductor (ASIC), field programmable gate array (FPGA), quantum processing unit (or quantum processor, QPU), etc.). One or more processors may be configured to execute instructions, computer-readable instructions, and / or other instructions described herein that are stored (or included) in the storage unit (130). The instruction generation system (100) according to the present invention may perform data processing described below in cooperation with memory and at least one processor. The processor may perform a series of operations and data processing using data and information stored in memory. In this case, memory may be a component of the storage unit (130).
[0046] In addition, the instruction generation system (100) according to the present invention can be implemented in a classical computer environment capable of large-scale language model-based computation, and can automate the entire process of instruction generation, scoring, evaluation, refinement, and determination of optimal instructions by utilizing a deep learning framework (PyTorch, TensorFlow, etc.) and a natural language processing library (e.g., HuggingFace Transformers, etc.).
[0047] For example, the present invention can improve the inference processing speed of large-scale language models by utilizing parallel computing devices such as GPUs or TPUs, and can also support a distributed learning environment for model parameter optimization and performance evaluation.
[0048] Additionally, the implementation system (100) of the present invention may be linked via a RESTful API or a cloud-based microservice structure and integrated with an external evaluation system, an LMS (Learning Management System), or an online test platform.
[0049] Meanwhile, the input unit (110) can be configured in various ways as a means of data input. For example, the input unit (110) can be configured to receive user input. The input unit (110) can be configured to receive user input from a user terminal (10). Here, “receiving input” may mean receiving an input signal (or decision signal) corresponding to the user’s input based on input made by the user through the input unit configuration provided in the user terminal (10).
[0050] The user terminal (10) may include at least one of a mobile phone, a smartphone, a notebook computer, a laptop computer, a slate PC, a tablet PC, an ultrabook, a desktop computer, a digital broadcasting terminal, a PDA (personal digital assistants), a PMP (portable multimedia player), a navigation device, and a wearable device (e.g., a smartwatch, a smart glass, a head-mounted display).
[0051] In addition, the input unit (110) in the present invention does not necessarily mean a hardware means, but can be understood as a channel for receiving input from a user.
[0052] The input unit (110) may also be referred to as a user interface module. The input unit (110) may include a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, or other similar devices. However, the present invention does not limit the type of input unit (110).
[0053] Here, user input may include documents, text, images (or videos), voice, etc. In this case, the instruction generation system (100) may further include a module that converts voice into text.
[0054] Next, the output unit (120) can output information through an output unit configuration (e.g., a display unit, a touch screen, a speaker, etc.) provided in a user terminal linked to the instruction generation system (100) according to the present invention. For example, the output unit (120) can output a page (or service page) linked to the instruction generation system (100) according to the present invention to the display unit of the user terminal. Additionally, the output unit (120) does not necessarily mean a hardware means, but can be understood as a channel for outputting results to a user.
[0055] Next, the storage unit (130, or memory) serves to store various data related to the present invention and may include one or more non-transient computer-readable storage media that can be read and / or accessed by at least one of one or more processors.
[0056] One or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage devices. In some examples, the storage unit (130) may be implemented using a single physical device (e.g., one optical, magnetic, organic, or other memory or disk storage device), whereas in other examples, the storage unit (130) may be implemented using two or more physical devices.
[0057] The storage unit (130) may include computer-readable instructions and additional data. The storage unit (130) may include a storage necessary to perform at least some of the methods, scenarios, and techniques described herein and / or at least some of the functions of the device and network.
[0058] Furthermore, at least a portion of the storage unit (130) may be a cloud storage or a cloud server. The storage unit (130) may store at least some of the essay-type question-and-answer data corresponding to user input received from the input unit (110), assigned scores and response information, generated multiple scoring instructions, scoring results, and performance evaluations.
[0059] That is, the storage unit (130) is sufficient as a space where information necessary for the operation of the instruction generation system (100) according to the present invention is stored, and it can be understood that there are no restrictions on the physical space.
[0060] Furthermore, the storage unit (130) can store a computer program including computer program instructions. Furthermore, the storage unit (130) can store a computer program including computer program instructions that control the operation of the system (100) or control the operation of the control unit (140) when loaded into the processor of the system (100).
[0061] Next, the control unit (140) can perform the role of controlling the overall operation of the instruction generation system (100) related to the present invention. The control unit (140) can process signals, data, information, etc. that are input or output through the components of the instruction generation system (100) described above, or perform a series of data processing to provide or process appropriate information and functions to the user. The control unit (140) can be physically implemented by the processor described above.
[0062] Meanwhile, the “Large Language Model (LLM, 150)” may be named “Language Model (150)” and may be a model that has been pre-trained (or already trained) to process natural language based on essay-type question-and-answer data (200) and generate multiple scoring instructions.
[0063] The large-scale language model (150) may include a Transformer structure, in which case it may include at least one of positional encoding, multi-head attention, and a feed-forward network.
[0064] The Transformer structure can be used to analyze the relationship between the questions and answers of the input essay-type question-and-answer data (200) in parallel, and to interpret the interrelationships and semantic structures between vocabulary within the sentences to extract or construct information corresponding to evaluation criteria.
[0065] The large-scale language model (150) may include a Mixture of Experts (MoE) structure. In this case, it may include multiple sub-models that are independently trained to process different types of scoring criteria for the same input data.
[0066] Each sub-model may be pre-trained based on specific evaluation criteria, such as determining whether there are grammatical errors, evaluating topic relevance, and analyzing sentence composition completeness, and the Gating Network may use the output of one or more sub-models in calculations according to pre-defined rules or probability distributions among them, depending on the type or characteristics of the input essay-type question-answer data (200).
[0067] At this time, the gate network can calculate the activation ratio of each sub-model according to the Softmax function or a predefined rule, and derive the final output by weighting and summing each output value.
[0068] A large-scale language model (150) can be trained or tuned based on at least one of supervised learning, reinforcement learning, and supervised fine-tuning, in addition to prior training.
[0069] In addition, the above-mentioned large-scale language model can be processed on a single computing device, and if the scale of computation is large, it can be trained or executed by being distributed across multiple computing devices (such as GPUs) through a pipeline parallelism structure.
[0070] In this way, the large-scale language model (150) may be configured to include sub-models corresponding to multiple scoring criteria, and may generate, adjust, evaluate, and determine scoring instructions by reflecting only information related to specific evaluation items according to the characteristics of the input essay-type question-and-answer data (200).
[0071] More specifically, the entire processing process of the instruction generation system (100) according to the present invention will be explained with reference to FIG. 2.
[0072] The essay-type question-and-answer data (200) received through the input unit (110) may include at least one of a question, an answer, and an assigned score, and within the instruction generation system (100), the essay-type question-and-answer data (200) may be divided into three types of data according to purpose. Specifically, it may be divided into data for composing a few-shot example, data for scoring an initial instruction, and data for scoring an adjusted instruction, and stored in the storage unit (130).
[0073] A large-scale language model (150) can combine some of the stored data (questions, answers, assigned scores) to form a few-shot example, and use the example along with essay-type question-and-answer data (200) as input values to automatically generate multiple scoring instructions.
[0074] Each generated scoring instruction is applied to essay-type question-and-answer data (200) to calculate a score for the response, and the result can be stored in the storage unit (130). Subsequently, each scoring result can be compared with the assigned score to evaluate the scoring performance of the corresponding scoring instruction. At this time, a degree of consistency (quantitative indicators such as QWK) is calculated, and all scoring instructions can be classified into a Scored Instructions state with a performance score assigned.
[0075] The control unit (140) can adjust the scoring criteria included in each scoring text based on essay-type question-and-answer data (200) containing errors between the scoring results and the assigned scores. At this time, the characteristics of the response in which the error occurred can be analyzed through contrastive example learning or contrastive example-based learning, and the scoring instructions can be adjusted by reflecting this. The control unit (140) can improve the scoring instructions by reflecting the relevant information so as not to repeat the error, and classify the improved scoring instructions as 'Refined Instructions'.
[0076] Each of the adjusted multiple scoring instructions is applied again to the essay-type question-and-answer data (200) to score the response, and the degree of alignment between the corresponding scoring result and the assigned score can be calculated. At this time, the control unit (140) can compare the degree of alignment with the final scoring result for each scoring instruction and determine the instruction with the highest degree of alignment as the optimal scoring instruction (300). The optimal scoring instruction (300) can finally be provided externally through the output unit (120) or used for scoring.
[0077] Through this, the instruction generation system (100) according to the present invention automates everything from initial instruction generation to grading performance evaluation, error-based instruction improvement, and final optimal instruction determination without user intervention, and can perform grading of essay-type answers precisely and consistently. Below, we will examine the instruction generation method according to the present invention in more detail.
[0078] Meanwhile, in the present invention, a process of receiving essay-type question-and-answer data configured such that at least one question with a score and an essay-type response to the question form a pair (S310, see FIG. 3).
[0079] The control unit (140) can receive essay-type question-and-answer data configured such that at least one question and an essay-type response to the question are paired and assigned a score through the input unit (110).
[0080] Here, the essay-type question-and-answer data may be data including at least one of a question, an essay-type response to the question, and a score assigned, and may be various forms of data that a user inputs into the system through an input unit (110) to perform scoring and evaluation.
[0081] Specifically, “essay-type question-and-answer data” may include a question, an essay-type response to the question, and a score assigned to the essay-type response according to pre-established criteria.
[0082] Here, “question” refers to an essay-type evaluation item that may be named as “inquiry,” “item,” or “problem,” and may be an item presented by the user to the subject of evaluation for evaluation purposes.
[0083] “Essay-type response” may also be referred to as “essay-type answer,” and is an answer submitted by the subject to a question, which may be an essay-type response written in text form.
[0084] “Scores assigned according to pre-established criteria” may refer to numerical evaluation values directly assigned by a person (e.g., teacher, evaluator, expert, etc.) to each essay-type response included in the essay-type question-and-answer data, by referring to predetermined evaluation criteria or guidelines.
[0085] At this time, the assigned score may be assigned by reflecting the score distribution for each scoring criterion item, the reflection of weights, and a comprehensive judgment based on qualitative elements of the response (e.g., logic, creativity, grammatical accuracy, etc.), and may be a value assigned according to internal or external evaluation guidelines to ensure consistency and fairness in the evaluation. Furthermore, this score may be used as a reference score (ground truth score) or verification criterion (reference score) as reference data used to calculate the degree of consistency (e.g., agreement, error rate, etc.) by comparing it with the scoring results produced by the instruction generation system (100) thereafter.
[0086] The input unit (110) is an interface that transmits user input data into the system and can receive data via keyboard input, file upload, or network connection. For example, an evaluator (or user) may write a response to an essay-type question included in the essay-type question-and-answer data, save it in file form, and upload it, or directly input data through a separate user interface (UI), and the control unit (140) can receive this through the input unit (110).
[0087] The input unit (110) can automatically receive essay-type question-and-answer data from a remote server or external database (DB) through various communication networks, and can also receive data through various methods such as optical character recognition (OCR), speech recognition, and image recognition.
[0088] The received essay-type question-and-answer data can be stored in a storage unit (130), and the storage unit (130) can be composed of various storage devices inside and / or outside the system, such as a database, memory, or cloud storage. The control unit (140) can store the essay-type question-and-answer data received through the input unit (110) in the storage unit (130).
[0089] At this time, the essay-type question-and-answer data may be stored separately according to the purpose of use. For example, some of the received essay-type question-and-answer data may be used to construct a few-shot example for generating multiple scoring instructions in a large-scale language model (150), while the rest may be stored separately as data for different purposes, such as for calculating the consistency of scoring instructions or improving scoring instructions.
[0090] When the control unit (140) receives essay-type question-and-answer data through the input unit (110), it can verify the integrity of the data or perform pre-processing. For example, if the received essay-type question-and-answer data differs from a pre-set format and structure (e.g., JSON structure, text field, etc.) or if essential input information is missing, the control unit (140) can request the user terminal to re-enter the data or supplement some of the data.
[0091] Meanwhile, in the present invention, a process of generating multiple scoring instructions using essay-type question-and-answer data may be carried out (S320, see FIG. 3).
[0092] The control unit (140) can generate multiple scoring instructions using a large-scale language model (150) based on essay-type question-and-answer data received through the input unit (110).
[0093] Here, the scoring instructions may be text-based data containing scoring criteria applied when evaluating essay responses included in essay-type question-and-answer data, and specific and explicit guidelines or explanations for applying those criteria to the evaluation.
[0094] The large language model (150) may be a pre-trained large language model (LLM), and in this specification, a language model based on a transformer structure is described as an example. In addition, various model structures such as Mixture of Experts (MoE), Recurrent Neural Network (RNN), or combinations or modifications thereof may be used.
[0095] The control unit (140) can generate multiple scoring instructions in a few-shot manner by inputting some of the essay-type question-and-answer data stored in the storage unit (130) into a large-scale language model (150).
[0096] A few-shot method may mean a method that can induce the generation of scoring instructions using a small amount of example data without performing additional training (fine-tuning) on a large-scale language model (150).
[0097] The control unit (140) can process example data pairs (e.g., question-answer-assigned score) including a question, an essay-answer, and an assigned score corresponding to the answer among essay-type question-answer data as input to the large-scale language model (150) in order to construct a few-shot example to be input to the large-scale language model (150).
[0098] At this time, the control unit (140) may adjust the configuration of the input data or add a prompt to pre-set the structure of the scoring instructions generated by the large-scale language model (150). For example, the control unit (140) may input an imperative sentence, such as “Write a scoring instruction to evaluate the response to the next question,” into the large-scale language model (150) along with example data, so that the large-scale language model (150) generates a scoring instruction.
[0099] A large-scale language model (150) can receive these few-shot example data and prompts as input, analyze them, and generate multiple scoring instructions based on the analysis results. At this time, each scoring instruction generated may include scoring criteria for evaluating the essay-type response to be scored.
[0100] Scoring criteria may include specific guidelines regarding evaluation items and scoring methods. For example, they may include each item of the evaluation criteria, such as accuracy, logic, grammatical accuracy, appropriateness, and clarity of essay-type responses, or include detailed guidelines on specific evaluation methods (e.g., methods for checking the presence of key keywords in the response, explanations of scoring methods, etc.).
[0101] In one embodiment, as illustrated in FIG. 4, the essay-type question-and-answer data (200) received through the input unit (110) may be composed of a structure such as “Question: Describe a place you visited that left a deep impression” and a response to it, “I visited Gyeongju, and the historical sites were very impressive.” Additionally, a pre-evaluated score of 5.5 may be provided for the response.
[0102] When such essay-type question-and-answer data (200) is input into a large-scale language model (150), the large-scale language model (150) can generate multiple scoring instructions that include different scoring criteria, such as “scoring based on grammatical accuracy and vocabulary diversity,” “consistency of response, relevance, and application of grammatical criteria,” and “evaluation based on consistency with the question and content fidelity.” The multiple scoring instructions can each be generated to include different criteria items or scoring perspectives.
[0103] For convenience of explanation, the present specification provides an example in which a large-scale language model (150) receives data in text format and generates scoring instructions in text format, but the present invention is not limited thereto and may be provided in various input / output formats, such as converting non-text data, such as images or voice, into text data and inputting it into the large-scale language model (150), or converting generated scoring instructions into voice and providing them.
[0104] The control unit (140) can store a plurality of generated scoring instructions in the storage unit (130) respectively.
[0105] Meanwhile, in the present invention, a process of scoring an essay-type response using a plurality of scoring instructions and calculating a scoring result for the essay-type response may be carried out (S330, see FIG. 3).
[0106] The control unit (140) can score essay-type question-and-answer data using each of the multiple scoring instructions and produce a scoring result accordingly.
[0107] “Scoring” refers to evaluating essay responses included in received essay-type question-and-answer data and quantifying and calculating, in the form of a score, how well the response meets pre-set scoring criteria.
[0108] The control unit (140) can apply the scoring criteria included in each of the multiple scoring instructions to the response of the essay-type question-and-answer data during the scoring process, and express the degree to which the response meets the criteria as a numerical score.
[0109] Each of the multiple scoring instructions may include different evaluation criteria. For example, a specific scoring instruction may include criteria for evaluating the logicality of the response, while other scoring instructions may include various criteria individually, such as the accuracy, creativity, grammatical completeness, presence or absence of key keywords, and clarity of expression of the response, or may include a combination of multiple evaluation criteria. The control unit (140) applies the evaluation criteria of each scoring instruction individually to the essay-type response to calculate an independent score. Therefore, even if the essay-type response is the same, the score calculated as a result of the scoring may differ depending on the scoring instruction.
[0110] In one embodiment, as illustrated in FIG. 5, for essay-type question-and-answer data (200), the control unit (140) may apply a plurality of scoring instructions generated by a large-scale language model (150) to each and score them separately. For example, the first scoring instruction may produce a scoring result of 5.0 according to grammar-centered scoring criteria, the second scoring instruction may produce a scoring result of 5.5 according to content-centered scoring criteria, and the third scoring instruction may produce a scoring result of 6.0 according to vocabulary-centered scoring criteria.
[0111] Specifically, the control unit (140) can perform the scoring operation using a large-scale language model (150) during scoring. That is, the control unit (140) inputs each scoring instruction and the essay-type question-and-answer data (200) to be evaluated into the large-scale language model (150), and the large-scale language model (150) can evaluate the responses included in the essay-type question-and-answer data (200) based on the scoring criteria included in the scoring instruction and output the result converted into a score. For example, if a specific scoring instruction includes a scoring criterion that assigns a score based on whether a specific core concept is included, the large-scale language model (150) can analyze the response to determine whether the core concept is included and then calculate the final score accordingly.
[0112] At this time, the large-scale language model (150) can comprehensively analyze the input response through contextual comparison with evaluation criteria, semantic analysis, keyword matching, and evaluation of lexical and syntactic accuracy.
[0113] The scoring result can generally be expressed as a numerical score, for example, the score can be calculated as an integer or real value between 0 and 10 points that are pre-set.
[0114] The calculated scoring results may be stored individually in the storage unit (130) corresponding to each of the multiple scoring instructions, and in addition to the individual scores calculated according to each scoring instruction, detailed evaluation history for the response to be evaluated (e.g., score assignment history by evaluation criteria, explanatory data indicating the basis for evaluation or reason for judgment generated by the large-scale language model (150) during the evaluation process, etc.) may also be additionally stored.
[0115] The control unit (140) can calculate the degree of alignment to determine whether an error is included in a scoring instruction based on the scoring result included in each scoring instruction, and compare the calculated degree of alignment with a preset threshold value (S710, see FIG. 7).
[0116] The control unit (140) can calculate the degree of alignment between each scoring instruction and the assigned score by comparing the scoring result calculated using each scoring instruction with the assigned score included in the essay-type question-and-answer data. For example, the degree of alignment (QWK, Quadratic Weighted Kappa) can be used as a measure to calculate the degree of alignment.
[0117] QWK is a representative statistical metric that quantifies and evaluates the degree of agreement between automatic scoring results and assigned scores. It ranges from -1 to 1, and the closer it is to 1, the higher the degree of agreement between the two evaluation results.
[0118] In this specification, for convenience of explanation, a measure of consistency is described as an example, but the present invention is not limited thereto, and various statistical measures and evaluation measures such as correlation coefficient, mean squared error (MSE), mean absolute error (MAE), and accuracy may be used to calculate the consistency of scoring.
[0119] In order to calculate the degree of alignment, the control unit (140) compares the difference between the two scores by comparing the score result calculated by each scoring instruction with the assigned score, and calculates the degree of alignment value based on the difference.
[0120] As illustrated in FIG. 6, when the score assigned to the essay-type question-and-answer data (200) is set to 5.5, the control unit (140) can calculate the degree of alignment based on the difference between each scoring result and the assigned score.
[0121] In one embodiment, when the scoring result scored by the first scoring instruction is 5.0, the difference from the assigned score is 0.5, and the degree of agreement can be calculated as 0.82.
[0122] In another embodiment, when the scoring result scored by the second scoring instruction is 5.4, the difference from the assigned score is 0.1, and the degree of agreement can be calculated as 0.98.
[0123] In another embodiment, when the scoring result scored by the third scoring instruction is 6.0, the difference from the assigned score is 0.6, and the degree of agreement can be calculated as 0.79.
[0124] At this time, the control unit (140) can calculate the degree of alignment by comprehensively comparing the scores for all responses in this manner.
[0125] The control unit (140) can store the calculated degree of alignment in the storage unit (130). At this time, the degree of alignment of each scoring instruction can be stored in the form of metadata associated with the scoring instruction.
[0126] The control unit (140) can record various information that may occur during the process of calculating the degree of alignment. For example, if a specific scoring instruction repeatedly calculates a degree of alignment below a preset threshold for a specific type of response, the control unit (140) can record this as separate error analysis information to specifically provide the reason and direction for the need to improve the scoring instruction.
[0127] Meanwhile, in the present invention, a process of improving a plurality of scoring instructions based on the calculated scoring results and generating a plurality of improved scoring instructions may be carried out (S340, see FIG. 3).
[0128] The control unit (140) can improve a plurality of scoring instructions based on the calculated alignment degree to generate a plurality of improved scoring instructions.
[0129] Here, the improvement of scoring instructions may mean analyzing the problems or limitations of each scoring instruction using the scoring performance results, and generating new scoring instructions with higher scoring performance by adjusting the scoring criteria or modifying the content of the scoring instructions to reflect this.
[0130] The control unit (140) analyzes the degree of consistency based on the degree of consistency of each scoring instruction stored in the storage unit (130), the scoring result according to the instruction, and the assigned score included in the essay-type question-and-answer data, and thereby can determine the scoring instruction containing an error.
[0131] Here, a scoring instruction containing an error may refer to a scoring instruction that corresponds to a case where the scoring result calculated by the scoring instruction differs significantly from the assigned score, that is, a case where the degree of consistency is less than a preset threshold value (e.g., 0.9 or less).
[0132] The control unit (140) can analyze essay-type question-and-answer data containing errors to determine the reason why each scoring instruction shows low consistency in the response evaluation process. For example, it can analyze cases where a specific scoring instruction failed to evaluate specific aspects of the response included in the essay-type question-and-answer data (e.g., logic, presence or absence of keywords, sentence structure, etc.), or where the large-scale language model (150) failed to produce consistent results in the scoring process because the evaluation criteria were unclear or ambiguous, and store the results in the storage unit (130).
[0133] Based on the results of analyzing a scoring instruction containing errors, the control unit (140) can adjust the scoring criteria included in the scoring instruction containing errors (S720, see FIG. 7).
[0134] Adjusting the scoring criteria here may mean modifying the details of the evaluation criteria or the evaluation method to be clearer and more specific, or changing or adding the application conditions of the criteria, the scoring method, or the priority of the evaluation. For example, if a specific scoring instruction fails to evaluate a specific element of the response, the control unit (140) may use a large-scale language model (150) to clarify the scoring criteria or improve the scoring instruction to add missing key evaluation items.
[0135] The control unit (140) can generate multiple improved scoring instructions by reflecting the improved scoring criteria in each existing scoring instruction (S730, see FIG. 7).
[0136] The control unit (140) can automatically generate improved scoring instructions using a large-scale language model (150), and can generate improved scoring instructions by inputting a prompt into the large-scale language model (150) that explains the results of the analysis of problems with existing scoring instructions and the direction of improvement. For example, the control unit (140) can input a prompt into the large-scale language model (150) such as, “The existing scoring instructions were insufficient when evaluating the logicality of the response. Rewrite the scoring instructions to include improved scoring criteria so that logicality can be evaluated more clearly,” through which the large-scale language model (150) can generate improved scoring instructions including improved scoring criteria.
[0137] Multiple improved scoring instructions may be stored in the storage unit (130), and each scoring instruction may be stored together with metadata describing the differences or improvements from existing scoring instructions. For example, if a specific scoring instruction includes an improvement such as “more specifically defining the criteria for evaluating the logicality of the response,” the control unit (140) may store this information together with the scoring instruction in the storage unit (130).
[0138] The control unit (140) can compare the difference between the existing scoring instruction and the improved scoring instruction. Here, the comparison may include at least one of the composition of the scoring criteria items, the clarity of the evaluation wording, whether the scoring method has been changed, and the adjustment of the scope of application of the criteria. Based on the comparison results, the control unit (140) can generate adjustment history information, such as information on changes to the scoring criteria, reasons for adjustment, and details of the addition and deletion of evaluation items, and this information may be stored in the storage unit (130) in the form of metadata.
[0139] Meanwhile, in the present invention, among a plurality of improved scoring instructions, a process of determining the optimal scoring instruction that satisfies a preset criterion may be carried out (S350, see FIG. 3).
[0140] The control unit (140) can determine the optimal scoring instruction among the multiple improved scoring instructions stored in the storage unit (130).
[0141] Determining the optimal scoring instruction here may mean comparing the consistency of each improved scoring instruction and selecting a single scoring instruction with a scoring criterion that provides excellent scoring accuracy based on the highest consistency of the essay-type question-and-answer data.
[0142] The control unit (140) can re-score the essay-type question-and-answer data stored in the storage unit (130) using improved multiple scoring instructions and produce a scoring result accordingly (S810, see FIG. 8). This process can be carried out in the same way as the process of scoring and producing a scoring result using scoring instructions as described above, but differs in that it uses improved scoring instructions.
[0143] The control unit (140) can re-evaluate the responses included in the essay-type question-and-answer data using the scoring criteria included in each of the improved multiple scoring instructions. At this time, a large-scale language model (150) may be used in the scoring process, and the large-scale language model (150) can score the responses included in the essay-type question-and-answer data based on the scoring criteria and methods included in each improved scoring instruction. For example, if a specific improved scoring instruction is improved to evaluate the presence of specific core keywords and the clarity of logical components in the response, the large-scale language model (150) can score and score the evaluation elements in the response based on these specific criteria.
[0144] The scoring results of each of the improved scoring instructions produced in this way can be stored in the storage unit (130), and the control unit (140) can evaluate the scoring performance of each improved scoring instruction by comparing the scoring results with the assigned score (S820, see FIG. 8).
[0145] As explained above, the evaluation method can quantify the degree of agreement between the scoring results and the assigned scores using the degree of consistency.
[0146] As illustrated in FIG. 9, the control unit (140) calculates the degree of alignment for each of the improved multiple scoring instructions and, based on this, can determine the scoring instruction with the highest degree of alignment as the optimal scoring instruction (300) (S830, see FIG. 8).
[0147] Here, the scoring instruction with the highest degree of consistency can be determined as the instruction that can appropriately evaluate the response because the calculated scoring result most closely matches the assigned score included in the essay-type question-and-answer data.
[0148] In this case, while the determination of the optimal scoring instruction may be based solely on the consistency value, it may also be determined based on more comprehensive criteria by considering various additional factors (e.g., complexity of the instruction's evaluation criteria, clarity, scoring speed, efficient use of resources, etc.).
[0149] The optimal scoring instruction (300) finally determined is stored in the storage unit (130) and can be provided to a user or an external system through the output unit (120). The optimal scoring instruction provided through the output unit (120) can be used for future evaluation and scoring of essay-type question-and-answer data, and can be used in various evaluation environments, such as automatic evaluation systems or online evaluation platforms.
[0150] Meanwhile, for the convenience of explanation, the present specification has focused on an embodiment in which the steps described above are automatically performed by the control unit (140); however, the present invention is not limited thereto and may include various embodiments in which a user (e.g., a teacher or an administrator) can decisively intervene in the automated procedure.
[0151] In one embodiment, the control unit (140) may allow manual review and supplementation by the user during the evaluation process of the calculated scoring results. Specifically, the user may check some of the scoring results presented through the output unit (120), determine the appropriateness of the results or whether they contain errors, and manually adjust the scores or add comments. Such user feedback may be provided to the instruction generation system (100) through the input unit (110), and the control unit (140) may re-evaluate the accuracy of the scoring results based on the user feedback or reflect the consistency of each of the multiple scoring instructions in the calculation process.
[0152] In another embodiment, the control unit (140) may incorporate user intervention during the process of generating a plurality of improved scoring instructions. After reviewing the draft of the improved scoring instructions through the output unit (120), the user may convey a request for modification or supplementary comments through the input unit (110). For example, the user may perform at least one of deleting a specific scoring criterion item, adding a new criterion, adjusting weights between items, or modifying the evaluation expression method. The control unit (140) may review whether such user input is consistent with the previous alignment calculation results or error response characteristics, and then regenerate a plurality of improved scoring instructions by reflecting the content. At this time, user intervention may also be structured to be determined in a semi-automatic manner.
[0153] In another embodiment, the control unit (140) may allow user intervention in the process of determining the optimal scoring instruction. The user may manually review some of the candidate scoring instructions presented through the output unit (120), input a priority for a specific scoring instruction, or supplement the final decision, and the control unit (140) may receive such input and determine the final optimal scoring instruction.
[0154] The control unit (140) can store information regarding the scoring results, corrections to scoring instructions, decisions, etc., in the storage unit (130) in a separate manner, and this can be used for scoring reliability analysis or standard improvement.
[0155] The control unit (140) can score essay-type responses included in additionally received essay-type question-and-answer data using the determined optimal scoring instruction (300). At this time, the scoring criteria included in the optimal scoring instruction (300) may include various evaluation elements such as the accuracy, logic, grammatical completeness, and clarity of expression of the response.
[0156] The control unit (140) can evaluate essay-type responses according to scoring criteria using a large-scale language model (150) and produce the result in the form of a score. The produced scoring result can be stored in the storage unit (130) or provided to the user through the output unit (120), and the optimal scoring instruction determined therefrom can be used in the subsequent automatic scoring process.
[0157] As described above, the method and system for generating scoring instructions for automatic scoring according to the present invention can generate a plurality of scoring instructions and determine the optimal scoring instruction among them. Through this, even if essay-type responses to the same question are written in various ways, scores can be calculated by applying the criterion with the highest evaluation accuracy, thereby ensuring consistency in scoring results. Furthermore, by the system independently determining appropriate criteria without evaluator intervention, the overall scoring quality can be stably maintained, and a reliable automatic evaluation environment can be realized.
[0158] In addition, the method and system for generating instructions for automatic scoring according to the present invention can improve scoring instructions by comparing the difference between the scoring result and the assigned score and reflecting it. Through this, the model learns from previous errors and does not repeat them, and only instructions close to the actual scoring criteria remain, so that the system can be structured to evolve on its own.
[0159] Furthermore, the method and system for generating instructions for automatic scoring according to the present invention can automatically adjust scoring criteria and reflect them in scoring instructions. This eliminates the need for a person to repeatedly modify or review the criteria, and enables effective response to various types of essay-type assessment situations. Additionally, since the criteria automatically adapt even if the input data or question type changes, they can be utilized in diverse environments without being fixed to a specific situation.
[0160] Furthermore, the method and system for generating instructions for automatic scoring according to the present invention can re-verify the performance of scoring instructions by re-scoring based on the adjusted scoring instructions. Through this, the excellence of the scoring instructions can be quantified and measured by repeatedly verifying actual scoring results rather than through simple comparison, and standards can be determined based on objective grounds.
[0161] Furthermore, the method and system for generating instructions for automatic scoring according to the present invention can generate scoring instructions by utilizing essay-type responses and assigned scores together. Through this, the direction of judgment can be learned by referring to actual human scoring results, and as a result, natural scoring criteria similar to the flow of human evaluation can be automatically configured.
[0162] Furthermore, the method and system for generating instructions for automatic scoring according to the present invention can automatically generate scoring instructions. Through this, scoring criteria can be quickly generated or adjusted for various evaluation items without the need for a person to design the criteria directly, thereby significantly improving the efficiency and scalability of the scoring system.
[0163] Meanwhile, the present invention described above can be implemented as a program that is executed by one or more processes on a computer and can be stored on a computer-readable medium (or recording medium).
[0164] Furthermore, the present invention described above can be implemented as computer-readable code or instructions on a medium on which a program is recorded. That is, the present invention can be provided in the form of a program.
[0165] Meanwhile, computer-readable media include all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
[0166] Furthermore, the computer-readable medium may be a server or cloud storage that includes a storage and is accessible to an electronic device via communication. In this case, the computer may download the program according to the present invention from the server or cloud storage via wired or wireless communication.
[0167] Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, namely a CPU (Central Processing Unit), and no special limitations are placed on its type.
[0168] Meanwhile, the above detailed description should not be interpreted restrictively in all respects but should be considered exemplary. The scope of the invention shall be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention.
Claims
Claim 1 A method for generating instructions for automatic scoring, characterized by comprising: receiving essay-type question-and-answer data configured such that at least one question to which a score is assigned and an essay-type response to the question form pairs; generating a plurality of scoring instructions using the essay-type question-and-answer data; scoring the essay-type response using the plurality of scoring instructions and calculating a scoring result for the essay-type response; improving the plurality of scoring instructions based on the calculated scoring result to generate a plurality of improved scoring instructions; and determining an optimal scoring instruction among the plurality of improved scoring instructions that satisfies a preset criterion. Claim 2 A method for generating instructions for automatic scoring, characterized in that, in the step of generating the plurality of scoring instructions according to claim 1, the essay-type question-and-answer data is input into a large-scale language model (LLM), and the plurality of scoring instructions including scoring criteria related to the essay-type question-and-answer data are generated in the large-scale language model. Claim 3 A method for generating instructions for automatic scoring, characterized in that, in the second paragraph, the scoring criteria related to the essay-type question-and-answer data include scoring criteria for each of the plurality of scoring instructions, and in the step of calculating the scoring result, the essay-type response is scored based on the scoring criteria for each of the plurality of scoring instructions, and the scoring result for the essay-type response is calculated. Claim 4 A method for generating instructions for automatic scoring, characterized in that, in paragraph 3, the degree of alignment between the scoring result and the score assigned to the essay-type question-and-answer data is calculated, and the plurality of scoring instructions are improved based on the calculated degree of alignment to generate the improved plurality of scoring instructions. Claim 5 A method for generating instructions for automatic scoring, characterized in that, in claim 4, it further includes a step of determining whether there is an error in the scoring instructions by comparing the above-mentioned consistency with a preset threshold value. Claim 6 A method for generating instructions for automatic scoring, characterized in that, in the 5th paragraph, if the result of the above judgment is that the degree of conformity is below a preset threshold value, it is determined that the scoring instruction contains an error. Claim 7 A method for generating instructions for automatic scoring, characterized in that, in claim 6, it further includes a step of adjusting the scoring criteria of the scoring instructions containing the above error. Claim 8 A method for generating instructions for automatic scoring, characterized in that, in the step of generating the improved plurality of scoring instructions in claim 7, the improved plurality of scoring instructions are generated using the adjusted scoring criteria. Claim 9 A method for generating instructions for automatic scoring, characterized in that, in claim 8, the step of determining the optimal scoring instruction further comprises: a step of scoring the essay-type response using the improved plurality of scoring instructions; and a step of calculating a scoring result for the essay-type response. Claim 10 A method for generating instructions for automatic scoring, characterized in that, in claim 9, it further includes the step of calculating the degree of consistency between the scoring result and the score assigned to the essay-type question-and-answer data. Claim 11 A method for generating instructions for automatic scoring, characterized in that, in claim 10, by comparing the calculated consistency degrees and determining the scoring instruction with the highest consistency degree among the improved plurality of scoring instructions as the optimal scoring instruction. Claim 12 A method for generating instructions for automatic scoring, characterized in that, in claim 11, it further comprises the step of scoring at least one specific essay-type response using the optimal scoring instructions determined above; and the step of generating a scoring result for the specific essay-type response. Claim 13 A method for generating instructions for automatic scoring, characterized in that, in claim 1, it further comprises: a step of comparing the difference between the improved plurality of scoring instructions and the existing scoring instructions; and a step of storing information including at least one of change information and adjustment details of the scoring criteria based on the comparison result. Claim 14 A system for generating instructions for automatic scoring, comprising: an input unit for receiving essay-type question-and-answer data configured such that at least one question to which a score is assigned and an essay-type response to the question form a pair; a large-scale language model for generating a plurality of scoring instructions using the essay-type question-and-answer data; and a control unit for scoring the essay-type response using the plurality of scoring instructions and calculating a scoring result for the essay-type response, wherein the control unit improves the plurality of scoring instructions based on the calculated scoring result to generate a plurality of improved scoring instructions, and determines an optimal scoring instruction among the plurality of improved scoring instructions that satisfies a preset criterion. Claim 15 A program stored on a computer-readable recording medium that is executed by one or more processes in an electronic device, the program comprising instructions for performing the following steps: receiving essay-type question-and-answer data configured such that at least one question to which a score is assigned and an essay-type response to the question form pairs; generating a plurality of scoring instructions using the essay-type question-and-answer data; scoring the essay-type response using the plurality of scoring instructions and calculating a scoring result for the essay-type response; improving the plurality of scoring instructions based on the calculated scoring result to generate a plurality of improved scoring instructions; and determining an optimal scoring instruction among the plurality of improved scoring instructions that satisfies a preset criterion.
