Automatic scoring system for free-form writing questions

The automatic scoring system uses AI to create scoring criteria for free-text questions, addressing subjectivity and inefficiencies in current systems by objectively evaluating answers based on solution processes and rubrics, enhancing grading efficiency and fairness.

JP7598513B1Active Publication Date: 2024-12-11RYBINNOVATIONS CO LTD

Patent Information

Application Number
JP2024151044
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2024-09-02
Publication Date
2024-12-11
Estimated Expiration
2044-09-02

AI Technical Summary

Technical Problem

Current automatic scoring systems are unable to objectively and efficiently score free-text questions due to the variability of answers and the subjectivity of human scoring, requiring significant effort and time to create scoring criteria, and often result in biased grading.

Method used

An automatic scoring system using artificial intelligence to create scoring criteria based on the solution process for free-text questions, incorporating constraints, rubrics, and estimated thought processes to objectively evaluate answers.

Benefits of technology

Enables efficient, objective, and fair grading of free-text questions, reducing the time and effort required for scoring and minimizing grading biases.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007598513000001_ABST
    Figure 0007598513000001_ABST
Patent Text Reader

Abstract

To provide an automatic free description question marking system and an automatic free description question marking method, in which an artificial intelligence automatically creates a marking standard for free description questions and the artificial intelligence automatically and objectively marks questions based on the marking standard. [Solution] The artificial intelligence has a scoring criteria creation unit 31 that creates scoring criteria according to the solution process from the free-form question to the correct answer based on free-form question information indicating free-form questions related to a specified test subject, correct answer information indicating the correct answer to the written question information, and scoring information indicating the number of points related to the free-form question information, and a scoring unit 32 that scores the answer of the solver based on the scoring criteria from the answer information indicating the solver's answer.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an automatic free-response question marking system in which an artificial intelligence automatically creates marking criteria for free-response questions and automatically and objectively marks questions based on the marking criteria. [Background technology]

[0002] Conventionally, there have been automatic marking systems (such as digital marking systems) that automatically mark the answers (answers to symbolic questions) of many examinees and test takers using artificial intelligence (AI). However, since the answers to free-form writing questions vary widely, it is said that it is difficult for a system to automatically mark them, and ultimately, marks are given according to the marking criteria set by the subjective marker or according to the marker's ability.

[0003] In response to this, Patent Document 1 proposes a system for automatically marking sentence answer questions using AI, which has a construction means for composing a learning pattern structure for the AI ​​for judging answer sentences into up to four categories, namely, correct answer as the correct answer, reverse conclusion as the incorrect answer, reverse premise or reason, and exception, and a storage means for storing example answers divided into four patterns in the AI, a judgment means for having the AI ​​judge the closeness of the answered sentence to the example patterns stored in that category and judge the probability in each category, and a judgment means for judging and displaying the correct answer rate of the learner's answer to the sentence problem as a degree of appropriateness, and which proposes an automatic marking system for sentence answer questions that improves judgment accuracy through machine learning of a small number of patterns and makes it possible to significantly reduce human checking by changing the combination of the premise (subject or reason part) and conclusion part of the sentence and having the AI ​​memorize answer patterns for not only correct answers but also incorrect answers. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2023-86037 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the automatic marking systems currently in use only use AI to automatically mark handwritten symbols and numerical values ​​(digits); they cannot automatically mark English words, kanji characters, etc., and ultimately humans (markers) visually determine the correct answer.

[0006] In addition, although the above-mentioned automatic scoring system for written answer questions can improve the accuracy of determining correct answers and significantly reduce the need for human (grader) checking, it has the problem that it cannot perform completely automatic scoring.

[0007] In addition, even with the above-mentioned automatic scoring system for essay-answer questions, in order to score objectively, a human must create scoring criteria for each given question, and creating the scoring criteria requires a considerable amount of time and effort.

[0008] Furthermore, even when marking is done by humans (graders), when multiple graders are marking the same questions, there is the problem that the marking results can be biased due to the ability of each grader. There is also the problem that a huge amount of time may be spent before marking in order to increase inter-rater reliability.

[0009] In consideration of the above-mentioned circumstances, the present invention aims to provide an automatic scoring system and method for free-form writing questions in which artificial intelligence automatically creates scoring criteria for free-form writing questions and the artificial intelligence automatically and objectively scores questions based on the scoring criteria.

[0010] Furthermore, the present invention allows the scoring criteria to be based on the solution process leading to the correct answer, making it possible to grasp the current stage of the solver's solution, what knowledge and skills the solver is lacking, and what the challenges are for the group of solvers.

[0011] Furthermore, the present invention will make it possible to easily set free-form questions as needed, which question writers have been hesitant to set because of the time and effort required for grading, thus revolutionizing the way questions are set in all exams and dramatically improving the academic ability of all humanity. It has the potential to greatly change education in all fields around the world that requires free-form writing in exams.

[0012] These problems (issues) do not preclude the existence of other problems. Also, each aspect of the present invention described below does not necessarily solve all of these problems (issues). Furthermore, it is possible to extract problems (issues) other than these from the description of the specification, drawings, or claims. [Means for solving the problem]

[0013] The inventors of the present invention have conducted intensive research into the above-mentioned problems and have come up with the following revolutionary automatic marking system for free-form writing questions.

[0014] A first aspect of the present invention for solving the above problems is an automatic scoring system for free-form writing questions that uses artificial intelligence to automatically score a solver's answer to a free-form writing question related to a specified examination subject, characterized in that the artificial intelligence has a scoring criteria creation unit that creates scoring criteria according to the solution process from the free-form writing question to the correct answer based on free-form writing question information indicating the free-form writing question related to the specified examination subject, correct answer information indicating the correct answer to the free-form writing question, and scoring information indicating the number of points scored for the free-form writing question, and a scoring unit that scores the solver's answer from the answer information indicating the solver's answer based on the scoring criteria.

[0015] Here, "free-form questions" refers to questions in which the answer is given not only in words but also in "sentences," such as short essays, English compositions, questions that require handwritten calculations in mathematics (arithmetic) and science (biology, chemistry, physics, earth science) on answer sheets with free-form writing sections, free-form questions that require explanations in social studies (geography, history, ethics, political economy), and other types of questions used in university and qualification examinations. "Solution process" refers to the steps taken to arrive at the correct answer to a problem.

[0016] According to the first aspect, an automatic grading system for free-form questions can be provided in which an artificial intelligence automatically creates grading criteria in accordance with the solution process, and the artificial intelligence automatically and objectively grades questions based on the grading criteria.

[0017] A second aspect of the present invention resides in the automatic free-form question grading system described in the first aspect, characterized in that the grading criteria creation unit creates grading criteria according to the solution process from the free-form question to the correct answer, based on constraint information indicating conditions to be taken into consideration when creating the grading criteria, free-form question information indicating a free-form question related to a specified examination subject, correct answer information indicating the correct answer to the free-form question information, and grading information indicating the number of marks related to the free-form question information.

[0018] According to the second aspect, an automatic free-form question grading system can be provided in which an artificial intelligence automatically creates grading criteria (satisfying constraint information) for free-form questions, and the artificial intelligence automatically and objectively grades questions based on the grading criteria.

[0019] A third aspect of the present invention resides in the automatic grading system for free-form questions according to the second aspect, characterized in that the constraint information is a predetermined rubric.

[0020] Here, a "rubric" refers to a scale with several levels that indicates the degree of success, and more specifically, to an assessment tool for measuring the achievement of learning that has the following three characteristics: (1) Visualize the evaluation perspectives and scales in a table. (2) It is used to judge the degree of achievement of learning objectives. (3) It is a tool that serves as a criterion for absolute evaluation, showing the evaluation "perspectives" and "scales" in a matrix table.

[0021] It should be noted that the rubric in this embodiment is created by a person who provides guidance or support (e.g., a teacher, lecturer, professor, etc.).

[0022] According to the third aspect, an automatic free-writing question grading system can be provided in which an artificial intelligence automatically creates grading criteria for free-writing questions that also satisfy the rubric, and the artificial intelligence automatically and objectively grades the questions based on the grading criteria. As a result, the instructor or supporter (e.g., a teacher, lecturer, professor, etc.) and the learner can share the evaluation criteria, and a fair and transparent evaluation can be performed.

[0023] A fourth aspect of the present invention is the free-form question automatic marking system according to any one of the first to third aspects, characterized in that the marking criterion creation unit creates a marking criterion that takes into consideration an estimated thought process that is not shown in the free-form question information and the correct answer information but is estimated to be considered consciously or unconsciously by a human being from a free-form question to a correct answer. That is, the marking criterion creation unit of this aspect may create a marking criterion according to a solution process from a free-form question to a correct answer based on free-form question information showing a free-form question related to a predetermined test subject, correct answer information showing a correct answer to the free-form question, marking information showing the number of marks for the free-form question, and the estimated thought process described above, or may create a marking criterion according to a solution process from a free-form question to a correct answer based on constraint information showing a condition to be considered when creating the marking criterion, free-form question information showing a free-form question related to a predetermined test subject, correct answer information showing a correct answer to the free-form question information, marking information showing the number of marks for the free-form question information, and the estimated thought process described above.

[0024] According to the fourth aspect, the marking criteria creation unit can create marking criteria that take into account estimated thought processes, making it possible to provide an automatic marking system for free-form questions that marks based on more appropriate marking criteria.

[0025] A fifth aspect of the present invention resides in the automatic scoring system for free-form writing questions according to the fourth aspect, characterized in that the inferred thought process is at least one of metacognition (recognition of one's own thought process), critical thinking, logical thinking, intuitive thinking and problem-solving ability.

[0026] According to the fifth aspect, it is possible to provide an automatic free-writing question marking system that marks questions based on more appropriate marking criteria.

[0027] A sixth aspect of the present invention is the free-form question automatic marking system according to any one of the first to third aspects, characterized in that the marking criterion creation unit creates a marking criterion that also takes into account a rubric. That is, the marking criterion creation unit of this aspect may create a marking criterion according to the solution process from a free-form question to a correct answer based on free-form question information indicating a free-form question related to a predetermined test subject, correct answer information indicating a correct answer to the free-form question, marking information indicating the number of marks for the free-form question, and the above-mentioned rubric, or may create a marking criterion according to the solution process from a free-form question to a correct answer based on constraint information indicating conditions to be considered when creating the marking criterion, free-form question information indicating a free-form question related to a predetermined test subject, correct answer information indicating a correct answer to the free-form question information, marking information indicating the number of marks for the free-form question information, and the above-mentioned rubric.

[0028] According to the sixth aspect, the marking criteria creation unit can create marking criteria that take rubrics into consideration, making it possible to provide an automatic marking system for free-form questions that marks the academic ability of each respondent based on more appropriate marking criteria.

[0029] A seventh aspect of the present invention resides in the automatic grading system for free-form questions as described in claim 1, characterized in that the grading criteria creation unit creates grading criteria so that a multiple of the number of solution processes is equal to the grading information.

[0030] According to the seventh aspect, it is possible to provide an automatic free-description question marking system that marks the answers of answerers more objectively.

[0031] An eighth aspect of the present invention resides in the automatic grading system for free-form writing questions described in the first aspect, characterized in that the grading criteria created by the grading criteria creation unit are modified by a human.

[0032] Here, the "person" may be not only the grader or the creator of the free-response question, but also a third party.

[0033] According to the eighth aspect, a human can modify the scoring criteria to reflect conditions (criteria) that were not taken into account by artificial intelligence or conditions (criteria) that must be taken into account in the scoring criteria.

[0034] A ninth aspect of the present invention resides in an automatic scoring system for free-form writing questions as described in the first aspect, characterized in that the answer of the solver is handwritten on an answer sheet, and the system further comprises an answer information extraction unit that extracts answer information from the handwritten answer on the answer sheet.

[0035] According to the ninth aspect, it is possible to provide an automatic free-writing question marking system that automatically marks answers handwritten by answerers without requiring much time and effort.

[0036] In the present invention, the terms "database," "system," and "unit" do not simply mean physical means, but also include cases where the functions of the "database," "unit," and "system" are realized by software. Furthermore, the functions of one "database," "system," or "unit" may be realized by two or more physical means or devices, and the functions of two or more "databases," "systems," or "units" may be realized by one physical means or device. [Brief description of the drawings]

[0037] [Figure 1] FIG. 1 is a schematic diagram of an automatic marking system for free-form writing questions according to the first embodiment. [Diagram 2] FIG. 2 is a schematic diagram of the terminal of the first embodiment. [Diagram 3] FIG. 3 is a schematic diagram of the free-answer question automatic marking server of the first embodiment. [Figure 4] FIG. 4 is a flowchart showing the operation of the free-answer question automatic marking system of the first embodiment. [Diagram 5] FIG. 5 is an image showing information on the free-form questions (question 1, question 2) in the first embodiment. [Figure 6] FIG. 6 shows the marking criteria for Problem 1 in Example 1. [Figure 7] FIG. 7 shows the marking criteria for question 2 in Example 1. [Figure 8] FIG. 8 is an image showing the answer given by the solver in Example 1. [Figure 9] FIG. 9 shows the marking results for Problem 1 in Example 1. [Figure 10] FIG. 10 shows the marking results for question 2 in Example 1. [Figure 11] FIG. 11 is an image showing the answer given by the contestant in Example 2. [Figure 12] FIG. 12 shows the marking results for question 1 in Example 2. [Figure 13] FIG. 13 shows the marking results for problem 2 in Example 2. [Figure 14] FIG. 14 is an image showing a free description question in the third embodiment. [Figure 15] FIG. 15 is an image showing the correct answer in Example 3. [Figure 16] FIG. 16 shows the scoring criteria for Example 3. [Figure 17] FIG. 17 is an image showing the answer (incorrect answer) for Example 3. [Figure 18] FIG. 18 shows the scoring results for Example 3. [Figure 19] FIG. 19 shows the marking criteria for question 1 in Example 4. [Figure 20] FIG. 20 shows the marking criteria for question 2 in Example 4. [Figure 21] FIG. 21 shows the marking results for question 1 in Example 4. [Figure 22] FIG. 22 shows the marking results for question 1 in Example 4. [Figure 23] FIG. 23 shows the marking results for problem 1 in Example 5. [Figure 24] FIG. 24 shows the marking results for problem 2 in Example 5. [Diagram 25] FIG. 25 shows the scoring criteria for Example 6. [Figure 26] FIG. 26 shows the scoring results of Example 6. [Figure 27] FIG. 27 is an image showing the problem in Example 7. [Figure 28] Figure 28 is the rubric for Example 7. [Figure 29] FIG. 29 shows the scoring criteria for Example 7. [Diagram 30] FIG. 30 is an image showing the answers of students A to C in Example 7. [Diagram 31] FIG. 31 shows the grading results of Student A in Example 7. [Diagram 32] FIG. 32 shows the grading results of Student B in Example 7. [Diagram 33] FIG. 33 shows the grading results of Student C in Example 7. [Diagram 34] FIG. 34 shows the problem in Example 8. [Diagram 35] Figure 35 is the correct answer for Example 8. [Diagram 36] Figure 36 is the rubric for Example 8. [Figure 37] FIG. 37 shows the scoring criteria for Example 8. [Figure 38] Figure 38 is an image showing the answer to Example 8. [Figure 39] FIG. 39 shows the scoring results for Example 8. [Diagram 40] FIG. 40 shows the scoring criteria for Example 9. [Diagram 41] FIG. 41 shows the grading results of Student A in Example 9. [Diagram 42] FIG. 42 shows the grading results of Student B in Example 9. [Diagram 43] FIG. 43 shows the grading results of Student C in Example 9. [Diagram 44] FIG. 44 shows the scoring criteria for Example 10. [Diagram 45] FIG. 45 shows the scoring results for Example 10. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of an automatic marking system for free-description questions according to the present invention will be described below with reference to the accompanying drawings. Note that the present invention is not limited to the following embodiment.

[0039] (Embodiment 1) The free-writing question automatic marking system 1 of this embodiment uses an artificial intelligence (AI) that is trained to generate a solution process for free-writing questions in a specified test subject based on free-writing questions related to the specified test subject and the correct answers to the free-writing questions, and the AI ​​automatically creates marking criteria and automatically marks the answer of the answerer based on the marking criteria. As shown in Fig. 1, the free-writing question automatic marking system 1 is composed of a plurality of terminals 10 and a free-writing question automatic marking server 30 connected to the terminals 10 via a network 20. The network 20 is not limited to a wired or wireless network as long as it can connect the terminals 10 and the free-writing question automatic marking server 30, and examples of the network include the Internet and an intranet.

[0040] Here, the answerer may be one or more. The predetermined test subjects refer to, for example, subjects that are predetermined in elementary schools, junior high schools, high schools, universities, etc., or subjects related to qualification tests such as the bar exam, patent attorney exam, judicial scrivener exam, certified public accountant exam, tax accountant exam, social insurance labor consultant exam, administrative scrivener exam, etc., and specific examples include English, Japanese, mathematics (arithmetic), science, social studies (geography, history, and civics), constitutional law, civil law, civil procedure code, criminal law, criminal procedure code, etc.

[0041] First, the terminal 10 will be described. The terminal 10 is used by solvers taking a test that includes free-answer questions on a specified test subject, or by people (for example, teachers, lecturers, professors, etc.) who guide or support solvers, and as shown in Fig. 2, has an output unit 11 that outputs the grading results of the free-answer question automatic grading system 1, and an input unit 12 through which solvers input free-answer question information, correct answer information, and answer information to the free-answer question automatic grading system 1. It goes without saying that when the correct answer information is created by artificial intelligence, the correct answer information does not need to be input to the input unit.

[0042] The output unit 11 is not particularly limited as long as it is capable of outputting the grading results of the free-form question automatic grading system 1 (e.g., text, video, images, audio, etc.), and examples include an LCD display, a Braille display, and a speaker.

[0043] The input unit 12 is not particularly limited as long as it allows the respondent to input free-form question information, correct answer information and answer information (for example, text, video, image (including image files or PDF files of scanned answer sheets), QR code (registered trademark), voice, etc.) to the free-form question automatic scoring system 1, and examples include a keyboard, touch panel, microphone, scanner, camera, USB memory, and file transfer via a network. Here, the correct answer information may be created in advance by artificial intelligence.

[0044] The terminal 10 is not particularly limited as long as it has the above-mentioned output unit 11 and input unit 12, and examples thereof include a personal computer, a tablet computer, and a smartphone.

[0045] Next, the free-writing question automatic grading server 30 will be described. The free-writing question automatic grading server 30 is equipped with artificial intelligence (AI) as shown in FIG. 3. This artificial intelligence is trained to generate a solution process for free-writing questions in a specific test subject based on free-writing questions for teacher data related to the specific test subject and teacher data correct answers for the free-writing questions for the teacher data. Here, teacher data (teaching data) refers to data used by the AI ​​when performing machine learning.

[0046] Here, the artificial intelligence is not particularly limited as long as it is an AI (generative AI) that can generate various information such as images, text, voice, program code, and structured data based on patterns and trends learned from a large amount of data, and examples include ChatGPT (registered trademark), Microsoft Copilot, and Google's Gemini, which use a large-scale language model (LLM). Furthermore, the solution process for a free-text question refers to a procedure for deriving a correct answer to the free-text question from the free-text question, and is composed of multiple procedures (steps). The solution process of the present invention may be the same as or different from the human solution process. Specific examples of the solution process will be described later.

[0047] The free-writing questions for teacher data are free-writing questions on a specified test subject, such as constitutional law essay questions in past bar exams, university law department final exam questions and national exam essay questions, regular and end-of-term exams in educational institutions such as elementary schools, junior high schools, and high schools, free composition and written questions in school entrance exams, writing questions in English proficiency tests such as the Test of Practical English Proficiency and TOEIC, etc. Correct answers for teacher data are correct answers to the free-writing questions for teacher data, such as model answers or sample answers to the above-mentioned exam questions, or model answers or sample answers from educational institutions to constitutional law essay questions in past bar exams.

[0048] The learning method of the artificial intelligence is not particularly limited, and for example, the artificial intelligence may be trained using a large-scale language model (LLM). Here, the "large-scale language model" refers to a natural language model that is constructed with a large amount of calculation, a large amount of data, and a large number of parameters among language models.

[0049] Next, the artificial intelligence will be described in detail. The artificial intelligence includes a marking criterion creating unit 31 and a marking unit 32, as shown in FIG.

[0050] The marking criteria creation unit 31 is a function in which artificial intelligence creates marking criteria according to the solution process from the free-form question to the correct answer, based on free-form question information indicating a free-form question related to a specified test subject, correct answer information indicating the correct answer to the free-form question information, and marking information indicating the marking for the free-form question information.

[0051] Here, the scoring information refers to the maximum score (maximum number of points) that can be awarded if the free-form question is answered correctly. For example, if the maximum score is 10 points, it indicates 10 points, and if the maximum score is 5 points, it indicates 5 points.

[0052] The scoring criteria created by the scoring criteria creation unit 31 has a plurality of combinations of solution processes and the points to be given to the solution processes. That is, the scoring criteria can be expressed as a table consisting of the solution processes and the points to be given to the solution processes, for example, 1 point to be given to solution process 1, 2 points to be given to solution process 2, etc. The total of the points to be given to all the solution processes is equal to the scoring information (the maximum points to be given when the free description question is answered correctly).

[0053] The number of points given to each solution process may or may not be equal, but it is preferable that the scoring information (total number of points given) is a multiple of the number of solution processes. In other words, it is preferable that the scoring criteria creation unit 31 creates the scoring criteria so that a multiple of the number of solution processes is equal to the scoring information. Specifically, when the scoring information is "100", the scoring criteria is created so that the number of solution processes is "10", "20", "50", and "100". Such a relationship allows the solver's answer to be scored more simply and objectively.

[0054] The scoring unit 32 is a function in which the artificial intelligence scores the answer of the answerer from the answer information indicating the answer of the answerer based on the created scoring criteria. Specifically, the artificial intelligence judges which solution process the answerer has cleared (passed) from the answer of the answerer along each solution process. Then, the score is calculated by adding up the points given to the solution processes up to the cleared solution process. That is, if the artificial intelligence judges that the answer of the answerer is correct, the answer is scored with the maximum point, which is the scoring information, and if the artificial intelligence judges that the answerer has not reached the correct answer but has cleared the solution process up to a certain stage, the answer is scored with the total points given to the solution processes cleared up to that point. As described above, each solution process is a procedure for deriving the correct answer to the free description question, so the artificial intelligence will not judge that the answerer has cleared a solution process located downstream without passing through the upstream solution process.

[0055] Next, the operation of the free-form question automatic marking system 1 of this embodiment will be described. As shown in FIG. 4, when the free-form question automatic marking system 1 is operated, the artificial intelligence installed in the free-form question automatic marking server 30 creates marking criteria by the marking criteria creation unit 31 based on the free-form question information, the correct answer information, and the marking information (S1). Here, the created marking criteria may be confirmed and modified by a third party such as a marker or a creator of the free-form question. By such a person modifying the marking criteria, it is possible to reflect conditions (criteria) that were not considered by the artificial intelligence or conditions (criteria) that must be considered in the marking criteria. As a result, the free-form question automatic marking system 1 can perform more appropriate marking.

[0056] Thereafter, the answerer or a person who is guiding or supporting the answerer inputs the answerer's answer information into the input unit 12 of the terminal 10 (S2). The answer information is then transmitted to the free-form question automatic grading server 30, and the artificial intelligence installed in the free-form question automatic grading server 30 grades the answer information using the grading unit 32 according to the above-mentioned grading criteria (S3). The grading results are then transmitted to the terminal 10 via the network 20 and displayed on the output unit 11 (S4), and the operation of the free-form question automatic grading system 1 is completed.

[0057] By operating the automatic free-form question grading system 1 configured as described above in this manner, an automatic free-form question grading system can be provided in which the artificial intelligence automatically creates grading criteria according to the solution process, and the artificial intelligence automatically and objectively grades the questions based on the grading criteria.

[0058] (Embodiment 2) In the first embodiment, the scoring criteria are created using artificial intelligence without any restrictions, but the present invention is not limited to this. For example, the automatic free-writing question grading system may be configured so that the scoring criteria creation unit creates a scoring criterion according to the solution process from the free-writing question to the correct answer based on constraint information indicating the conditions to be considered when creating the scoring criteria, free-writing question information indicating a free-writing question related to a specific test subject, correct answer information indicating a correct answer to the free-writing question information, and grading information indicating the grading of the free-writing question information.

[0059] Here, the constraint information is not particularly limited as long as it is something that should be considered when creating a grading criterion. Examples of the constraint information include "the number of solution processes must match the number of grading information," "the structure of the answer must follow a specific format," "the use of specific keywords is mandatory," and a predetermined rubric. The constraint conditions may be set before the free-form question automatic grading system is operated, or may be set after the free-form question automatic grading system is operated. It goes without saying that when a predetermined rubric is used as the constraint information of this embodiment, the rubric itself must be created in advance.

[0060] By configuring the free-form question automatic marking system to have such a marking criteria creation unit, it is possible to create marking criteria that reflect the intention (way of thinking) of the person using the free-form question automatic marking system (question creator, marker, etc.). As a result, it is possible to perform marking that reflects the intention (way of thinking) of the person using the free-form question automatic marking system.

[0061] It goes without saying that the artificial intelligence used in such an automatic free-response question marking system must be trained to generate a solution process for free-response questions in a specific test subject based on free-response questions for teacher data on a specific test subject and correct answers for the free-response questions for teacher data, including constraints. In fact, ChatGPT (registered trademark) has already learned this information.

[0062] (Embodiment 3) In embodiment 1, the marking criteria creation unit creates marking criteria according to the solution process from the free-form question to the correct answer based on the free-form question information, the correct answer information for the free-form question, and the marking information, but the present invention is not limited to this.

[0063] For example, the scoring criteria creation unit may create scoring criteria that take into account the estimated thought processes that a person is estimated to consider consciously or unconsciously from the free-form question to the correct answer, even though these processes are not shown in the free-form question information and correct answer information.

[0064] Here, the "estimated thought process" is not particularly limited as long as it is a process (procedure) that is estimated to be considered consciously or unconsciously by a human being from the free-answer question to the correct answer, although it is not shown in the free-answer question information and the correct answer information. Examples of the "estimated thought process" include "metacognition," "critical thinking," "intuitive thinking," and "logical thinking." Here, "metacognition" is described in detail in, for example, Reference 1: "Metacognition: An Overview" by John H. Flavell (1979), Reference 2: "Metacognition and Cognitive Neuropsychology: Monitoring and Control Processes" edited by GRD McClelland, GLP Williams, and PALG Delacour (2003), and Reference 3: "Handbook of Metacognition" edited by John Dunlosky and Janet Metcalfe (2009).

[0065] Specifically, for example, by giving the artificial intelligence further prompts such as "Please create scoring criteria that take metacognition into consideration," "Please create scoring criteria that take critical thinking into consideration," "Please create scoring criteria that take intuitive thinking into consideration," "Please create scoring criteria that take logical thinking into consideration," etc., the scoring criteria creation section of the artificial intelligence can create scoring criteria that also take these estimated thought processes into consideration.

[0066] By configuring an automatic scoring system for free-form writing questions that has a scoring criteria creation unit with such functionality, the scoring criteria creation unit can create scoring criteria that take into account estimated thought processes, thereby making it possible to provide an automatic scoring system for free-form writing questions that scores based on more appropriate scoring criteria.

[0067] Needless to say, the artificial intelligence used in such an automatic scoring system for free-response questions must be trained to generate a solution process for free-response questions in a specific test subject based on the free-response questions for the teacher data for the specific test subject and the correct answers for the free-response questions for the teacher data, including estimated thought processes (metacognition, critical thinking, intuitive thinking, logical thinking, etc.). In fact, ChatGPT (registered trademark) has already learned this information.

[0068] (Embodiment 4) In embodiment 1, the marking criteria creation unit creates marking criteria according to the solution process from the free-form question to the correct answer based on the free-form question information, the correct answer information for the free-form question, and the marking information, but the present invention is not limited to this.

[0069] For example, the marking criteria creation unit may create marking criteria that take into account the rubric. In the second embodiment, a predetermined rubric created in advance is used as the restriction information, but in this embodiment, the marking criteria creation unit creates marking criteria taking into account the concept of the rubric. Therefore, the free-writing question automatic marking system of the second embodiment is completely different from the free-writing question automatic marking system of this embodiment.

[0070] Specifically, for example, by further providing the artificial intelligence with a prompt such as "Please create grading criteria that take the rubric into consideration," the grading criteria creation unit of the artificial intelligence can create grading criteria that also take the rubric into consideration.

[0071] By configuring an automatic grading system for free-form writing questions that has a grading criteria creation unit with such functionality, the grading criteria creation unit can create grading criteria that take rubrics into consideration, making it possible to provide an automatic grading system for free-form writing questions that grades based on more appropriate grading criteria.

[0072] Needless to say, the artificial intelligence used in such an automatic free-writing question marking system must be trained to generate a solution process for free-writing questions in a specific test subject based on free-writing questions for teacher data on a specific test subject, including rubrics, and correct answers for the free-writing questions for teacher data. In fact, ChatGPT (registered trademark) has already learned about rubric information.

[0073] (Other embodiments) In the above embodiment, the format of the answer of the answerer is not particularly limited, but the answer of the answerer may be handwritten on an answer sheet such as paper. Note that the material of the answer sheet is not limited to paper, and the concept includes those made of other materials.

[0074] In this case, the input unit of the terminal 10 captures the answer written on the answer sheet as an image. The free-form question automatic grading system may be configured so that at least one of the terminal and the free-form question automatic grading server has an answer information extraction unit that extracts answer information indicating the answerer's answer from the image. Here, the answer information extraction unit is not particularly limited as long as it can extract answer information from the answerer's handwritten answer written on the answer sheet or the like. As the answer information extraction unit, for example, commercially available software using optical character recognition technology (including software using artificial intelligence) can be mentioned.

[0075] In this way, by configuring an automatic free-answer question marking system that further includes an answer information extraction unit, marks can be automatically made from the answerer's handwritten answers without much time and effort.

[0076] <<Example>> Examples are shown below, but in these examples, ChatGPT (registered trademark) was used as the artificial intelligence installed in the free-answer question automatic marking server. Note that, as the terminal, a commercially available personal computer connected to ChatGPT (registered trademark) via the Internet was used.

[0077] In addition, ChatGPT (registered trademark) also has an optical character recognition function (OCR function) that is an answer information extraction section. Therefore, by inputting the handwritten free-form questions, their correct answers, and the answerers' answers as image data, the free-form question information, correct answer information, and answer information can be input to the free-form question automatic grading system.

[0078] <Example 1 (in the case of arithmetic, correct answer)> The two problems shown in Figure 5 (upper formula: problem 1, lower formula: problem 2) were used as the free-form question information, and "8 / 15" (correct answer for problem 1) and "1 / 45" (correct answer for problem 2) were used as the correct answer information for these problems. In addition, "5 points" and "10 points" were used as the marking information, and "5-point evaluation" and "10-point evaluation" were used as the constraint information.

[0079] Then, this information was sent to the artificial intelligence to obtain the grading criteria (solution process (thinking process)) for each of these free-form questions. The grading criteria are shown in Figures 6 and 7.

[0080] As can be seen from these results, with the automatic scoring system for free-form writing questions of the present invention, it was found that even when arithmetic is specified as the specified subject, the scoring criteria creation unit of the artificial intelligence can automatically create scoring criteria in accordance with the solution process.

[0081] Next, the answers (correct answers) of the contestants shown in FIG. 8 were entered and graded, and the graded results shown in FIG. 9 and FIG. 10 were obtained.

[0082] As can be seen from these results, with the automatic scoring system for free-form writing questions of the present invention, even when arithmetic is specified as the specified subject, the artificial intelligence can automatically create scoring criteria (reflecting constraint information) for the free-form writing questions, and even if the correct answer is entered as the answer based on the scoring criteria, the artificial intelligence can automatically and objectively score the questions.

[0083] <Example 2 (in the case of arithmetic, when the answer is incorrect)> When the questions and marking criteria of Example 1 were used and the incorrect answers shown in FIG. 11 were entered and marked, the marks shown in FIGS. 12 and 13 were obtained.

[0084] As can be seen from these results, with the automatic scoring system for free-form writing questions of the present invention, when arithmetic is specified as the specified subject, even if an incorrect answer is entered, the artificial intelligence can automatically and objectively score the question based on the scoring criteria.

[0085] <Example 3 (English: Incorrect answer)> The information shown in Fig. 14 was used as the free-writing question information, and the English sentence shown in Fig. 15 was used as the correct answer to this free-writing question. In addition, "12 points" was used as the marking information, and "Create a 12-step solution process (thinking process)" was used as the constraint information. These pieces of information were then sent to the artificial intelligence to obtain the marking criteria for this written question. The marking criteria are shown in Fig. 16.

[0086] As can be seen from these results, with the automatic scoring system for free-form writing questions of the present invention, it was found that the scoring criteria creation unit of the artificial intelligence can automatically create scoring criteria in accordance with the solution process, even when English is specified as the specified subject.

[0087] Next, when the image data of the answer (incorrect answer) of the answerer shown in Fig. 17 is input to the artificial intelligence and a prompt "Please transcribe it" is input, the artificial intelligence recognizes the handwritten characters included in the image in Fig. 17 and outputs "My favorite place is Shibuya. Because I can buy something. Because I can eat something." When this sentence was scored using the free description question automatic scoring system according to the present invention, the scoring result shown in Fig. 18 was obtained.

[0088] As can be seen from these results, with the automatic scoring system for free-form writing questions of the present invention, even when English is specified as the specified subject, the artificial intelligence can automatically create scoring criteria for the free-form writing questions (which reflect the constraint information), and the artificial intelligence can automatically and objectively score based on those scoring criteria.

[0089] <Example 4 (In the case of arithmetic, in the case of incorrect answers, when the estimated thought process is taken into consideration) For the arithmetic problem and correct answer in Example 1, a scoring criterion was created after instructing the participants to consider metacognition as an estimated thought process. Specifically, the prompt was "If 1 is 5 points and 2 is 10 points, please create a thought process to arrive at the answer, with 1 being 5 levels and 2 being 10 levels. When doing so, please create the process with a focus on the cognitive aspects (information processing) of the answerer, such as "metacognitive knowledge" and "metacognitive activity." As a result, the scoring criteria created by the scoring criteria creation unit of the artificial intelligence are shown in Figures 19 and 20.

[0090] As can be seen from these results, the automatic scoring system for free-form writing questions according to the present invention is able to create scoring criteria that take into account metacognition, which is one type of inferred thought process, even when arithmetic is specified as the specified subject.

[0091] Next, the answers (correct answers) of the contestants shown in FIG. 8 were entered and graded, and the graded results shown in FIG. 21 and FIG. 22 were obtained.

[0092] As can be seen from these results, with the automatic scoring system for free-form writing questions of the present invention, even when arithmetic is specified as the specified subject, the artificial intelligence can automatically create scoring criteria for free-form writing questions that take into account metacognition, which is one of the presumed thought processes, and even if the correct answer is entered as the answer based on the scoring criteria, the artificial intelligence can automatically and objectively score the answer.

[0093] <Example 5 (In the case of arithmetic, in the case of incorrect answers, when the estimated thought process is taken into consideration) When the answers (incorrect answers) of the solvers shown in FIG. 11 were entered and graded using the grading criteria of the fourth embodiment, the grading results shown in FIG. 23 and FIG. 24 were obtained.

[0094] As can be seen from these results, with the automatic scoring system for free-form writing questions of the present invention, the artificial intelligence automatically creates scoring criteria for free-form writing questions that take into account metacognition, which is one of the estimated thought processes, and based on these scoring criteria, the artificial intelligence can automatically and objectively score even if an incorrect answer is entered.

[0095] <Example 6 (English, incorrect answer, considering estimated thought process) For the English questions and correct answers in Example 3, a marking criterion was created after instructing the participants to consider metacognition as an estimated thought process. Specifically, the following prompt was used: "The following question is worth 12 points. Please create a 12-step thought process from the question and model answer to the answer, and use this as the marking criterion. When doing so, please focus on the cognitive aspects (information processing) of the answerer, such as "metacognitive knowledge" and "metacognitive activity." As a result, the marking criterion created by the AI ​​marking criterion creation unit is shown in FIG. 25.

[0096] As can be seen from these results, the automatic scoring system for free-form writing questions according to the present invention is able to create scoring criteria that take into account metacognition, which is one type of estimated thought process, even when English is specified as the specified subject.

[0097] Next, the answers (incorrect answers) of the solvers shown in FIG. 17 were entered and graded in the same manner as in Example 3, and the graded results shown in FIG. 26 were obtained.

[0098] As can be seen from these results, with the automatic scoring system for free-form writing questions of the present invention, even when English is specified as the specified subject, the artificial intelligence can automatically create scoring criteria for free-form writing questions that take into account metacognition, which is one of the presumed thought processes, and can automatically and objectively score questions based on the scoring criteria, even if the correct answer is entered.

[0099] <Example 7 (In the case of mathematics, when "rubric" is used as restriction information)> The problem shown in Fig. 27 was used as the free description problem information, and "-2x+19" was used as the correct answer information for this problem. In addition, "10 points" was used as the marking information, and the rubric shown in Fig. 28 was used as the constraint information.

[0100] Then, this information was sent to the artificial intelligence to obtain the grading criteria (solution process (thinking process)) for this free-form question. The grading criteria are shown in Figure 29.

[0101] As can be seen from these results, with the automatic scoring system for free-form writing questions of the present invention, even when mathematics is specified as the specified subject and a specified rubric is used as the restrictive condition, the scoring criteria creation unit of the artificial intelligence can automatically create scoring criteria in accordance with the solution process.

[0102] Next, the answers of the three solvers (students A to C) shown in FIG. 30 were entered and graded, and the graded results shown in FIG. 31 to FIG. 33 were obtained.

[0103] As can be seen from these results, with the automatic scoring system for free-form writing questions of the present invention, even when mathematics is specified as the specified subject and a specified rubric is used as the restricting condition, the artificial intelligence can automatically create scoring criteria for the free-form writing questions and automatically and objectively score based on those scoring criteria.

[0104] <Example 8 (English: When using "rubric" as restricted information)> The information shown in Fig. 34 was used as the free-writing question information, and the English sentence shown in Fig. 35 was used as the correct answer to this free-writing question. In addition, "10 points" was used as the marking information, and the rubric shown in Fig. 36 was used as the constraint information. These pieces of information were then sent to the artificial intelligence to obtain the marking criteria for this written question. The marking criteria are shown in Fig. 37.

[0105] As can be seen from these results, with the automatic scoring system for free-form writing questions of the present invention, even when English is specified as the specified subject and a specified rubric is used as the restrictive condition, the AI ​​scoring criteria creation unit can automatically create scoring criteria in accordance with the solution process.

[0106] Next, image data of the answer given by the answerer shown in Figure 38 was input into the artificial intelligence, and the text in this image data was scored using the automatic scoring system for free-form writing questions according to the present invention, resulting in the scoring results shown in Figure 39.

[0107] As can be seen from these results, with the automatic scoring system for free-form writing questions of the present invention, even when English is specified as the specified subject and a specified rubric is used as the restricting condition, the artificial intelligence can automatically create scoring criteria for the free-form writing questions (reflecting the rubric), and the artificial intelligence can automatically and objectively score based on the scoring criteria.

[0108] <Example 9 (Mathematics: Considering the rubric)> For the math problem and correct answer in Example 7, "10 points" was used as the marking information, and a marking criterion was created after instructing the subject to take the rubric into consideration. Specifically, the prompt "Please create a marking criterion that takes the rubric into consideration" was used. As a result, the marking criterion created by the marking criterion creation unit of the artificial intelligence is shown in FIG. 40.

[0109] As can be seen from these results, with the automatic scoring system for free-form writing questions according to the present invention, it was found that artificial intelligence can create scoring criteria that take into account rubrics, even when mathematics is specified as the specified subject.

[0110] Next, the answers of the three solvers (students A to C) shown in FIG. 30 were entered and graded, and the graded results shown in FIG. 41 to FIG. 43 were obtained.

[0111] As can be seen from these results, with the automatic scoring system for free-form writing questions of the present invention, even when mathematics is specified as the specified subject, the artificial intelligence can automatically create scoring criteria that take into account the rubrics for the free-form writing questions, and the artificial intelligence can automatically and objectively score based on those scoring criteria.

[0112] <Example 10 (in English, considering the rubric)> For the English questions and correct answers in Example 8, "10 points" was used as the marking information, and a marking criterion was created after instructing the subject to take the rubric into consideration. Specifically, the prompt was "Please create the marking criterion taking the rubric into consideration." As a result, the marking criterion created by the marking criterion creation unit of the artificial intelligence is shown in FIG. 44.

[0113] As can be seen from these results, the automatic scoring system for free-form writing questions according to the present invention is able to use artificial intelligence to create scoring criteria that take rubrics into account, even when English is specified as the specified subject.

[0114] Next, the answers of the solvers shown in FIG. 38 were entered and graded in the same manner as in the eighth embodiment, and the graded results shown in FIG. 45 were obtained.

[0115] As can be seen from these results, with the automatic scoring system for free-form writing questions of the present invention, even when English is specified as the specified subject, the artificial intelligence can automatically create scoring criteria that take into account the rubrics for the free-form writing questions, and the artificial intelligence can automatically and objectively score based on those scoring criteria. [Explanation of symbols]

[0116] 1. Automatic scoring system for free-form writing questions 10 Terminal 11 Output section 12 Input section 20 Network 30 Free-form question automatic marking server 31 Scoring Criteria Creation Department 32 Scoring Department

Claims

1. An automatic scoring system for free-form questions that uses artificial intelligence to automatically score answers given by test takers to free-form questions about a specified test subject, The artificial intelligence is a marking criterion creation unit that creates marking criteria according to a solution process from the free-writing problem to the correct answer, based on free-writing problem information indicating the free-writing problem related to the predetermined test subject, correct answer information indicating a correct answer to the free-writing problem, and marking information indicating a maximum number of marks for the free-writing problem; and A marking unit that marks the answer of the answerer from answer information indicating the answer of the answerer based on the marking criteria. The system is characterized by an automatic scoring system for free-form writing questions.

2. The scoring criteria creation unit, Constraint information indicating conditions to be taken into consideration when creating the scoring criteria; free-writing question information indicating the free-writing questions related to the predetermined test subject; Correct answer information indicating a correct answer to the free description question information; Based on the score information indicating the score number for the free description question information, 2. The free-form question automatic marking system according to claim 1, further comprising a marking criterion that corresponds to the solution process from the free-form question to the correct answer.

3. 3. The system for automatically marking free-form questions according to claim 2, wherein the constraint information is a predetermined rubric.

4. the marking criterion creation unit creates the marking criterion taking into consideration an estimated thought process that is not shown in the free-description question information and the correct answer information but is estimated to be considered consciously or unconsciously by a human being from the free-description question to the correct answer, 4. The free description question automatic marking system according to claim 1.

5. The inferred thought process is at least one of metacognition, critical thinking, logical thinking, and intuitive thinking.

5. The system for automatically marking free-form questions according to claim 4.

6. The grading criteria creation unit creates the grading criteria taking into account a rubric.

4. The free description question automatic marking system according to claim 1.

7. 2. The system for automatically marking free-form questions according to claim 1, wherein the marking criterion creation unit creates the marking criterion so that a multiple of the number of the solution processes is equal to the marking information.

8. The scoring criteria created by the scoring criteria creation unit are modified by a human being.

2. The system for automatically marking free-form questions according to claim 1.

9. The answer of the respondent is handwritten on an answer sheet, 2. The automatic writing question marking system according to claim 1, further comprising an answer information extracting unit for extracting the answer information from the handwritten information on the answer sheet.

Citation Information

Patent Citations

  • Language generation model based on composition theory feature classifier and writing support method

    CN112182210A

  • Point rating support device, and control method and control program therefor

    JP2019113591A

  • Report evaluation device, report evaluation method, and program

    JP2021140426A

  • System and method for automatically scoring answer of sentence to question

    JP2023086037A

  • JPP7521860B

Cited By

  • Automatic scoring system for written questions, automatic scoring method for written questions, and program

    JP7858895B1

  • Scoring support device, program

    JP7859724B1

  • Scoring device, scoring method, and program

    JP7879999B1