Automatic scoring system for open-ended questions

Abstract

This invention provides an automated scoring system and method for free-response questions in which artificial intelligence automatically creates scoring criteria for free-response questions and then automatically and objectively scores the answers based on those criteria. [Solution] The artificial intelligence includes a scoring criteria creation unit 31 that creates scoring criteria according to the solution process from the free-response questions to the correct answers, based on free-response question information indicating free-response questions relating to a predetermined examination subject, correct answer information indicating the correct answers to the free-response question information, and scoring information indicating the score for the free-response question information, and a scoring unit 32 that scores the respondent's answers from answer information indicating the respondent's answers based on the scoring criteria.

Description

Technical Field

[0001] The present invention relates to a free description problem automatic scoring system in which artificial intelligence automatically creates a scoring standard for free description problems and, based on the scoring standard, the artificial intelligence automatically and objectively scores.

Background Art

[0002] Conventionally, there has been an automatic scoring system (for example, a digital scoring system, etc.) that automatically scores the answers (answers to symbol problems) of a large number of examinees / testers by artificial intelligence (AI). However, since the answers to free description problems vary widely, it is said that automatic scoring by the system is difficult, and ultimately, subjective scoring is performed according to the scoring criteria determined by the scorer or the ability of the scorer.

[0003] On the other hand, in Patent Document 1, there is a system for automatically scoring essay questions by AI. As a learning pattern structure of AI for determining the answer essay, there are up to four categories, namely, the correct answer as the correct answer, the conclusion reversed as the wrong answer, the premise or reason reversed, and off-topic. A storage means for storing in AI answer examples divided into four patterns each for the compositional means, and a determination means for causing AI to determine the approximation between the answered essay and the pattern example stored in that category and determine the probability in each category, and a determination means for determining and displaying the correct answer rate as the appropriateness for the answer to the learner's essay question. By changing the combination of the premise (subject or reason part) and the conclusion part of the essay and storing not only the correct answer but also the answer pattern for wrong answers in AI, it is possible to improve the determination accuracy by machine learning with a small number of patterns and significantly reduce the human check. An automatic scoring system for essay questions has been proposed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

[0005] However, currently available automated scoring systems use AI to automatically score only handwritten symbols and numbers. They cannot automatically score English words, kanji, etc., and ultimately, a human (scorer) visually determines the correct answer.

[0006] Furthermore, while the aforementioned automated scoring system for written answer questions can improve the accuracy of correct answer determination and significantly reduce the need for human (scorer) checks, it has the drawback of not being able to perform automated scoring completely.

[0007] Furthermore, even with the automated scoring system for written answer questions mentioned above, objective scoring requires a human to create scoring criteria each time, depending on the given question. This process of creating scoring criteria requires considerable effort and time, which is a significant problem.

[0008] Furthermore, even with human-based scoring, when multiple scorers grade the same problem, there are issues such as bias in the scoring results due to the individual abilities of the scorers, and the need to spend an enormous amount of time before scoring in order to improve inter-rater reliability.

[0009] In view of the circumstances described above, the present invention aims to provide an automated scoring system and method for free-response questions in which artificial intelligence automatically creates scoring criteria for free-response questions and automatically and objectively scores the answers based on those scoring criteria.

[0010] Furthermore, since the solution process leading to the correct answer can be used as a scoring criterion in this invention, it becomes possible to understand what stage a respondent's answer has reached at this point, what knowledge and skills are the respondent's challenges, and where the challenges lie for the group of respondents.

[0011] Furthermore, this invention will make it possible to easily include free-response questions, which examiners have hesitated to include due to the time and effort required for grading, as needed. This will revolutionize the way questions are presented in all examinations and dramatically improve the academic abilities of all humanity. It has the potential to significantly change education in all fields around the world that require free-response answers in examinations.

[0012] Furthermore, these problems (issues) do not preclude the existence of other problems. Also, each aspect of the present invention described later does not need to solve all of these problems (issues). Moreover, it is possible to extract other problems (issues) from the description, drawings, or claims. [Means for solving the problem]

[0013] As a result of diligent research into the aforementioned problems, the inventors of this invention have discovered the following groundbreaking automated scoring system for free-response questions.

[0014] A first aspect of the present invention for solving the above problems is an automated free-response question scoring system that uses artificial intelligence to automatically score a respondent's answer to a free-response question relating to a predetermined examination subject, wherein the artificial intelligence comprises: a scoring criteria creation unit that creates scoring criteria according to the solution process from the free-response question to the correct answer based on free-response question information indicating a free-response question relating to a predetermined examination subject; correct answer information indicating the correct answer to the free-response question; and scoring information indicating the score for the free-response question; and a scoring unit that scores the respondent's answer from answer information indicating the respondent's answer based on the scoring criteria.

[0015] Here, "free-response questions" refer to questions in which the answer must be provided not just as a single word, but as a complete sentence. Examples include short essays, English compositions, questions requiring the handwritten description of calculation processes in mathematics (arithmetic), science (biology, chemistry, physics, earth science) on an answer sheet with a free-response section, free-response questions requiring explanations in social studies (geography, history, ethics, political economy), and other types of questions used in university and qualification exams. Furthermore, "solution process" refers to the steps taken to arrive at the correct answer to a problem.

[0016] According to this first embodiment, an automated scoring system for free-response questions can be provided in which artificial intelligence automatically creates scoring criteria according to the solution process, and the artificial intelligence automatically and objectively scores the answers based on those scoring criteria.

[0017] A second aspect of the present invention is an automatic free-response question scoring system according to the first aspect, characterized in that the scoring criteria creation unit creates scoring criteria according to the solution process from the free-response question to the correct answer, based on constraint information indicating conditions to be considered when creating scoring criteria, free-response question information indicating free-response questions relating to a predetermined examination subject, correct answer information indicating the correct answer to the free-response question information, and scoring information indicating the number of points scored for the free-response question information.

[0018] According to this second embodiment, an automated scoring system for free-response questions can be provided in which artificial intelligence automatically creates scoring criteria (that satisfy constraint information) for free-response questions, and the artificial intelligence automatically and objectively scores the answers based on those scoring criteria.

[0019] A third aspect of the present invention is the automatic scoring system for free-response questions according to the second aspect, characterized in that the constraint information is a predetermined rubric.

[0020] Here, "rubric" refers to a scale of several levels that indicates the degree of success, and more specifically, it refers to an assessment tool for measuring the degree of learning achievement that has the following three characteristics. (1) Visualize the evaluation viewpoints and scales in a list table. (2) It is used to judge the degree of achievement of learning goals. (3) It is a tool that serves as the judgment criterion for absolute evaluation, showing the "viewpoints" and "scales" of evaluation in a matrix table.

[0021] Note that the rubric of this aspect is created by those who provide guidance or support (such as teachers, lecturers, professors, etc.).

[0022] According to such a third aspect, an automatic scoring system for free-description questions can be provided, in which an artificial intelligence automatically creates scoring criteria that meet the rubric for free-description questions, and based on the scoring criteria, the artificial intelligence automatically and objectively scores the free-description questions. As a result, those who provide guidance or support (such as teachers, lecturers, professors, etc.) and learners can share the evaluation criteria, and fair and highly transparent evaluation can be carried out.

[0023] A fourth aspect of the present invention is that the scoring criteria creation unit creates scoring criteria that also consider the presumed thinking process that is not shown in the free-description question information and the correct answer information but is presumed to be consciously or unconsciously considered by a human from the free-description question to the correct answer, in the free-description question automatic scoring system according to any one of the first to third aspects. That is, the scoring criteria creation unit of this aspect may create scoring criteria corresponding to the solution process from the free-description question to the correct answer based on the free-description question information indicating the free-description question related to a predetermined test subject, the correct answer information indicating the correct answer to the free-description question, the scoring information indicating the score for the free-description question, and the above-mentioned presumed thinking process. Or, based on the constraint information indicating the conditions to be considered when creating the scoring criteria, the free-description question information indicating the free-description question related to a predetermined test subject, the correct answer information indicating the correct answer to the free-description question information, the scoring information indicating the score for the free-description question information, and the above-mentioned presumed thinking process, create scoring criteria corresponding to the solution process from the free-description question to the correct answer.

[0024] According to such a fourth aspect, since the scoring criteria creation unit can create scoring criteria considering the estimated thinking process, it is possible to provide a free description question automatic scoring system that scores based on more appropriate scoring criteria.

[0025] A fifth aspect of the present invention is the free description question automatic scoring system according to the fourth aspect, wherein the estimated thinking process is at least one of metacognition (recognition of one's own thinking process), critical thinking, logical thinking, intuitive thinking, and problem-solving ability.

[0026] According to such a fifth aspect, it is possible to provide a free description question automatic scoring system that scores based on even more appropriate scoring criteria.

[0027] A sixth aspect of the present invention is the free description question automatic scoring system according to any one of the first to third aspects, wherein the scoring criteria creation unit creates scoring criteria considering a rubric. That is, the scoring criteria creation unit of this aspect may create scoring criteria according to the solution process from the free description question to the correct answer based on free description question information indicating a free description question related to a predetermined test subject, correct answer information indicating the correct answer to the free description question, scoring information indicating the score for the free description question, and the above-described rubric, or based on constraint information indicating conditions to be considered when creating the scoring criteria, free description question information indicating a free description question related to a predetermined test subject, correct answer information indicating the correct answer to the free description question information, scoring information indicating the score for the free description question information, and the above-described rubric, create scoring criteria according to the solution process from the free description question to the correct answer.

[0028] According to such a sixth aspect, since the scoring criteria creation unit can create scoring criteria considering a rubric, it is possible to provide a free description question automatic scoring system that scores the academic ability of each respondent based on more appropriate scoring criteria.

[0029] A seventh aspect of the present invention is the automatic scoring system for free-response questions according to claim 1, characterized in that the scoring criteria creation unit creates scoring criteria such that the number of multiples of the number of solution processes is equal to the scoring information.

[0030] According to this seventh embodiment, it is possible to provide an automated scoring system for free-response questions that scores respondents' answers more objectively.

[0031] An eighth aspect of the present invention is an automated scoring system for free-response questions according to the first aspect, characterized in that the scoring criteria created by the scoring criteria creation unit are modified by a human.

[0032] Here, "person" can refer not only to the grader or the creator of the free-response question, but also to a third party.

[0033] According to this eighth aspect, by having a human revise the scoring criteria, conditions (criteria) that were not considered by artificial intelligence, or conditions (criteria) that must be considered, can be reflected in the scoring criteria.

[0034] A ninth aspect of the present invention is an automatic scoring system for free-response questions according to the first aspect, characterized in that the respondent's answer is handwritten on an answer sheet, and the system further comprises an answer information extraction unit that extracts answer information from the handwritten information on the answer sheet.

[0035] According to this ninth embodiment, it is possible to provide an automated scoring system for free-response questions that automatically scores answers from handwritten responses by respondents without requiring any time or effort.

[0036] In this invention, "database," "system," and "part" do not merely refer to physical means, but also include cases where the functions of the "database," "part," or "system" are realized by software. Furthermore, even if the functions of one "database," "system," or "part" are realized by two or more physical means or devices, the functions of two or more "databases," "systems," or "parts" may be realized by one physical means or device. [Brief explanation of the drawing]

[0037] [Figure 1] Figure 1 is a schematic diagram of the automated scoring system for free-response questions according to Embodiment 1. [Figure 2] Figure 2 is a schematic diagram of the terminal of Embodiment 1. [Figure 3] Figure 3 is a schematic diagram of the automated scoring server for free-response questions according to Embodiment 1. [Figure 4] Figure 4 is a flowchart showing the operation of the automated scoring system for free-response questions in Embodiment 1. [Figure 5] Figure 5 is an image showing the information from the free-response questions (Question 1 and Question 2) of Example 1. [Figure 6] Figure 6 shows the scoring criteria for Problem 1 of Example 1. [Figure 7] Figure 7 shows the scoring criteria for Problem 2 in Example 1. [Figure 8] Figure 8 is an image showing the answer of a respondent in Example 1. [Figure 9] Figure 9 shows the scoring results for Problem 1 of Example 1. [Figure 10] Figure 10 shows the scoring results for Problem 2 of Example 1. [Figure 11] Figure 11 is an image showing the answer of a respondent in Example 2. [Figure 12] Figure 12 shows the scoring results for Problem 1 of Example 2. [Figure 13] Figure 13 shows the scoring results for Problem 2 of Example 2. [Figure 14] Figure 14 is an image showing the free-response question for Example 3. [Figure 15] Figure 15 is an image showing the correct solution for Example 3. [Figure 16] Figure 16 shows the scoring criteria for Example 3. [Figure 17] Figure 17 is an image showing the incorrect answer for Example 3. [Figure 18] Figure 18 shows the scoring results for Example 3. [Figure 19] Figure 19 shows the scoring criteria for Problem 1 in Example 4. [Figure 20] Figure 20 shows the scoring criteria for Problem 2 in Example 4. [Figure 21] Figure 21 shows the scoring results for Problem 1 of Example 4. [Figure 22] Figure 22 shows the scoring results for Problem 1 of Example 4. [Figure 23] Figure 23 shows the scoring results for Problem 1 of Example 5. [Figure 24] Figure 24 shows the scoring results for Problem 2 of Example 5. [Figure 25] Figure 25 shows the scoring criteria for Example 6. [Figure 26] Figure 26 shows the scoring results for Example 6. [Figure 27] Figure 27 is an image illustrating the problem in Example 7. [Figure 28] Figure 28 is the rubric for Example 7. [Figure 29] Figure 29 shows the scoring criteria for Example 7. [Figure 30] Figure 30 is an image showing the answers of students A to C in Example 7. [Figure 31] Figure 31 shows the scoring results for student A in Example 7. [Figure 32] Figure 32 shows the scoring results for student B in Example 7. [Figure 33] Figure 33 shows the scoring results for student C in Example 7. [Figure 34] Figure 34 shows the problem in Example 8. [Figure 35] Figure 35 shows the correct solution for Example 8. [Figure 36] Figure 36 is the rubric for Example 8. [Figure 37] Figure 37 shows the scoring criteria for Example 8. [Figure 38] Figure 38 is an image showing the solution for Example 8. [Figure 39] Figure 39 shows the scoring results for Example 8. [Figure 40] Figure 40 shows the scoring criteria for Example 9. [Figure 41] Figure 41 shows the scoring results for student A in Example 9. [Figure 42] Figure 42 shows the scoring results for student B in Example 9. [Figure 43] Figure 43 shows the scoring results for student C in Example 9. [Figure 44] Figure 44 shows the scoring criteria for Example 10. [Figure 45] Figure 45 shows the scoring results for Example 10. [Modes for carrying out the invention]

[0038] The embodiments of the automated scoring system for free-response questions according to the present invention will be described below with reference to the attached drawings. However, the present invention is not limited to the embodiments described below.

[0039] (Embodiment 1) The automatic free-response question scoring system 1 of this embodiment uses artificial intelligence (AI) that has been trained to generate solution processes for free-response questions in a predetermined examination subject based on free-response questions and their correct answers in a predetermined examination subject. The AI ​​automatically creates scoring criteria and automatically scores the respondent's answer based on those scoring criteria. As shown in Figure 1, this automatic free-response question scoring system 1 consists of a plurality of terminals 10 and an automatic free-response question scoring server 30 connected to these terminals 10 via a network 20. The network 20 is not limited to wireless or wired connections as long as it can connect the terminals 10 and the automatic free-response question scoring server 30, and examples include the internet and intranets.

[0040] Here, there may be one or more respondents. Furthermore, the prescribed examination subjects refer to subjects predetermined in elementary schools, junior high schools, high schools, universities, etc., as well as subjects related to qualification examinations such as the bar examination, patent attorney examination, judicial scrivener examination, certified public accountant examination, tax accountant examination, social insurance labor consultant examination, and administrative scrivener examination. Specific examples include English, Japanese language, mathematics (arithmetic), science, social studies (geography, history, civics), constitutional law, civil law, civil procedure law, criminal law, and criminal procedure law.

[0041] First, let's describe terminal 10. Terminal 10 is used by test takers who take an exam that includes free-response questions on a predetermined subject, or by those who instruct or support the test takers (e.g., teachers, lecturers, professors, etc.). As shown in Figure 2, terminal 10 has an output unit 11 that outputs the scoring results of the free-response question automatic scoring system 1, and an input unit 12 in which the test taker or others input free-response question information, correct answer information, and answer information into the free-response question automatic scoring system 1. Needless to say, if the correct answer information is created by artificial intelligence, it is not necessary to input the correct answer information into the input unit.

[0042] The output unit 11 is not particularly limited as long as it can output the scoring results of the free-response question automatic scoring system 1 (for example, text, video, images, audio, etc.), and examples include a liquid crystal display, a braille display, and a speaker.

[0043] The input unit 12 is not particularly limited as long as it allows respondents to input free-response question information, correct answer information, and answer information (for example, text, video, images (including scanned image files or PDF files of the answer sheet), QR codes (registered trademark), audio, etc.) into the automatic free-response question scoring system 1. Examples include keyboards, touch panels, microphones, scanners, cameras, USB memory, and file transfer via a network. Here, the correct answer information may be created in advance by artificial intelligence.

[0044] The terminal 10 is not particularly limited as long as it has the output unit 11 and input unit 12 described above, and examples include personal computers, tablet computers, and smartphones.

[0045] Next, we will explain the automated free-response question scoring server 30. As shown in Figure 3, the automated free-response question scoring server 30 is equipped with artificial intelligence (AI). This AI is trained to generate solution processes for free-response questions in a given examination subject, based on free-response questions used as training data for that subject and the correct answers for those free-response questions used as training data. Here, training data refers to the data that the AI ​​uses when performing machine learning.

[0046] Here, artificial intelligence is not particularly limited as long as it is an AI (generative AI) that can generate various types of information such as images, text, audio, program code, and structured data based on patterns and trends learned from large amounts of data. Examples include ChatGPT (registered trademark) using large-scale language models (LLMs), Microsoft Copilot, and Google's Gemini. Furthermore, the solution process for a free-response question refers to the procedure for deriving the correct answer to a free-response question from that question itself, and consists of multiple steps (processes). The solution process of the present invention may be the same as or different from a human solution process. Specific examples of the solution process will be described later.

[0047] Free-response questions for teacher data are free-response questions related to a specified examination subject. Examples include essay questions on constitutional law from past bar examinations, final exam questions from university law departments and essay questions from national examinations, free-response and descriptive questions from periodic and end-of-term exams in educational institutions such as elementary schools, junior high schools, and high schools, and entrance examination questions in schools, as well as writing questions from English proficiency tests such as the Practical English Proficiency Test and TOEIC. Correct answers for teacher data are correct answers to the free-response questions for teacher data. Examples include model answers or sample answers to the aforementioned examination questions, or model answers or sample answers from educational institutions to essay questions on constitutional law from past bar examinations.

[0048] Furthermore, the method of training artificial intelligence is not particularly limited; for example, a large-scale language model (LLM) may be used to train the AI. Here, a "large-scale language model" refers to a natural language model constructed by increasing the "computational complexity," "data volume," and "number of parameters" among language models.

[0049] Next, we will describe the artificial intelligence in detail. As shown in Figure 3, the artificial intelligence has a scoring criteria creation unit 31 and a scoring unit 32.

[0050] The scoring criteria creation unit 31 is a function in which artificial intelligence creates scoring criteria based on free-response question information indicating free-response questions relating to a given examination subject, correct answer information indicating the correct answers to the free-response question information, and scoring information indicating the scoring of the free-response question information, according to the solution process from the free-response question to the correct answer.

[0051] Here, the scoring information refers to the maximum score (maximum score) that can be awarded for correctly answering the free-response question. For example, if the maximum score is 10 points, it will show 10 points, and if the maximum score is 5 points, it will show 5 points.

[0052] The scoring criteria created by the scoring criteria creation unit 31 will have multiple combinations of solution processes and the points awarded to those solution processes. That is, they can be represented by a table consisting of solution processes and the points awarded to those solution processes. For example, solution process 1 may be awarded 1 point, solution process 2 2 points, and so on. The total points awarded to all solution processes will be equal to the scoring information (the maximum points awarded if the free-response question is answered correctly).

[0053] The points awarded to each solution process may or may not be equal, but it is preferable that the scoring information (total number of points awarded) be a multiple of the number of solution processes. In other words, it is preferable that the scoring criteria creation unit 31 create the scoring criteria so that the scoring information is equal to a multiple of the number of solution processes. Specifically, if the scoring information is "100", the scoring criteria will be created so that the number of solution processes is "10", "20", "50", and "100". With this relationship, the answerer's answer can be scored more concisely and objectively.

[0054] The scoring unit 32 is a function in which artificial intelligence scores a respondent's answer based on the answer information that shows the respondent's answer, according to the created scoring criteria. Specifically, the artificial intelligence determines from the respondent's answer how far the respondent has progressed through each solution process. Then, it scores the answer by summing the points awarded for the solution processes up to the completed solution process. In other words, if the artificial intelligence determines that the respondent's answer is correct, the answer is scored with the maximum score in the scoring information. If the AI ​​determines that the respondent has not reached the correct answer but has progressed up to a certain stage of the solution process, the answer is scored with the sum of the points awarded for the solution processes completed up to that point. As mentioned above, each solution process is a procedure for deriving the correct answer to a free-response question, so the artificial intelligence will not determine that a downstream solution process has been completed without going through an upstream solution process.

[0055] Next, the operation of the automatic free-response question scoring system 1 of this embodiment will be described. As shown in Figure 4, when the automatic free-response question scoring system 1 is activated, the artificial intelligence installed in the automatic free-response question scoring server 30 creates scoring criteria using the scoring criteria creation unit 31 based on the free-response question information, correct answer information, and scoring information (S1). Here, the created scoring criteria may be reviewed and modified by a third party, such as a scorer or the creator of the free-response question. By modifying the scoring criteria in this way, conditions (criteria) that were not considered by the artificial intelligence or conditions (criteria) that must be considered can be reflected in the scoring criteria. As a result, the automatic free-response question scoring system 1 can perform more appropriate scoring.

[0056] Subsequently, the respondent or a person instructing or assisting the respondent inputs the respondent's answer information into the input unit 12 of the terminal 10 (S2). This answer information is then transmitted to the free-response question automatic scoring server 30, and the artificial intelligence installed in the free-response question automatic scoring server 30 scores the answer information using the scoring unit 32 according to the scoring criteria described above (S3). The scoring result is then transmitted to the terminal 10 via the network 20 and displayed on the output unit 11 (S4), and the operation of the free-response question automatic scoring system 1 ends.

[0057] By operating the free-response question automatic scoring system 1 configured as described above, it is possible to provide an automatic scoring system for free-response questions in which artificial intelligence automatically creates scoring criteria according to the solution process, and the artificial intelligence automatically and objectively scores the answers based on those scoring criteria.

[0058] (Embodiment 2) In Embodiment 1, scoring criteria were created using artificial intelligence without any limitations, but the present invention is not limited thereto. For example, the automatic scoring system for free-response questions may be configured such that the scoring criteria creation unit creates scoring criteria according to the solution process from the free-response question to the correct answer, based on constraint information indicating conditions to be considered when creating the scoring criteria, free-response question information indicating free-response questions relating to a predetermined examination subject, correct answer information indicating the correct answer to the free-response question information, and scoring information indicating the scoring for the free-response question information.

[0059] Here, constraint information is not particularly limited as long as it is something that should be considered when creating scoring criteria. Examples of constraint information include "the number of solution processes must match the number of scoring information," "the structure of the answer must follow a specific format," "the use of specific keywords must be mandatory," and a predetermined rubric. The constraints may be set before the automated scoring system for free-response questions is operational, or they may be set after the automated scoring system for free-response questions is operational. Needless to say, if a predetermined rubric is used as the constraint information in this embodiment, the rubric itself must be created in advance.

[0060] By configuring the automated free-response question scoring system to include such a scoring criteria creation unit, it becomes possible to create scoring criteria that reflect the intentions (thinking processes) of those who use the automated free-response question scoring system (question creators, scorers, etc.). As a result, scoring can be performed that reflects the intentions (thinking processes) of those who use the automated free-response question scoring system.

[0061] It goes without saying that the artificial intelligence used in such an automated scoring system for free-response questions must be trained to generate solution processes for free-response questions in a given exam subject, based on the free-response questions used as training data for that subject, and the correct answers for those free-response questions, including any constraints. In fact, ChatGPT (registered trademark) has already been trained on this information.

[0062] (Embodiment 3) In Embodiment 1, the scoring criteria creation unit creates scoring criteria based on free-response question information, correct answer information for that free-response question, and scoring information, according to the solution process from the free-response question to the correct answer. However, the present invention is not limited to this.

[0063] For example, the scoring criteria creation department may create scoring criteria that take into account the estimated thought process that a person consciously or unconsciously considers when arriving at the correct answer from a free-response question, even though this process is not shown in the free-response question information or the correct answer information.

[0064] Here, "estimated thought process" refers to any steps (procedures) that are not shown in the free-response question information or the correct answer information, but are presumed to be consciously or unconsciously considered by a person in order to arrive at the correct answer from a free-response question. Examples of "estimated thought processes" include "metacognition," "critical thinking," "intuitive thinking," and "logical thinking." Here, "metacognition" is described in detail, for example, in Reference 1: "Metacognition: An Overview" by John H. Flavell (1979), Reference 2: "Metacognition and Cognitive Neuropsychology: Monitoring and Control Processes" edited by GRD McClelland, GLP Williams, and PALG Delacour (2003), and Reference 3: "Handbook of Metacognition" edited by John Dunlosky and Janet Metcalfe (2009).

[0065] Specifically, by providing the artificial intelligence with further prompts such as "Create scoring criteria that take metacognition into consideration," "Create scoring criteria that take critical thinking into consideration," "Create scoring criteria that take intuitive thinking into consideration," and "Create scoring criteria that take logical thinking into consideration," the scoring criteria creation unit of the artificial intelligence can create scoring criteria that also take these estimation thought processes into consideration.

[0066] By configuring an automated free-response question scoring system that includes a scoring criteria creation unit with such functionality, the scoring criteria creation unit can create scoring criteria that also take into account the estimated thought process, thereby providing an automated free-response question scoring system that scores based on more appropriate scoring criteria.

[0067] It goes without saying that the artificial intelligence used in such an automated scoring system for free-response questions must be trained to generate solution processes for free-response questions in a given exam subject, based on free-response questions used as training data for that subject, and the correct answers for those free-response questions, including estimated thought processes (metacognition, critical thinking, intuitive thinking, logical thinking, etc.). In fact, ChatGPT (registered trademark) has already been trained on this information.

[0068] (Embodiment 4) In Embodiment 1, the scoring criteria creation unit creates scoring criteria based on free-response question information, correct answer information for that free-response question, and scoring information, according to the solution process from the free-response question to the correct answer. However, the present invention is not limited to this.

[0069] For example, the scoring criteria creation unit may create scoring criteria that also take rubrics into consideration. In Embodiment 2, a predetermined rubric created in advance was used as constraint information, but in this embodiment, the scoring criteria creation unit is made to create scoring criteria while taking the concept of rubrics into consideration. Therefore, the automatic scoring system for free-response questions in Embodiment 2 and the automatic scoring system for free-response questions in this embodiment are completely different.

[0070] Specifically, by providing the artificial intelligence with further prompts such as "Please create scoring criteria that take the rubric into consideration," the AI's scoring criteria creation unit can create scoring criteria that also take the rubric into consideration.

[0071] By configuring an automated free-response question scoring system that includes a scoring criteria creation unit with such functionality, the scoring criteria creation unit can create scoring criteria that also take rubrics into consideration, thereby providing an automated free-response question scoring system that scores based on more appropriate scoring criteria.

[0072] It goes without saying that the artificial intelligence used in such an automated scoring system for free-response questions must be trained to generate solution processes for free-response questions in a given exam subject, based on free-response questions used as training data for that subject, including rubrics, and the correct answers for those free-response questions used as training data. In fact, ChatGPT® has already been trained on rubric information.

[0073] (Other embodiments) In the embodiments described above, the format of the respondent's answer was not particularly limited, but the respondent's answer may be handwritten on an answer sheet, such as paper. The material of the answer sheet is not limited to paper, but includes other materials.

[0074] In this case, the input unit of terminal 10 captures the answers written on the answer sheet as an image. The free-response question automatic scoring system may be configured such that at least one of the terminal and the free-response question automatic scoring server has an answer information extraction unit that extracts answer information indicating the respondent's answers from the image. Here, the answer information extraction unit is not particularly limited as long as it can extract answer information from the respondent's handwritten answers written on the answer sheet, etc. Examples of answer information extraction units include commercially available software using optical character recognition technology (including those using artificial intelligence).

[0075] In this way, by configuring an automated scoring system for free-response questions that further includes an answer information extraction unit, it is possible to automatically score handwritten answers from respondents without requiring any time or effort.

[0076] <<Examples>> The following are examples of implementations, in which ChatGPT® was used as the artificial intelligence installed in the automated scoring server for free-response questions. A commercially available personal computer connected to ChatGPT® via the internet was used as the terminal.

[0077] Furthermore, ChatGPT (registered trademark) also has an optical character recognition (OCR) function, which is the answer information extraction unit. Therefore, by inputting handwritten free-response questions, their correct answers, and the respondents' answers as image data, the free-response question information, correct answer information, and answer information can be input into the automatic free-response question scoring system.

[0078] <Example 1 (in the case of arithmetic, correct answer)> As free-response question information, we used the two questions shown in Figure 5 (the upper calculation formula: Question 1, the lower calculation formula: Question 2), and used "8 / 15" (correct answer to Question 1) and "1 / 45" (correct answer to Question 2) as the correct answer information for these questions. In addition, we used "5 points" and "10 points" as scoring information, and "5-point scale" and "10-point scale" as constraint information.

[0079] Then, this information was sent to artificial intelligence to obtain the scoring criteria (solution process (thought process)) for each of these free-response questions. These scoring criteria are shown in Figures 6 and 7.

[0080] As can be seen from these results, it has been found that, according to the automatic scoring system for free-response questions of the present invention, even when mathematics is specified as the predetermined subject, the artificial intelligence scoring criteria creation unit can automatically create scoring criteria according to the solution process.

[0081] Next, the correct answers of the respondents shown in Figure 8 were entered and scored, resulting in the scoring results shown in Figures 9 and 10.

[0082] As can be seen from these results, the automatic scoring system for free-response questions according to the present invention allows artificial intelligence to automatically create scoring criteria (reflecting constraint information) for free-response questions, even when mathematics is specified as the predetermined subject. Based on these scoring criteria, even if the correct answer is entered, the artificial intelligence can automatically and objectively score the answer.

[0083] <Example 2 (in the case of arithmetic, incorrect answer)> Using the problem and scoring criteria from Example 1, when the incorrect answers shown in Figure 11 were entered and scored, the scoring results shown in Figures 12 and 13 were obtained.

[0084] As can be seen from these results, the automatic scoring system for free-response questions according to the present invention demonstrates that even when arithmetic is specified as the subject, and an incorrect answer is entered, artificial intelligence can automatically and objectively score the answer based on the scoring criteria.

[0085] <Example 3 (English case - Incorrect answer)> The information shown in Figure 14 was used as the free-response question information, and the English text shown in Figure 15 was used as the correct answer to this free-response question. In addition, "12 points" was used as the scoring information, and "create a 12-step solution process (thought process)" was used as the constraint information. This information was then sent to artificial intelligence to obtain the scoring criteria for this written question. These scoring criteria are shown in Figure 16.

[0086] As can be seen from these results, it has been found that, according to the automatic scoring system for free-response questions of the present invention, even when English is specified as the subject, the artificial intelligence scoring criteria creation unit can automatically create scoring criteria according to the solution process.

[0087] Next, when the AI ​​is given image data of the respondent's answer (incorrect) shown in Figure 17 and prompted with "Please transcribe it," the AI ​​recognizes the handwritten characters in the image in Figure 17 and outputs "My favorite place is Shibuya. Because I can buy something. Because I can eat something." Then, when this text is scored using the free-response question automatic scoring system according to the present invention, the scoring result shown in Figure 18 is obtained.

[0088] As can be seen from these results, the automatic scoring system for free-response questions according to the present invention allows artificial intelligence to automatically create scoring criteria (reflecting constraint information) for free-response questions, even when English is specified as the subject, and to automatically and objectively score the answers based on those criteria.

[0089] <Example 4 (In the case of arithmetic, in the case of an incorrect answer, considering the estimation thought process)> For the arithmetic problem and correct answer in Example 1, scoring criteria were created after instructing the system to consider metacognition as part of the estimation thinking process. Specifically, the prompt used was: "If 1 is worth 5 points and 2 is worth 10 points, create a thinking process to arrive at the answer for 1 on a 5-point scale and for 2 on a 10-point scale. When doing so, focus on cognitive aspects (information processing) such as the respondent's 'metacognitive knowledge' and 'metacognitive activity'." As a result, the scoring criteria created by the AI ​​scoring criteria creation unit are shown in Figures 19 and 20.

[0090] As can be seen from these results, the automatic scoring system for free-response questions according to the present invention has been shown to be able to create scoring criteria that take into account metacognition, which is one of the estimation thought processes, even when mathematics is specified as the predetermined subject.

[0091] Next, the answers (correct answers) of the respondents shown in Figure 8 were entered and scored, resulting in the scoring results shown in Figures 21 and 22.

[0092] As can be seen from these results, the automatic scoring system for free-response questions according to the present invention, even when mathematics is specified as the predetermined subject, shows that artificial intelligence can automatically create scoring criteria for free-response questions that also take into account metacognition, which is one of the estimation thought processes, and that even if the correct answer is entered as the answer based on those scoring criteria, the artificial intelligence can automatically and objectively score it.

[0093] <Example 5 (In the case of arithmetic, in the case of an incorrect answer, considering the estimation thought process)> Using the scoring criteria of Example 4, the answers (incorrect) of the respondents shown in Figure 11 were entered and scored, resulting in the scoring results shown in Figures 23 and 24.

[0094] As can be seen from these results, the automatic scoring system for free-response questions according to the present invention allows artificial intelligence to automatically create scoring criteria for free-response questions that also take into account metacognition, which is one of the estimation thought processes. Based on these scoring criteria, the artificial intelligence can automatically and objectively score the answers even if incorrect answers are entered.

[0095] <Example 6 (English case, incorrect answer, considering the estimation thought process)> For the English question and correct answer in Example 3, scoring criteria were created after instructing the user to consider metacognition as part of the estimated thought process. Specifically, the prompt used was: "The following question is worth 12 points. Based on the question and model answer, create a 12-step thought process from the question to the answer, and use this as the scoring criteria. When doing so, focus on cognitive aspects (information processing) such as the respondent's 'metacognitive knowledge' and 'metacognitive activity'." As a result, the scoring criteria created by the AI ​​scoring criteria creation unit are shown in Figure 25.

[0096] As can be seen from these results, the automated scoring system for free-response questions according to the present invention has been shown to be able to create scoring criteria that take into account metacognition, which is one of the estimation thought processes, even when English is specified as the subject.

[0097] Next, similar to Example 3, the answers (incorrect) of the respondents shown in Figure 17 were entered and scored, resulting in the scoring results shown in Figure 26.

[0098] As can be seen from these results, the automatic scoring system for free-response questions according to the present invention allows artificial intelligence to automatically create scoring criteria for free-response questions that take into account metacognition, which is one of the estimation thought processes, even when English is specified as the subject. Based on these scoring criteria, the artificial intelligence can automatically and objectively score the answers even when the correct answers are entered.

[0099] <Example 7 (Mathematics - Using a "rubric" as the constraint information)> The question shown in Figure 27 was used as the free-response question information, and "-2x+19" was used as the correct answer information for this question. In addition, "10 points" was used as the scoring information, and the rubric shown in Figure 28 was used as the constraint information.

[0100] Then, this information was sent to artificial intelligence to obtain scoring criteria (solution process (thought process)) for this open-ended question. These scoring criteria are shown in Figure 29.

[0101] As can be seen from these results, the automatic scoring system for free-response questions according to the present invention can be used even when mathematics is specified as the subject and a predetermined rubric is used as a constraint, as the artificial intelligence scoring criteria creation unit can automatically create scoring criteria according to the solution process.

[0102] Next, the answers of the three respondents (students A to C) shown in Figure 30 were entered and scored, resulting in the scoring results shown in Figures 31 to 33.

[0103] As can be seen from these results, the automatic scoring system for free-response questions according to the present invention can automatically create scoring criteria for free-response questions using artificial intelligence, even when mathematics is specified as the subject and a predetermined rubric is used as a constraint. Based on these scoring criteria, the artificial intelligence can automatically and objectively score the answers.

[0104] <Example 8 (English version; using "rubric" as restriction information)> The information shown in Figure 34 was used as the free-response question information, and the English text shown in Figure 35 was used as the correct answer to this free-response question. In addition, "10 points" was used as the scoring information, and the rubric shown in Figure 36 was used as the constraint information. This information was then transmitted to artificial intelligence to obtain the scoring criteria for this essay question. These scoring criteria are shown in Figure 37.

[0105] As can be seen from these results, the automatic scoring system for free-response questions according to the present invention can be used even when English is specified as the subject and a predetermined rubric is used as a constraint, as the artificial intelligence scoring criteria creation unit can automatically create scoring criteria according to the solution process.

[0106] Next, the image data of the respondent's answer shown in Figure 38 was input to the artificial intelligence, and the text within this image data was scored using the automated free-response question scoring system according to the present invention, resulting in the scoring results shown in Figure 39.

[0107] As can be seen from these results, the automatic scoring system for free-response questions according to the present invention can automatically create scoring criteria (reflecting the rubric) for free-response questions, even when English is specified as the subject and a predetermined rubric is used as a constraint, and the artificial intelligence can automatically and objectively score the answers based on those scoring criteria.

[0108] <Example 9 (Mathematics Case - Considering Rubrics)> For the mathematics problem and correct answer in Example 7, "10 points" was used as scoring information, and scoring criteria were created after instructing the system to consider the rubric. Specifically, the prompt used was "Please create scoring criteria that take the rubric into consideration." As a result, the scoring criteria created by the AI ​​scoring criteria creation unit are shown in Figure 40.

[0109] As can be seen from these results, the automatic scoring system for free-response questions according to the present invention has been shown to enable artificial intelligence to create scoring criteria that also take rubrics into consideration, even when mathematics is specified as the subject.

[0110] Next, the answers of the three respondents (students A to C) shown in Figure 30 were entered and scored, resulting in the scoring results shown in Figures 41 to 43.

[0111] As can be seen from these results, the automatic scoring system for free-response questions according to the present invention can automatically create scoring criteria that take into account the rubric for the free-response questions, even when mathematics is specified as the subject, and the artificial intelligence can automatically and objectively score the answers based on those criteria.

[0112] <Example 10 (English version, considering rubrics)> For the English questions and correct answers in Example 8, "10 points" was used as scoring information, and scoring criteria were created after instructing the system to consider the rubric. Specifically, the prompt used was "Please create scoring criteria considering the rubric." As a result, the scoring criteria created by the AI ​​scoring criteria creation unit are shown in Figure 44.

[0113] As can be seen from these results, the automatic scoring system for free-response questions according to the present invention has been shown to enable artificial intelligence to create scoring criteria that also take rubrics into consideration, even when English is specified as the subject.

[0114] Next, similar to Example 8, the answers of the respondents shown in Figure 38 were entered and scored, resulting in the scoring results shown in Figure 45.

[0115] As can be seen from these results, the automatic scoring system for free-response questions according to the present invention can automatically create scoring criteria that take into account the rubric for the free-response questions, even when English is specified as the subject, and the AI ​​can automatically and objectively score the answers based on those criteria. [Explanation of symbols]

[0116] 1. Automatic scoring system for open-ended questions 10 devices 11 Output section 12 Input section 20 Networks 30. Automatic scoring server for free-response questions 31. Grading Criteria Creation Department 32. Grading Department

Claims

1. An automated scoring system for free-response questions that uses artificial intelligence to automatically score the answers of respondents to free-response questions related to a specified examination subject, The aforementioned artificial intelligence, A scoring criteria creation unit creates scoring criteria according to the solution process from the free-response question to the correct answer, based on free-response question information indicating the free-response question relating to the predetermined examination subject, correct answer information indicating the correct answer to the free-response question, and scoring information indicating the score for the free-response question. A scoring unit that scores the answer of the respondent based on the answer information indicating the respondent's answer, based on the aforementioned scoring criteria, An automated scoring system for free-response questions, characterized by the following features.

2. The aforementioned scoring criteria creation unit, Constraint information indicating the conditions to be considered when creating the aforementioned scoring criteria, Free-response question information indicating the free-response questions relating to the specified examination subjects, The correct answer information for the aforementioned free-response question information, Based on the scoring information showing the number of points scored for the free-response question information, The automatic scoring system for free-response questions according to claim 1, characterized in that it creates scoring criteria corresponding to the solution process from the free-response question to the correct answer.

3. The automatic scoring system for free-response questions according to claim 2, characterized in that the constraint information is a predetermined rubric.

4. The scoring criteria creation unit is characterized by creating the scoring criteria that also take into account the estimated thought process that a person is presumed to consciously or unconsciously consider when arriving at the correct answer from the free-response question, even though this process is not shown in the free-response question information and the correct answer information. An automated scoring system for free-response questions according to any one of claims 1 to 3.

5. The aforementioned estimation thinking process is characterized by comprising at least one of metacognition, critical thinking, logical thinking, intuitive thinking, and problem-solving ability. The automatic scoring system for free-response questions according to claim 4.

6. The scoring criteria creation unit is characterized by creating the scoring criteria that also take rubrics into consideration. An automated scoring system for free-response questions according to any one of claims 1 to 3.

7. The automatic scoring system for free-response questions according to claim 1, characterized in that the scoring criteria creation unit creates the scoring criteria such that the multiple of the number of solution processes is equal to the scoring information.

8. The scoring criteria created by the scoring criteria creation unit are characterized by being modified by a human. The automatic scoring system for open-ended questions according to claim 1.

9. The aforementioned respondent's answer was handwritten on the answer sheet. The automatic scoring system for written questions according to claim 1, further comprising an answer information extraction unit for extracting the answer information from handwritten information on the answer sheet.

Images