Grading automation assistance system, program, and method

Abstract

To provide a system for generating an automatic grading reference for realizing complete grading automation of Japanese descriptive questions.SOLUTION: This grading automation assistance system comprises: answer sentence acquisition means 401 for acquiring an answer sentence taken out as a sample from all answer sentences of descriptive questions to be graded and a manual visual grading result; grading reference acquisition means 405 for acquiring a grading reference for automatic grading; automatic grading means 407 for automatically grading the answer sentence on the basis of a newest grading reference; grading result comparison means 408 for comparing the visual grading result with the automatic grading result; grading reference update means 409 for updating the grading reference when there remains an answer sentence in which the comparison result shows unmatching; and certified grading reference output means 410 for certifying the used grading reference as a grading reference among the questions when a matching rate between the visual grading result and the automatic grading result of all the answer questions exceeds a prescribed threshold value. The grading reference includes a grading reference character string and a grading logic.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a system for creating automatic marking criteria to realize complete automation of marking of essay-style questions in Japanese. [Background technology]

[0002] The number of essay-style questions is on the rise, but most of them are marked manually. There are many problems with this method of marking. First, a large number of grading staff must be gathered, but given the labor shortage, it is not easy to recruit the necessary number of people. Secondly, there is the possibility of some variation depending on the grader. For example, if the correct answer is "There is a lot of precipitation" and the answer is "It rains a lot," one grader may judge that the content is understood despite the difference in wording and mark it as correct, while another grader may mark it as incorrect because even junior high school students do not know the term "precipitation." If grading is done according to a set grading standard, it is thought that such variation is unlikely to occur. In order to solve these problems, attempts have been made to perform scoring automatically, and the invention disclosed in Patent Document 1 is one example of such an approach. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-185754 A Summary of the Invention [Problem to be solved by the invention]

[0004] The invention of Patent Document 1 recognizes the problems of automatic grading using a neural network and attempts to overcome them. Conventional methods using neural networks use pre-graded data as teacher data, learn as a binary classification problem of correct and incorrect answers, and have a second grader utilize the learning results. However, even if answer sheets for a certain number of people are manually graded and learned as graded data, and the learning results are applied to answer sheets for the remaining number of people, there is no guarantee of accuracy. In addition, partial points cannot be given because of the binary classification of correct and incorrect answers. Even if the binary classification is changed to a multi-value classification, the accuracy will decrease. To solve this problem, the invention of Patent Document 1 performs morphological analysis on the character string information (example correct answer) that is the basis for scoring and the character string information to be scored (the examinee's answer) to calculate the correspondence between the morphemes. Based on the calculated correspondence, the similarity between the example correct answer and the answer is calculated, and the score for the answer is calculated based on the information on the calculated similarity. In other words, the algorithm of this invention tries to determine whether a question is correct or incorrect based on the correspondence between the morphemes contained in the character string information of the example correct answer and the answer. However, test takers' answers usually contain sentences and words that are not related to whether the answer is right or wrong, and in such cases, morpheme correspondence is often ineffective in narrowing down the answers.

[0005] In consideration of these problems, the inventors thought that it would be more efficient to search for correct answer examples in the answer text, rather than targeting all character strings in the answer text. In searching for correct answer examples, they decided to create a scoring standard that enables full automation by using an existing intent information extraction process (disclosed in Japanese Patent Application No. 2021-79401). [Means for solving the problem]

[0006] In order to solve the above problems, the system of the present invention comprises: An answer acquisition means for acquiring answer sentences taken as samples from all answer sentences of the descriptive questions to be graded and manual visual grading results; A scoring criterion acquisition means for acquiring a scoring criterion for automatic scoring; An automatic marking means for automatically marking the answer sheet based on the latest marking criteria; A scoring result comparison means for comparing the visual scoring result with the automatic scoring result; A marking standard update means for updating the marking standard when the result of the comparison shows that the answer sheet does not match; a finalized grading criterion output means for finalizing the grading criterion used as the grading criterion for the question when the coincidence rate between the visual grading results and the automatic grading results for all answer sentences exceeds a predetermined threshold; The present invention is characterized by comprising: The answer referred to here is not an image of the handwritten answer, but data that has been converted into digital form using OCR or by hand.

[0007] The scoring criteria includes a scoring criteria string and a scoring logic. The scoring criteria string is classified into a sentence type and an exact match type. The sentence type may include an alternative string with the same meaning.

[0008] Furthermore, the automatic marking means may extract a marking standard character string that matches the answer sentence, and apply the extracted character string to a marking logic to determine whether the answer sentence is correct. Effect of the Invention

[0009] A portion of the answers from all test takers are used as samples to create a scoring standard for each question that can withstand automatic grading. This allows the majority of the remaining answers, other than the samples, to be scored automatically with a high degree of accuracy. This significantly reduces the amount of manpower required for grading. Mock exams often repeatedly use questions from previous years. If part-time students were recruited to grade exams every time a mock exam was held, the know-how of grading would not be passed on. However, in the present invention, the grading criteria are stored in a database. When the same questions are asked again several years later, the grading criteria can be used almost unchanged. This reduces the burden on the organizers of the mock exam, allowing them to focus their efforts on creating new questions, etc. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a flow chart showing an outline of a process according to an embodiment. [Diagram 2] FIG. 11 is a diagram illustrating a scoring criteria character string constituting the scoring criteria according to the embodiment. [Diagram 3] FIG. 2 is a diagram for explaining a scoring logic constituting the scoring criteria according to the embodiment. [Figure 4] FIG. 11 is a diagram for explaining an intention information extraction process according to the embodiment. [Diagram 5] 1 is a diagram showing a system configuration and a functional block configuration of an information processing device according to an embodiment; [Figure 6] FIG. 2 is a diagram illustrating answer data according to an embodiment. [Figure 7] FIG. 11 is a diagram illustrating an example of an automatic marking result (in the case of a correct answer) according to the embodiment. [Figure 8] FIG. 13 is a diagram illustrating an example of an automatic marking result (in the case of a wrong answer) according to the embodiment. [Figure 9] FIG. 13 is a diagram illustrating a comparison of scoring results according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] With reference to the drawings, the following describes a system according to an embodiment of the present invention (hereinafter, "the system"). The following will be explained in order: 1. Overview of the system and explanation of key concepts 1-1. Overview of this system 1-2. Explanation of the scoring criteria 1-3. Overview of intent information extraction processing 2. System Configuration 3. Operation of this system 4. Use of the deliverables of this system

[0012] 1. Overview of the system and explanation of key concepts 1-1. Overview of this system The outline of this system will be explained with reference to FIG. In the following explanation, the answers of all examinees are referred to as "all answers," and the answer sentences used as samples from all answers to create the marking criteria are simply referred to as "answer sentences." Sample answer sentences are extracted from all answer sentences, and are manually graded. The answer sentences and the visual grading results are associated with each other and stored in a database (F1). Create the initial marking criteria and save it in the database (F2). The answer is automatically graded according to the grading criteria. (F3) The automated scoring results are compared with the visual scoring results to determine whether the match rate is 100% (F4). If there is a mismatch between the automatic and visual marks on any answer sheet (No in F4), update the marking criteria so that they match (F5). Then, return to F3 and automatically mark using the updated marking criteria. Repeat the F3-->F4-->F5 loop until the match rate is 100%. If the automatic and visual scoring results match 100% (Yes in F4), the scoring criteria used at that time is determined as the deliverable of this system (F6).

[0013] 1-2. Explanation of the scoring criteria The scoring criteria is the most important concept for the present system. The scoring criteria consists of a scoring criteria string and a scoring logic.

[0014] The scoring criteria is a string that must be included in the answer to be judged as correct. There are two types of scoring criteria: sentence type and exact match type, as shown in Figure 2. If the correct answer is "The temperature is high and the precipitation is heavy," then the possible scoring criteria strings are "Temperature," "Precipitation," "Temperature is high," and "Heavy precipitation." Of these, "Temperature is high" and "Heavy precipitation" are sentence-type strings. If exact matches were used as a search criterion, the number of searches for sentence-type strings would be significantly reduced. Since the answer may contain expressions such as "It is hot" and "The precipitation is high," alternative expressions that correspond to these variations in expression are also included in the scoring criteria strings. On the other hand, "temperature" and "precipitation" are single nouns and are treated as exact match strings. The search criteria are that the string is identical to "temperature" and "precipitation". However, alternative strings may also be included. For example, "temperature" may also be included as an alternative to "temperature", and the search criteria are that the string is identical to "temperature" or "temperature".

[0015] When this system first starts, it is sufficient to have a set of expected character strings prepared as the marking criteria. However, suppose that as a result of automatic marking, an answer sentence in which "There is a lot of rain" is written instead of "There is a lot of precipitation" is judged to be an incorrect answer by the automatic marking, which does not match the visual marking result (which is judged to be the "correct answer"). In this case, the marking criteria is updated by adding "There is a lot of rain." In other words, when trying to match the automatic marking result with the visual marking result (which is considered to be more reliable than the automatic marking result), deficiencies in the marking criteria become apparent, so the marking criteria are updated and refined to correct these deficiencies. The reference character string may include character strings that should not be included in the correct answer. For example, "the temperature is low" or "dry" are examples of the above correct answers.

[0016] On the other hand, the marking logic is the logic for determining whether an answer is correct or incorrect based on the combination of marking standard character strings, as shown in Figure 3. This determination logic differs for each question. Depending on the question, character strings that must not be included in an answer (NG elements) may be included in the marking standard character string, and answers containing these NG elements may be penalized and marked as partially correct or incorrect.

[0017] 1-3. Overview of intent information extraction processing The determination of the correctness of each answer is divided into two steps: first, a search for the marking criteria string, and second, determination of correctness based on the marking logic. Existing intent information extraction processing is used to search for the scoring criteria string in the first half. This is a type of AI, but it is natural language processing that does not rely on machine learning. There is a recent trend to often use machine learning for natural language processing. However, machine learning such as deep learning requires the collection of a large amount of training data and learning, which requires a large amount of manpower. The subject of this invention is an extremely limited range of Japanese expressions, namely answer sentences for each question, and there is no need to learn a large amount in advance. The latter half of the process, which determines whether the answer is correct or not, is a kind of expert system, because the judgment logic of the question creator, an expert, is transferred to an information processing device and realized.

[0018] The intention information extraction process for searching for a scoring reference character string will be described below. Here, "intention information" refers to information that the recipient of the expression made by the other party intends to extract from the other party's expression. In this system, it refers to correct answer examples that the software that receives the answer of the examinee attempts to extract from the answer for the purpose of determining whether the answer is correct or incorrect. For example, the correct answer examples "There is a lot of precipitation" and "The temperature is high" in the above example and their alternative expressions correspond to intention information. In this system, when a match is established between all or part of an answer sentence and any of the marking standard character strings, it is determined that the answer sentence contains the marking standard character string, i.e., the correct answer example.

[0019] The algorithm for the intention information extraction process will be described with reference to FIG. Answers must be written in Japanese and may be a sentence (simple, compound, complex, or compound sentence), one or more words, or a combination of adjectives and nouns. First, each answer sentence is divided into morphemes by morpheme analysis. A morpheme is the smallest unit that has meaning in a language, and morpheme information such as morpheme divisions, parts of speech of morpheme units, original forms, and conjugated forms are subjected to syntactic analysis. Syntactic analysis refers to morpheme information to determine the relationship between subject and predicate, and to analyze dependency relationships and anaphoric relationships between clauses. The answer sentence is broken down into one or more simple sentences based on the results of syntactic analysis. Here, the reason for simplifying is that words uttered by humans (answer sentences in this invention) often have complex structures or subjects and predicates are often separated, which can easily lead to misunderstanding the intention of people other than the person in question, so even complex sentences are divided into easy-to-understand simple sentences to prevent this. Incidentally, since existing technologies can be used for morphological analysis and syntactic analysis, detailed explanations will be omitted.

[0020] On the other hand, the marking criteria character string is extracted from the database that stores the marking criteria, and each character string in the marking criteria is subjected to morphological and syntactic analysis by the parser, and is broken down into simple sentences. The morphological, syntactic, and simple sentence processing is the same as for the answer sentence. Each scoring standard string is classified into three types based on the results of syntactic analysis: (1) a "declarative sentence type" that combines a specific predicate with multiple complements or modifiers, (2) a "one-word focus type" that consists of a single word, and (3) an "adjective focus type" that consists of a combination of a specific adjective and a noun. A judgment model is automatically generated according to the classified type. The decision model is a general term for a decision class and a decision expression. A decision class corresponds to a class in an object-oriented language, and a decision expression corresponds to an instance. The answer sentence after the monotony is applied to each judgment model and a matching process with the answer sentence is performed. As a result of the matching process, zero or more marking criteria strings contained in the answer sentence are extracted. The marking logic is applied to these to judge whether the answer sentence is correct or incorrect. Next, we will explain the success or failure of matching for each of the three types mentioned above.

[0021] { Matching of "declarative sentence types"} If the marking criteria string (after monolingualization) is a narrative type, a match is determined to be established if all elements that make up the decision formula of the marking criteria string are included in the answer text, and the "category" and "structure" of each element also match the analysis information of the corresponding word in the answer text. The word analysis information also includes information such as negation (e.g., "It's not raining," "Not much," etc.) and requests (e.g., "Please prepare (an umbrella)") inferred from information obtained as a result of syntactic analysis. The present invention does not stop at simple keyword matching, but also performs matching that takes into account analysis information, making it possible to extract intent information with high accuracy.

[0022] In the scoring criteria string for the "narrative type," words belonging to one of the following three "categories" become components of the judgement formula. ·predicate Complement (Note: The subject is considered a type of complement.) · Modifiers (Note: There are predicative and attributive modifiers.) For example, the scoring criterion string "Lots of rain falls all year round" is broken down as follows: Falling 《Predicate》 <------- All year round 《modifying a predicate》 <--(ga)-- rain {complement} <--(of)-- a lot {modifying a complement} In this example sentence, the words "large amount," "rain," "all year round," and "fall," which belong to one of the three categories mentioned above, namely, predicate, complement, or modifier, are components of the judgement expression.

[0023] A match is established when the "category" of each component of the reference string, its "word" and "structure" all match the elements of the answer after monocultural testing. For example, in the keyword "large amount," the "category" = (modifier), "word" = (large amount), and "structure" = (relationship that modifies the complement "rain"). Here, "construction" refers to the linking relationship between a complement and a predicate, or the linking relationship between a modifier and a modified word. If the answer sentence is "It rains all year round or snows a lot," then "a lot" modifies the complement "snow," so it cannot be said to match with the reference string "a lot" that modifies the complement "rain." Therefore, the match is not established and the answer sentence is not judged to be correct. Also, if the answer sentence is "It rains," the match is not established. This is because the answer sentence is missing the elements of the reference string, "all year round" and "a lot."

[0024] In addition, suffixes that indicate a style that does not belong to any of the "categories" of complement, modifier, or predicate (e.g., "desu", "dearu", "da"), particles (e.g., "no" and "ga" in "otoriyori ame ga"), and conjunctions are not components of the test formula. Therefore, if the answer is "Heavy rain falls all year round" or "Heavy rain falls all year round", matching will be successful because it satisfies all the elements of the marking criteria string above.

[0025] { Matching for "One word focus type"} In the "One Word Focus Type," a single noun, adjective, and adverb are registered as a scoring reference string. A match is made only when the same word is included in any part of the answer sheet.

[0026] { Matching of "adjective focus type"} The scoring criteria for the "adjective focus type" match combinations of specific adjectives and the nouns they modify, but the matching patterns differ depending on the type of adjective. For example, if "tai tem- perature" is registered as a grading standard string, the system will successfully match the following patterns: "tem- perature is high" in which the answer sentence is expanded into a declarative sentence, "tem- perature is high" in which the particle is omitted, and "tai tem- perature" in which the answer sentence is converted into a noun. On the other hand, "not high temperature" does not match because the negation of "high" is used, "not high." In this way, in the matching process, information such as negation is also compared along with the keywords, and this negation information is found in the results of syntax analysis. This "adjective focus type," along with the "declarative type," falls under the "sentence type" in the scoring criteria, and many of the "single word focus type" types fall under the "exact match type."

[0027] Above, we have explained the outline of the intention information extraction process, which is the key to the automatic marking process. Here, we will add a comment about Figure 4. In Figure 4, the answer text and the marking standard character string are shown as if they are processed simultaneously, but this is for the convenience of explanation. For the processing in this system, please refer to the explanation of the configuration and operation of this system described later.

[0028] 2. System Configuration The configuration of this system will be described below with reference to FIG. A system control computer (hereafter referred to as "server") 1 is connected to user terminals 2 used by users via a communications network N. Here, the "user" refers to the person who creates the data for the marking criteria, and is primarily the person who creates the questions. The person who creates the questions will understand the gist of the questions better than anyone else, and will be able to accurately judge whether the answers written by the examinees are right or wrong, even if the answers vary. Such question creators are the best suited to both mark the questions and to create criteria for automatic marking.

[0029] The user terminal 2 is an information processing device that enables a user of the present system to create new or update marking criteria, upload them to the server 1, and download marking results from the server 1. The user terminal 2 has an input unit 21 such as a keyboard or mouse, and an output unit 22 such as a printer or screen. Through the input unit 21, the user creates new or updates marking criteria. Through the output unit 22, the user can view answer sheets or the results of automatic marking based on the marking criteria. In addition, it also includes a communication interface unit (not shown) that enables communication with the server 1, etc., and a storage unit (not shown) that stores various data and programs for processing.

[0030] In addition to a memory unit 3 and a processing unit 4, the server 1 also includes a communication interface unit (not shown) for enabling connection to a user terminal 2 or other information processing devices, and an input / output unit (not shown) used by an operator of the server 1, etc.

[0031] The memory unit 3 has an answer database (hereinafter referred to as "answer DB") 31 which stores the answers of the examinees, and a marking criteria database (hereinafter referred to as "marking criteria DB") 32 which stores the marking criteria, and also stores various programs, data in the middle of processing, etc.

[0032] 5, the processing unit 4 of the server 1 includes an answer text acquisition unit 401, a morphological analysis unit 402, a syntax analysis unit 403, an answer text simple classification unit 404, a marking criterion acquisition unit 405, a judgment model generation unit 406, an automatic marking unit 407, a marking result comparison unit 408, a marking criterion update unit 409, and a final marking criterion output unit 410. Each of these units is realized by the CPU executing a program stored in the storage unit 3. Note that the block configuration of the processing unit 4 is shown for the convenience of explanation, and is not limited thereto.

[0033] The answer acquisition unit 401 acquires the answer sentences and the manual visual grading results stored in the answer sentence DB 31. As shown in FIG. 6, the answer sentence DB 31 stores the examinee's answer sentences and the visual grading results in association with automatically generated IDs, and the stored data is only referenced. The process of extracting answer sentences from all answer sentences and registering them in the answer sentence DB 31 may be executed by the server 1 or by a computer other than the server 1. Also, the answer sentence DB 31 may be stored in a data server other than the server 1.

[0034] The morpheme analysis unit 402 divides the input answer sentence and the marking reference character string into morphemes. The syntactic analysis unit 403 refers to morpheme information to determine the relationship between subject and predicate, and analyzes dependency relationships and anaphoric relationships between phrases. When the original answer sentence is a complex sentence, compound sentence, or overlapping sentence, the answer sentence simple division 404 breaks it down into one or more simple sentences based on the results of syntax analysis.

[0035] The marking criteria acquisition unit 405 acquires marking criteria for automatic marking. The marking criteria data input from the input unit 21 of the user terminal 2 is acquired via the communication network N. The marking criteria DB 32 stores the initial marking criteria, marking criteria in the process of being revised, and completed marking criteria.

[0036] The judgment model generation unit 406 performs syntax analysis on the marking reference string, classifies it into one of "declarative sentence type", "one word focus type" and "adjective focus type" based on its grammatical features, and generates a judgment formula from the judgment class corresponding to the type. The morphological and syntactic analysis of the marking reference string is the same as that of the answer text.

[0037] The automatic marking unit 407 automatically marks the answer sentence retrieved from the answer sentence DB 31 based on the latest marking criteria retrieved from the marking criteria DB 32. This automatic marking process can be divided into a first half of a matching process and a second half of an automatic marking result output process. In the first half of the process, the above-mentioned intention information extraction process is used to determine whether the answer sentence matches the marking criteria character string, and in the second half of the process, marking logic is applied to the matching result to perform automatic marking, and whether the answer sentence is correct, partially correct, or incorrect is output.

[0038] The marking result comparison unit 408 compares the visual marking result with the automatic marking result for each answer sentence. The comparison result is either a match or a mismatch. A match refers to three patterns in which both results are correct, both are partially correct, and both are incorrect. Any other pattern is treated as a mismatch.

[0039] The marking criteria update unit 409 updates the marking criteria when a mismatched answer sentence remains as a result of the comparison by the marking result comparison unit 408. The accuracy of the marking criteria must be sufficient to fully automate marking. Therefore, if the visual marking results and the automatic marking results do not match for all answer sentences, the realization of full automation is questionable. Therefore, every time the marking criteria are updated from the user terminal 2, this is stored in the marking criteria DB 32 as the latest marking criteria. Note that information identifying the user who updated the marking criteria (initials, ID, etc.) may also be stored. Incidentally, most of the update processing is the addition of substitute strings for the grading standard string. For example, suppose that "precipitation amount" is written as "soil volume" and is marked as an incorrect answer, which does not match the visual grading result that marked it as correct. In order to treat it as a correct answer in automatic grading, "soil volume" can be added as an exact match type grading standard string. Also, if "soil volume" is not found in any of the sample answer sentences, it may be considered to be a similar or synonymous string to "precipitation amount" or "soil volume", and "soil volume" may be added to the grading standard string. In this way, semantically identical expressions are set as substitute strings with reference to the visual grading result. Here, which strings are deleted or added as substitute strings depend on the know-how accumulated by the user. This system, in which expert users create the grading criteria, is a kind of expert system. Also, in (Logic 3) of Figure 3, if two elements appear, it is determined to be a correct answer. However, if all three elements A, B, and C are present, the answer "There are deserts because there is little rainfall and it is dry," can be considered correct, and the rest can be considered partially correct or incorrect. In short, it is desirable to grade all the answers of all examinees using a consistent grading standard.

[0040] When the coincidence rate between the visual grading results and the automatic grading results of all answer sentences exceeds a predetermined threshold, the finalized grading criterion output unit 410 finalizes the grading criterion used for the automatic grading at that time as the grading criterion for the question. It is appropriate to set the threshold at 100%. The scoring criteria may be stored on a data server or made available commercially.

[0041] 3. Operation of this system This system is run for each question, since the marking criteria differ for each question. To create a marking rubric, a predetermined number of sample answers are extracted for each question. For example, if grading criteria are created using 5% of all answer sheets as samples, the remaining 95% of the answer sheets can be graded by computer without human intervention. The realization of this complete automation is premised on the fact that the sampled answer sheets can be graded with 100% accuracy. To achieve this, the grading criteria must be perfect, and this system makes it possible to create such grading criteria. It is desirable to select samples that reflect the distribution of marks in the population. Also, the proportion of all answers to be selected can be determined appropriately.

[0042] Although it is possible to take an online exam or even an on-site exam using a tablet device, most exams still require students to write on answer sheets with a writing implement. Therefore, the answers, which are handwritten differently by each examinee, must be digitized. To digitize the answers, they can be read using OCR or scanned and converted into PDF format. Existing character recognition software can also be used. Recognition of handwritten answers is not essential to the present invention, so a detailed explanation is omitted.

[0043] Sample answer sentences extracted from all answer sentences are manually graded, and the answer sentences and the visual grading results are associated and stored in the answer sentence DB 31. The user downloads the sample answer sentences to the user terminal 2 as appropriate and displays them on the screen 22 (see FIG. 6).

[0044] The marking criteria acquired from the user terminal are stored in the marking criteria DB 32. In FIG. 2 showing an example of marking criteria character strings, "required" (4a) indicates that the answer must contain the wording. In this example, the correct answer "Temperatures are high and there is a lot of precipitation" (4b) is a compound sentence, so it is divided into two simple sentences, "Temperatures are high" (4c1) and "There is a lot of precipitation" (4c2). Both "Temperatures are high all year round" (4d1) and "There is a lot of precipitation" (4d2) are regarded as alternative expressions and are set as correct answer strings. What character strings are required and what expressions are acceptable as alternative expressions for the correct answer depend on the know-how that the grader has cultivated.

[0045] The answer sentence is automatically graded according to the grading criteria. The automatic grading is processed by the CPU executing the intention information extraction program implemented in the server 1 (in the functional block example of FIG. 5, it is mainly executed by the automatic grading unit 407). FIG. 7 shows the automatic grading result of the answer sentence "Because the temperature is high and the precipitation is very high." This is divided into two simple sentences, "The temperature is high" and "The precipitation is very high." This answer sentence contains both "Temperature" and "Precipitation," which are required character strings in the grading criteria. Furthermore, it contains both "Temperature is high" and "Precipitation is high" of the correct answer example 1. Therefore, the automatic grading result is "correct answer." The answer sentence contains a word "very" that is not in the correct answer examples, but it contains "precipitation" and "high" from the correct answer examples, so a match is established, that is, it is automatically graded as "correct answer." FIG. 8 shows an example of an automatic scoring result of "incorrect answer." The answer sentence "It's humid, and it feels like rain" does not contain the required words "temperature" and "precipitation." Moreover, it can be divided into two simple sentences, "humid" and "rainy," but neither of them matches the correct answer examples, "temperature is high" and "rainy." Therefore, it is automatically marked as an "incorrect answer."

[0046] The automated scoring results are compared with the visual scoring results. Figure 9(1) shows the comparison results. The answer sentence in the rectangle, "There is a lot of rain," was a semi-correct answer when visually marked. Points were subtracted because "the temperature is high" was not mentioned. However, in the automatic marking, it was marked as an incorrect answer because "precipitation" was not included. However, "the amount of rain is heavy" has the same meaning as "the amount of precipitation is high", so the visual scoring result is more appropriate. Therefore, "the amount of rain is heavy" is added to the correct answer example sentence of the scoring standard string. "Amount of rain" may be added as an alternative string for the exact match type string "amount of precipitation".

[0047] In this way, if there are any inconsistent answers, the marking criteria are updated to match the visual marking results. Once the marking criteria have been updated, the answers are automatically marked again and compared with the visual marking results. The screen that displays the comparison results after the update may display the history as shown in Figure 9(2). In Figure 9(2), the answer sentence "There is a lot of rain" has been marked as a partial correct answer by automatic marking. As a result, when compared with the visual marking results, it has changed from a mismatch to a match. This is expressed as "X-->O." If the updated scoring criteria result in 100% consistency, the scoring criteria used at that time will be the output of this system. The loop of updating the scoring criteria ---> automatic scoring ---> comparing the scoring results ---> updating the scoring criteria is repeated until the visual scoring results and the automatic scoring results match 100%.

[0048] 4. Use of the deliverables of this system The general usage is to extract the remaining answer sheets other than the sample in a format that can be processed electronically, and automatically grade them using the saved grading criteria. Here, visual checking is omitted. This is because the grading criteria have been created to the point where visual checking is no longer necessary. If visual checking were to be necessary in the first place, the main purpose of this invention, which is to reduce the number of people required for grading, would not be achieved.

[0049] Mock exams can also be taken online at home using one's own information processing device (PC, tablet, smartphone, etc.). If the grading criteria created by this system are prepared, the grading results can be returned quickly after the exam is completed. This meets the needs of test-takers who want to know their grades immediately.

[0050] Additionally, the finalized marking criteria can be saved together with the question text for future mock tests. The better the mock test questions, the more likely they are to be reused. For this reason, the question text and marking criteria are saved as a set. However, depending on the question, the correct answer may change slightly over time, so in that case, the existing marking criteria can be updated as appropriate.

[0051] The above-described embodiment is merely an example, and various modifications are possible without departing from the spirit of the invention as defined in the claims. Furthermore, the present invention can be used not only in mock exams for junior high and high school students, but also in various written exams such as mock exams for various qualification exams that require students to answer questions in a written format. [Industrial Applicability]

[0052] The present invention is expected to be utilized in various examination industries as a system that enables automatic marking of essay-style questions, which are expected to become more common in future examination formats. [Explanation of symbols]

[0053] 1: Server 3: Storage part 31: Answer text DB 32: Scoring criteria DB 4: Processing section 401: Answer text acquisition section 402: Morphological analysis section 403: Parser 404:Answer text monoculture section 405: Scoring Criteria Acquisition Department 406: Decision model generation unit 407: Automatic scoring section 408: Comparison of Scoring Results 409: Scoring Criteria Update Department 410: Final scoring criteria output section 2: User terminal 21: Input section 22: Output section N: Communication network

Claims

1. An answer acquisition means for acquiring answer sentences taken as samples from all answer sentences of the descriptive questions to be graded and manual visual grading results; A scoring criterion acquisition means for acquiring a scoring criterion for automatic scoring; An automatic marking means for automatically marking the answer sheet based on the latest marking criteria; A scoring result comparison means for comparing the visual scoring result with the automatic scoring result; A marking standard update means for updating the marking standard when the result of the comparison shows that the answer sheet does not match; a finalized grading criterion output means for finalizing the grading criterion used as the grading criterion for the question when the coincidence rate between the visual grading results and the automatic grading results for all answer sentences exceeds a predetermined threshold; A scoring automation support system comprising:

2. The scoring automation support system according to claim 1 , wherein the scoring criteria includes a scoring criteria character string and a scoring logic.

3. 3. The scoring automation support system according to claim 2, wherein the scoring reference character strings are classified into a sentence type and an exact match type, and the sentence type can include an alternative character string having the same meaning.

4. The automatic marking support system according to claim 2, characterized in that the automatic marking means extracts a marking standard character string that matches the answer sentence, applies the extracted character string to a marking logic, and determines whether the answer sentence is correct or incorrect.

5. Computer An answer acquisition means for acquiring answer sentences taken as samples from all answer sentences of the descriptive questions to be graded and manual visual grading results; A scoring criterion acquisition means for acquiring scoring criteria for automatic scoring; an automatic marking means for automatically marking the answer sheet based on the latest marking criteria; A scoring result comparison means for comparing the visual scoring result with the automatic scoring result; a marking standard update means for updating the marking standard when the comparison result indicates that there is a discrepancy in the answer sheet; a finalized grading criterion output means for finalizing the grading criterion used as the grading criterion for the question when the coincidence rate between the visual grading results and the automatic grading results for all answer sentences exceeds a predetermined threshold; and a marking automation support program for executing the marking automation support program.

6. Computer an answer acquisition step for acquiring answer sentences taken as samples from all answer sentences of the descriptive questions to be graded and manual visual grading results; A scoring criteria acquisition step for acquiring scoring criteria for automatic scoring; an automatic grading step of automatically grading the answer based on the latest grading criteria; a scoring result comparison step of comparing the visual scoring result with the automatic scoring result; a step of updating the grading criteria when the comparison result shows that the answer sheet is inconsistent; a step of outputting a finalized grading criterion for finalizing the grading criterion used as the grading criterion for the question when the coincidence rate between the visual grading results and the automatic grading results for all answer sentences exceeds a predetermined threshold; The scoring automation support method according to the present invention is characterized by carrying out the following steps.

Images