System, program, and method for automating scoring.
The system addresses accuracy and consistency issues in automated Japanese descriptive question scoring by using morphological and syntactic analysis with intent information extraction, reducing manpower needs and ensuring consistent scoring across multiple test administrations.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- HOKUSHIN TOSHO CO LTD
- Filing Date
- 2023-08-04
- Publication Date
- 2026-05-13
AI Technical Summary
Existing automated scoring methods using neural networks for descriptive questions in Japanese face accuracy issues due to reliance on binary classification, lack of partial credit assignment, and inefficiency in handling variations in wording, leading to inconsistent scoring and high manpower requirements.
A system that utilizes a scoring criteria database, morphological and syntactic analysis, and intent information extraction to create and update scoring criteria, enabling accurate automatic scoring by matching intent information and applying expert-defined logic to determine correctness.
Reduces manpower requirements by enabling high-accuracy automatic scoring, ensures consistent scoring standards, and allows for the reuse of scoring criteria across multiple test administrations, thereby reducing labor and enhancing efficiency.
Smart Images
Figure 0007857723000001 
Figure 0007857723000002 
Figure 0007857723000003
Abstract
Description
Technical Field
[0001] The present invention relates to a system for creating an automatic scoring standard for realizing complete automation of scoring descriptive questions in Japanese.
Background Art
[0002] The number of descriptive questions is on the increase, but most of their scoring is done manually. There are various problems with this manual scoring. First, a large number of scorers must be gathered, but it is not easy to recruit the necessary number of people when there is a shortage of manpower. Second, there is a problem that there may be some variation among scorers. For example, when the correct example is "precipitation is high" and the answer is "it rains frequently", one scorer may judge that it is a correct answer because the expression is different but the content is understood, while another scorer may judge it as an incorrect answer because a middle school student may not know the term "precipitation". If scoring is done according to a predetermined scoring standard, such variation is considered less likely to occur. In order to solve these problems, attempts have been made to perform scoring automatically, and the invention disclosed in Patent Document 1 is also an example of this.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] The invention described in Patent Document 1 recognizes the problems with automated scoring using neural networks and aims to overcome them. Conventional methods using neural networks involve training a second-classification problem of correct and incorrect answers using pre-scored data as training data, and then applying the learning results to the second-classifier. However, even if a certain number of answer sheets are scored manually, used as training data, and then the learning results are applied to the remaining number of answer sheets, there is no guarantee of accuracy. Furthermore, because it is a binary classification of correct and incorrect answers, partial credit cannot be given. Even if the binary classification is changed to a multi-class classification, the accuracy will decrease. To solve this problem, the invention described in Patent Document 1 performs morphological analysis on the string information that serves as the scoring criterion (correct answer example) and the string information that is to be scored (the examinee's answer), and calculates the correspondence between the morphemes. Based on the calculated correspondence, it calculates the similarity between the correct answer example and the answer, and calculates the score of the answer based on the calculated similarity information. In other words, the algorithm of this invention attempts to determine correctness based on the correspondence between the morphemes contained in the respective string information of the correct answer example and the answer. However, test-takers' answers typically contain sentences and words that are irrelevant to whether the answer is correct or incorrect. In such cases, relying solely on morphological correspondences is often ineffective and wasteful.
[0005] In light of these problems, the inventors considered it more efficient to search for whether a correct answer example is included in the answer text, rather than searching for every character in the answer text. In this search for correct answers, the inventors decided to create scoring criteria that enable full automation by utilizing existing intent information extraction processing (disclosed in Japanese Patent Application No. 2021-79401). [Means for solving the problem]
[0006] To solve the aforementioned problems, the system of the present invention is: It includes a scoring criteria database that stores the initial scoring criteria prepared for the first automatic scoring, and also stores the updated scoring criteria whenever the scoring criteria are updated. A means for obtaining answer texts, which includes sample answer texts extracted from all answer texts of the written questions to be graded, and the results of manual visual grading. For the first automatic scoring, the initially prepared scoring criteria are used; for subsequent automatic scoring, the most recently updated scoring criteria are used from the aforementioned scoring criteria database. Scoring criterion acquisition means for acquisition, The acquired Automatic scoring means for automatically scoring the answer text based on the scoring criterion, Scoring result comparison means for comparing the visual scoring result and the result of the automatic scoring, When there are remaining answer texts for which the result of the comparison is inconsistent, The scoring criteria database stored Scoring criterion update means for updating the scoring criterion, Determined scoring criterion output means for determining the used scoring criterion as the scoring criterion for the question when the coincidence rate between the visual scoring results and the automatic scoring results of all answer texts exceeds a predetermined threshold Equipped with, The automatic scoring means performs automatic scoring based on the scoring criteria acquired by the scoring criteria acquisition means until the scoring criteria are determined. It is characterized by the above. The answer text referred to here is not an image of a handwritten answer, but data digitally converted by OCR or manually.
[0007] The scoring criterion includes a scoring criterion string and a scoring logic. The scoring criterion string is classified into a sentence type and an exact match type, and the sentence type may include alternative strings of the same meaning.
[0008] Furthermore, the automatic scoring means may extract a scoring criterion string that matches the answer text, apply the extracted string to the scoring logic, and determine the correctness of the answer text.
Effect of the Invention
[0009] Using a part of the answers of all examinees as samples, create a scoring criterion that can withstand automatic scoring for each question. Therefore, the remaining majority of answers other than the samples can be automatically scored with high accuracy. As a result, the manpower required for scoring can be significantly reduced. In simulation tests, past exam questions are often repeatedly presented. If part-time jobs of students, etc. are recruited as scoring staff every time a simulation test is conducted, the know-how of scoring will not be inherited. However, in the present invention, the scoring criteria are stored in a database. When the same questions are presented several years later, the scoring criteria can be used almost as they are. For the organizer of the simulation test, the burden is reduced, and the labor can be diverted to the creation of new questions, etc.
Brief Description of the Drawings
[0010] [Figure 1] It is a flowchart showing the outline of the process according to the embodiment. [Figure 2] It is a diagram for explaining the scoring criterion string that constitutes the scoring criteria according to the embodiment. [Figure 3] It is a diagram for explaining the scoring logic that constitutes the scoring criteria according to the embodiment. [Figure 4] It is a diagram for explaining the intention information extraction process according to the embodiment. [Figure 5] It is a diagram showing the system configuration and the functional block configuration of the information processing apparatus according to the embodiment. [Figure 6] It is a diagram exemplifying the answer sheet data according to the embodiment. [Figure 7] It is a diagram exemplifying the automatic scoring result (in the case of a correct answer) according to the embodiment. [Figure 8] It is a diagram exemplifying the automatic scoring result (in the case of an incorrect answer) according to the embodiment. [Figure 9] It is a diagram for explaining the comparison of the scoring results according to the embodiment.
Modes for Carrying Out the Invention
[0011] Regarding the system of the embodiment of the present invention (hereinafter, "this system"), while referring to the drawings, It will be described in the following order. 《1. Explanation of the Outline and Main Concepts of this System》 〈1-1. Outline of this System〉 〈1-2. Explanation of Scoring Criteria〉 <1-3. Overview of Intent Information Extraction Process> 《2. System Configuration》 《3. Operation of this system》 《4. Regarding the use of deliverables from this system》
[0012] 《1. Overview of this system and explanation of its main concepts》 <1-1. Overview of this system> The system's overview will be explained with reference to Figure 1. In the following explanation, the answer documents of all examinees will be referred to as "all answer documents," and the answer documents used as samples from these all answer documents for creating the scoring criteria will simply be referred to as "answer documents." Sample answers are extracted from all submitted answers, manually graded, and the answers and the grading results are linked and stored in a database (F1). Create the initial scoring criteria and save them in the database. (F2) The answer will be automatically graded according to the grading criteria. (F3) The automated scoring results are compared with the manually scored results, and it is determined whether the agreement rate is 100% (F4). If there are any answer statements where the automated scoring result and the manual scoring result do not match (press F4 to press No), update the scoring criteria to make them match (press F5). Then, return to F3 and perform automated scoring using the updated scoring criteria. Repeat the F3-->F4-->F5 loop until the matching rate reaches 100%. If the automated scoring result and the manual scoring result match 100% (Yes, press F4), the scoring criteria used at that time will be finalized as the deliverable of this system. (F6)
[0013] <1-2. Explanation of Grading Criteria> The scoring criteria are the most important concept for this system. This scoring criterion consists of a scoring criterion string and scoring logic.
[0014] The scoring criteria string is a string that must be included in the answer in order for it to be judged as a correct answer. As shown in Figure 2, this string can be in the form of a sentence or an exact match. If the correct answer is "The temperature is high and the rainfall is heavy," then "temperature," "rainfall," "high temperature," and "heavy rainfall" could be the strings used for scoring. Of these, "high temperature" and "heavy rainfall" are sentence-type strings. If exact matching is used as the search condition for sentence-type strings, the number of results will be significantly reduced. Since the answer may also contain expressions such as "it is hot" or "the rainfall is heavy," alternative expressions that accommodate these variations in wording should also be included in the scoring criteria strings. On the other hand, "temperature" and "precipitation" are treated as single nouns and require an exact match. The search condition is that the result must be the same string as "temperature" or "precipitation". However, alternative strings may be included. For example, "temperature" can be included as an alternative to "temperature," and the search condition would be that the result must be the same string as "temperature" or "temperature."
[0015] At the start of this system, it is sufficient to have a set of expected strings prepared as scoring criteria. However, suppose that the automated scoring results show that an answer that says "a lot of rain" instead of "a lot of precipitation" is marked as incorrect by the automated scoring, which does not match the result of manual scoring (which is judged as "correct"). In that case, the scoring criteria are updated by adding "a lot of rain". In other words, when trying to make the automated scoring results match the manual scoring results (which are considered more reliable than the automated scoring results), shortcomings in the scoring criteria become apparent, so the scoring criteria are updated and refined to compensate for these shortcomings. Note that the reference string may include strings that should not be included in the correct answer. For example, "low temperature" or "dry" would be examples of correct answers in the above example.
[0016] On the other hand, the scoring logic, as shown in Figure 3, is the logic for determining correctness based on combinations of scoring criterion strings. This judgment logic differs for each question. Depending on the question, strings that should not be included in the answer (NG elements) may be included in the scoring criterion string, and answers containing these NG elements may be penalized, resulting in a partial correct answer or an incorrect answer.
[0017] <1-3. Overview of Intent Information Extraction Process> The correctness of each answer is determined in two stages: first, searching for the scoring criteria string, and second, determining correctness using the scoring logic. The search for scoring criteria strings in the first half utilizes existing intent information extraction processing. This is a type of AI, but it is natural language processing that does not rely on machine learning. The current trend is to often utilize machine learning in natural language processing. However, machine learning such as deep learning requires the collection and training of a large amount of training data, which requires a large amount of human effort. The subject of this invention is Japanese expressions within an extremely limited range, namely the answer texts for each question, and there is no need to train a large amount of data in advance. Furthermore, the true / false judgment process in the latter half is a kind of expert system. This is because it transfers the judgment logic of the expert question setter to an information processing device.
[0018] The following describes the process for extracting intent information when searching for scoring criteria strings. Here, "intent information" refers to the information that the recipient of an expression intends to extract from that expression. In this system, it refers to the correct answer examples that the software, upon receiving a test-taker's answer, attempts to extract from the answer for the purpose of determining its correctness. For example, the correct answer examples "high rainfall" and "high temperature" in the above example, as well as their alternative expressions, constitute intent information. In this system, when a match is found between all or part of each answer statement and any of the scoring criteria strings, it is determined that the answer statement contains the scoring criteria string, i.e., the correct answer example.
[0019] The algorithm for extracting intent information will be explained according to Figure 4. The answer must be written in Japanese and may consist of a sentence (simple, compound, complex, or redundant), one or more words, or a combination of an adjective and a noun. Each answer is first divided into morphemes through morphological analysis. A morpheme is the smallest meaningful unit in a language, and morphological information such as the morpheme's boundaries, part of speech, base form, and conjugated form is subjected to syntactic analysis. Syntactic analysis involves determining subject-predicate relationships and analyzing dependency relationships and anaphora between clauses by referring to morphological information. Based on the results of syntactic analysis, the answer is broken down into one or more simple sentences. The reason for breaking it down into simple sentences is that human speech (answers in this invention) often has complex structures, and the subject and predicate are often far apart, making it easy for anyone other than the speaker to misunderstand the intent. To prevent this, even complex sentences are divided into easily understandable simple sentences. Regarding morphological and syntactic analysis, existing technologies can be used, so detailed explanations will be omitted.
[0020] On the other hand, the scoring criteria string is retrieved from the database storing the scoring criteria, and each string in this scoring criteria is subjected to morphological and syntactic analysis by the parser, and is broken down into simple sentences. The processing of morphological analysis, syntactic analysis, and simple sentences is the same as in the case of the answer text. Each scoring criterion string is classified into three types based on the results of syntactic analysis: (1) the 'predicative type,' which combines a specific predicate with multiple complements and modifiers; (2) the 'single-word focus type,' which consists of a single word; and (3) the 'adjective focus type,' which consists of a combination of a specific adjective and a noun. A judgment model is automatically generated according to the classified type. A decision model is a concept that encompasses both decision classes and decision expressions. Decision classes are equivalent to classes in object-oriented languages, and decision expressions are equivalent to instances. The unified answer text is applied to each judgment model, and a matching process is performed against the answer text. As a result of the matching process, zero or more scoring criterion strings contained in the answer text are extracted. Scoring logic is applied to these to determine whether the answer text is correct or incorrect. Next, we will explain the success or failure of matching based on the three types mentioned above.
[0021] { Matching by 'Descriptive Text Type'} If the scoring criteria string (after monogramming) is of the descriptive type, a match is determined to be successful if all elements constituting the evaluation formula of that scoring criteria string are included in the answer text, and the 'category' and 'structure' of each element also match the parsed information of the corresponding word in the answer text. The word analysis information includes information such as negation (e.g., "it won't rain," "not much") and requests (e.g., "please prepare (an umbrella)") inferred from the information obtained from syntactic analysis. This invention goes beyond simple keyword matching and incorporates analysis information into the matching process, allowing for highly accurate extraction of intent information.
[0022] In the scoring criteria string for the "Descriptive Text Type," words belonging to one of the following three "Categories" become components of the judgment formula. ·predicate • Complement (Note: The subject is considered a type of complement) • Modifiers (Note: There are adverbial modifiers and attributive modifiers) For example, the scoring criterion string "It rains heavily all year round" can be broken down as follows: to fall (predicate) <------- All year round (modifies predicates) <--(ga)-- Rain (complement) <--(of)-- Large quantity (modifies a complement) In this example sentence, the words "a large amount," "rain," "all year round," and "fall" belong to one of the three categories mentioned above: predicate, complement, or modifier, and these are the components of the judgment formula.
[0023] A successful match occurs when the 'category,' 'word,' and 'structure' of each component of the reference string all match the elements of the unified answer. Taking the keyword "large quantity" as an example, the 'category' is (modifier), the 'word' is (large quantity), and the 'structure' is (a relationship that modifies the complement "rain"). Here, "structure" refers to the linking relationship between a complement and a predicate, or between a modifier and the word it modifies. If the answer is "It rains or snows heavily all year round," then "heavy" modifies the complement "snow," and therefore does not match the base string "heavy" which modifies the complement "rain." Thus, the match is not found, and the answer is not judged as correct. Also, if the answer is "It rains," the match is not found because the element "all year round" and "heavy" from the base string are missing from the answer.
[0024] Furthermore, sentence endings that do not belong to any of the categories of complements, modifiers, or predicates (e.g., "...desu," "...dearu," "...da"), particles (e.g., "no" and "ga" in "taoryo ame ga"), and conjunctions are not components of the judgment formula. Therefore, if the answer is "It rains heavily all year round" or "It rains heavily all year round," then all the elements of the scoring criteria string mentioned above are met, and the matching is successful.
[0025] { Matching based on the 'single word focus type'} In the "Single Word Focus Type" method, nouns, adjectives, and adverbs are each registered as single word strings for scoring. Matching is only successful if the same word appears anywhere in the answer text.
[0026] { Matching based on the 'Adjective Focus Type'} The scoring criteria string for the "Adjective Focus Type" matches specific adjectives with the nouns they modify, but the matching patterns differ depending on the adjective type. For example, suppose "high temperature" is registered as a scoring criterion string. The answer will successfully match all of the following patterns: "high temperature" (where the answer is expanded into a descriptive sentence), "high temperature" (where the particle is omitted), and "relatively high temperature" (where it is nominalized). On the other hand, "not high temperature" does not match because it uses the negation of "high," which is "not high." In this matching process, information such as negation is also checked along with the keywords, and this negation information can be found in the syntactic analysis results. This "adjective-focused type," along with the "descriptive type," falls under the "sentence type" category in the scoring criteria, while most of the "single-word-focused type" falls under the "exact match type."
[0027] The above outlines the intent information extraction process, which is the core of the automated scoring process. A note regarding Figure 4: Figure 4 shows the answer text and scoring criteria string being processed simultaneously, but this is for illustrative purposes only. For details on the system's processing, please refer to the description of the system's configuration and operation, which will be discussed later.
[0028] 《2. System Configuration》 The configuration of this system will be explained below in reference to Figure 5. The system management computer (hereinafter referred to as "server") 1 connects to user terminals 2 used by users via a communication network N. Here, "user" refers to the person who creates the scoring criteria data, mainly the question creator. The question creator will understand the intent of the question better than anyone else and will be able to accurately judge the correctness of the answers, even if the test-takers' answers vary. Such a question creator is well-suited to both be a grader and to create the criteria for automated scoring.
[0029] User terminal 2 is an information processing device used by users of this system to create or update scoring criteria, upload them to server 1, and download scoring results from server 1. User terminal 2 is equipped with an input unit 21 such as a keyboard and mouse, and an output unit 22 such as a printer and screen. Through the input unit 21, users can create or update scoring criteria. Through the output unit 22, users can view answer sheets and results automatically scored based on the scoring criteria. It also includes a communication interface unit (not shown) that enables communication with Server 1 and other devices, and a storage unit (not shown) that stores various data and programs during the processing.
[0030] Server 1 includes a storage unit 3, a processing unit 4, a communication interface unit (not shown) for connecting to user terminals 2 and other information processing devices, and an input / output unit (not shown) used by the operator of Server 1.
[0031] The memory unit 3 includes an answer document database (hereinafter referred to as "answer document DB") 31 that stores the examinees' answer documents and a scoring criteria database (hereinafter referred to as "scoring criteria DB") 32 that stores the scoring criteria, as well as storing various programs and data in progress.
[0032] As shown in Figure 5, the processing unit 4 of server 1 includes an answer text acquisition unit 401, a morphological analysis unit 402, a syntactic analysis unit 403, an answer text monographing unit 404, a scoring criteria acquisition unit 405, a judgment model generation unit 406, an automatic scoring unit 407, a scoring result comparison unit 408, a scoring criteria update unit 409, and a final scoring criteria output unit 410. Each of these units is realized by the CPU executing a program stored in the memory unit 3. Note that the block configuration of the processing unit 4 is for illustrative purposes only and is not limited to this.
[0033] The answer document acquisition unit 401 acquires the answer documents stored in the answer document DB 31 and the results of manual visual grading. As illustrated in Figure 6, the answer document DB 31 stores the examinee's answer documents and the visual grading results in association with automatically generated IDs, and the stored data is only referenced. The process of extracting answers from all submitted answers and registering them in the answer database (DB31) can be performed on Server 1 or on a computer other than Server 1. Furthermore, the answer database (DB31) may reside on a data server other than Server 1.
[0034] The morphological analysis unit 402 divides the input answer text and scoring criteria string into morphemes. The syntactic analysis unit 403 refers to morphological information to determine the relationship between subject and predicate, and to analyze dependency relationships and anaphora relationships between clauses. Answer Text Section 404 breaks down the original answer text into one or more simple sentences based on the results of syntactic analysis, if the original answer text is a complex, compound, or redundant sentence.
[0035] The scoring criteria acquisition unit 405 acquires scoring criteria for automatic scoring. It acquires the scoring criteria data entered from the input unit 21 of the user terminal 2 via the communication network N. The scoring criteria DB32 stores the initial scoring criteria, the scoring criteria in progress, and the completed scoring criteria.
[0036] The judgment model generation unit 406 syntactically analyzes the scoring criteria string and classifies it into one of three types based on its grammatical characteristics: 'descriptive sentence type', 'single word focus type', or 'adjective focus type'. It then generates a judgment expression from the judgment class corresponding to the type. The morphological and syntactic analysis of the scoring criteria string is the same as for the answer text.
[0037] The automatic scoring unit 407 automatically scores the answer texts retrieved from the answer text DB31 based on the latest scoring criteria retrieved from the scoring criteria DB32. This automatic scoring process can be divided into a matching process in the first half and an automatic scoring result output process in the second half. In the first half, the success or failure of the matching between the answer text and the scoring criteria string is determined using the intent information extraction process described above, and in the second half, the scoring logic is applied to the matching result to perform automatic scoring and output whether the answer text is correct, partially correct, or incorrect.
[0038] The scoring result comparison unit 408 compares the visual scoring result with the automated scoring result for each answer. The compared results are either identical or non-identical. Identical means that both results are correct, both are partially correct, or both are incorrect. Any other pattern is treated as non-identical.
[0039] The scoring criteria update unit 409 updates the scoring criteria if, as a result of the comparison by the scoring result comparison unit 408, there are still answer statements that do not match. The accuracy of the scoring criteria must be sufficient to enable fully automated scoring. Therefore, if the visual scoring result and the automated scoring result do not match for all answer statements, the realization of full automation will be questionable. For this reason, each time the scoring criteria are updated from the user terminal 2, this is stored in the scoring criteria DB 32 as the latest scoring criteria. Information identifying the user who updated the scoring criteria (initials, ID, etc.) may also be stored. By the way, most of the update process involves adding alternative strings to the scoring criteria strings. For example, suppose a student wrote "water flow" instead of "precipitation" and it was marked as incorrect, which contradicts the visually judged correct answer. To treat it as correct in the automated scoring system, "water flow" can be added as an exact match scoring criterion string. Also, if the term "concentration" is not found in any of the sample answer sentences, it can be considered a similar or synonymous string to "precipitation" or "water flow," and "concentration" can be added to the scoring criterion string. In this way, the system uses the visual scoring results as a reference to set alternative strings that are semantically the same. Here, the decision of which strings to delete or add as alternative strings depends on the know-how accumulated by the user. This system, where users, who are experts, create the scoring criteria, is a kind of expert system. Also, in Figure 3 (Logic 3), if two elements appear, it is judged as a correct answer. However, it is acceptable to consider "There is a desert because there is little rainfall and it is dry," which contains all three elements A, B, and C, as the correct answer, while other answers may be considered partially correct or incorrect. In short, it is desirable to grade all test-takers' answers using a consistent scoring standard.
[0040] The final scoring criteria output unit 410 finalizes the scoring criteria used for automatic scoring at the time if the agreement rate between the visual scoring results and the automatic scoring results for all answer texts exceeds a predetermined threshold. It is appropriate to set the threshold to 100%. These scoring criteria are stored on a data server or made available commercially.
[0041] 《3. Operation of this system》 This system is executed for each question because the scoring criteria differ for each question. To create grading criteria, a predetermined number of sample answers are extracted for each question. For example, if a scoring standard is created using 5% of all submitted answers as a sample, the remaining 95% of the answers can be scored by computer without human intervention. This complete automation is contingent on the sample answers being scored with 100% accuracy. To achieve this, the scoring standard must be perfect, and this system makes it possible to create such a standard. The sample should ideally be drawn to reflect the score distribution of the population. The proportion of each sample drawn from all submitted answers should also be determined appropriately.
[0042] While online exams and in-person exams using tablet devices are possible, many exams still rely on paper answer sheets with handwritten notes. Therefore, the handwritten answers, each with its own unique handwriting, must be digitized. This digitization can be achieved by using OCR (optical character recognition) or scanning and converting to PDF. Existing character recognition software can also be used. However, since the recognition of handwritten answers is not essential to this invention, a detailed explanation will be omitted.
[0043] Sample answers are extracted from all submitted answers, manually graded, and the answers and their corresponding visual grading results are saved in the Answer Database 31. Users can then download them to their user terminal 2 and display them on screen 22 (see Figure 6).
[0044] The scoring criteria obtained from the user's terminal are stored in the scoring criteria DB32. In Figure 2, which illustrates the scoring criteria strings, "Required" (4a) indicates that the answer must include certain words. The correct answer example in this case, "The temperature is high and the rainfall is heavy" (4b), is a compound sentence, so it is divided into two simple sentences, "The temperature is high" (4c1) and "The rainfall is heavy" (4c2). Both "The temperature is high all year round" (4d1) and "The rainfall is heavy" (4d2) are considered alternative expressions and are used as correct answer strings. What strings are required and what expressions are acceptable as alternatives to the correct answer depends on the expertise cultivated by the grader.
[0045] The answer is automatically graded according to the grading criteria. Automatic grading is processed by the CPU executing an intent information extraction program implemented on Server 1 (in the functional block example in Figure 5, this is mainly executed by the automatic grading unit 407). Figure 7 shows the automatic grading result for the answer "Because the temperature is high and the rainfall is very high." This is divided into two simple sentences, "The temperature is high" and "The rainfall is very high." This answer contains both "temperature" and "rainfall," which are required strings according to the grading criteria. Furthermore, it contains both "The temperature is high" and "The rainfall is high" from Correct Answer Example 1. Therefore, the automatic grading result is "Correct." The answer contains the word "very," which is not in the Correct Answer Example, but since it contains "rainfall" and "high" from the Correct Answer Example, a match is made, and it is automatically graded as "Correct." Figure 8 shows an example of an "incorrect answer" in the automated scoring results. The answer, "It's humid, like rain," does not contain the essential words "temperature" and "precipitation." Furthermore, although it can be separated into two simple sentences, "high humidity" and "like rain," neither matches the correct answer's components, "high temperature" and "high precipitation." Therefore, it will be automatically graded as an "incorrect answer."
[0046] The results of the automated scoring are compared with the results of the manual scoring. Figure 9(1) shows the comparison results. The answer statement "There is a lot of rain" within the rectangle was marked as partially correct in the visual scoring. Points were deducted because "the temperature is high" was not mentioned. However, in the automated scoring, it was marked as incorrect because "precipitation" was not included. However, "a lot of rain" and "a lot of precipitation" have the same meaning, and the visual scoring result is more appropriate. Therefore, "a lot of rain" will be added to the correct answer example sentences in the scoring criteria string. Alternatively, "a lot of rain" could be added as a substitute string for the exact match string "precipitation".
[0047] In this way, if there are any inconsistencies in the answer statements, the scoring criteria are updated to match the results of the visual grading. If the scoring criteria are updated, the system will perform automatic scoring again and compare it with the manual scoring results. On the screen displaying the updated comparison results, the history may be displayed as shown in Figure 9(2). In Figure 9(2), the answer statement "The amount of rain is large" was marked as partially correct by automatic scoring. As a result, when compared with the manual scoring results, it changed from a mismatch to a match. This is represented as "×-->〇". If the updated scoring criteria result in a 100% match, the scoring criteria used at that time will be considered the deliverable of this system. The process repeats the cycle of updating the scoring criteria --> automatic scoring --> comparing the scoring results --> updating the scoring criteria until the manual scoring results and the automatic scoring results match 100%.
[0048] 《4. Regarding the use of deliverables from this system》 The typical use case involves extracting the remaining answer texts (excluding the sample) in a format suitable for electronic processing and automatically scoring them using the stored scoring criteria. Here, visual checking is omitted, as the scoring criteria have been meticulously crafted to eliminate the need for visual checking. In fact, if visual checking were necessary, the primary objective of this invention—reducing the manpower required for scoring—would not be achieved.
[0049] The mock exam can also be taken online at home using your own information processing device (PC, tablet, smartphone, etc.). If the scoring criteria created for this system are available, the scoring results can be returned quickly after the exam is completed. This meets the needs of test-takers who want to know their scores immediately.
[0050] Additionally, the finalized scoring criteria can be saved along with the question text for future mock exams. Good mock exam questions are often reused. Therefore, it's a good idea to save the question text and scoring criteria together. However, the correct answer may change slightly over time for some questions, in which case the existing scoring criteria should be updated accordingly.
[0051] The above embodiments are merely examples. In short, the invention can be implemented in various ways, as long as it does not deviate from the spirit of the invention as described in the claims. Furthermore, this invention can be used not only in mock exams for junior high and high school students, but also in various written examinations, such as mock exams for various qualification exams that require written answers. [Industrial applicability]
[0052] This invention is expected to be utilized in various examination industries as a system that enables the automatic scoring of essay-type questions, which are anticipated to increase as a future examination format. [Explanation of Symbols]
[0053] 1: Server 3: Storage part 31: Answer text DB 32: Scoring Criteria Database 4: Processing Unit 401: Answer text acquisition section 402: Morphological analysis section 403: Parsing Unit 404:Answer text monoculture section 405: Grading Criteria Acquisition Department 406: Decision Model Generation Unit 407: Automatic scoring unit 408: Comparison of Scoring Results 409: Grading Criteria Update Section 410: Output section for final scoring criteria 2: User terminal 21: Input section 22: Output section N: Communication Network
Claims
1. A system that supports the execution of automatic scoring based on scoring criteria, It includes a scoring criteria database that stores the initial scoring criteria prepared for the first automatic scoring, and also stores the updated scoring criteria whenever the scoring criteria are updated. A means for obtaining answer texts, which includes sample answer texts extracted from all answer texts of the written questions to be graded, and the results of manual visual grading. For the first automatic scoring, the scoring criteria initially provided are obtained from the scoring criteria database, and for subsequent automatic scoring, the latest updated scoring criteria are obtained from the scoring criteria database. An automatic scoring means for automatically scoring the answer document based on the scoring criteria obtained above, A scoring result comparison means for comparing the visual scoring result with the automatic scoring result, If there are any answer documents in which the comparison results do not match, the scoring criteria update means updates the scoring criteria stored in the scoring criteria database. The system includes a confirmed scoring criteria output means that, when the agreement rate between the visual scoring results and the automated scoring results for all answer texts exceeds a predetermined threshold, confirms the scoring criteria used as the scoring criteria for that question. The automatic scoring means performs automatic scoring based on the scoring criteria acquired by the scoring criteria acquisition means until the scoring criteria are determined. A scoring automation support system characterized by the following features.
2. The scoring automation support system according to claim 1, characterized in that the scoring criteria include a scoring criteria string and scoring logic.
3. The scoring automation support system according to claim 2, characterized in that the scoring criteria strings are classified into sentence type and exact match type, and the sentence type may include alternative strings with the same meaning.
4. The automated scoring support system according to claim 2, characterized in that the automated scoring means extracts a scoring criterion string that matches the answer, applies the extracted string to the scoring logic, and determines whether the answer is correct or incorrect.
5. A computer equipped with a scoring criteria database which stores the scoring criteria initially prepared for the first automatic scoring, and which stores the updated scoring criteria each time the scoring criteria are updated. A means for obtaining answer texts, which are sample answer texts extracted from all answer texts of the written questions to be graded, and the results of manual visual grading. For the first automatic scoring, the scoring criteria obtained from the scoring criteria database are the initially prepared scoring criteria; for subsequent automatic scoring, the scoring criteria obtained are the most recently updated scoring criteria. An automatic scoring means for automatically scoring the answer document based on the scoring criteria obtained above, A scoring result comparison means for comparing the visual scoring result with the automatic scoring result, If there are any answer documents in which the results of the comparison described above do not match, a scoring criteria update means updates the scoring criteria stored in the scoring criteria database. If the agreement rate between the visually graded results and the automatically graded results for all answer statements exceeds a predetermined threshold, the system will operate as a confirmed grading criteria output means, which will then confirm the grading criteria used as the grading criteria for that question. The automatic scoring means performs automatic scoring based on the scoring criteria acquired by the scoring criteria acquisition means until the scoring criteria are determined. A scoring automation support program characterized by the following features.
6. A computer equipped with a scoring criteria database which stores the scoring criteria initially prepared for the first automatic scoring, and which stores the updated scoring criteria each time the scoring criteria are updated. The answer acquisition step involves obtaining sample answer texts from all answer texts to be graded, along with the results of manual visual grading. For the first automatic scoring, the scoring criteria initially prepared are retrieved from the scoring criteria database; for subsequent automatic scoring, the most recently updated scoring criteria are retrieved in the scoring criteria acquisition step. An automated scoring step that automatically scores the answer based on the scoring criteria obtained above, A scoring result comparison step of comparing the visual scoring result with the automatic scoring result, If there are any answer documents in which the comparison results do not match, a scoring criteria update step is performed to update the scoring criteria stored in the scoring criteria database. If the agreement rate between the visual grading results and the automated grading results for all answer texts exceeds a predetermined threshold, the final grading criteria output step is executed to finalize the grading criteria used as the grading criteria for that question. In the aforementioned automatic scoring step, automatic scoring is performed based on the scoring criteria obtained in the aforementioned scoring criteria acquisition step until the scoring criteria are finalized. A method for supporting automated scoring, characterized by the features described above.