Scoring device, scoring method, and program

Abstract

Traditionally, grading written answers in Japanese language exams has been extremely time-consuming, placing a heavy burden on graders, and making consistent grading difficult due to differences in grader interpretation. Furthermore, situations requiring the grading of a large number of answers quickly, such as before classes or career guidance sessions, present a significant operational burden. [Solution] The scoring device 1 comprises a rule management unit 112 that stores scoring rules for descriptive questions in Japanese language, an answer reception unit 121 that receives answer sheet information including handwritten answers that are the answers to descriptive questions in Japanese language, a scoring unit 133 that provides the handwritten answers and scoring rules included in the answer sheet information received by the answer reception unit 121 to a generating AI and obtains scoring results from the generating AI, and a scoring output unit 141 that outputs the scoring results. By appropriately scoring answers to descriptive questions in Japanese language automatically or semi-automatically with this scoring device 1, the burden on the scorer is reduced and fair scoring becomes possible.

Description

Technical Field

[0001] The present invention relates to a scoring device that scores answers to descriptive questions in Japanese, etc.

Background Art

[0002] Conventionally, there has been a computer system that enables both improving the efficiency during scoring and improving the quality of learning during scoring (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, it has not been easy to appropriately score answers to descriptive questions in Japanese. Also, conventionally, scoring answers to descriptive questions in Japanese has been very time-consuming, imposing a large burden on scorers, and it has been difficult to achieve consistent scoring due to differences in scorers' interpretations. Further, for example, when a large amount of scoring is required in a short time before a class or career guidance, there has been a problem of high operation load.

Means for Solving the Problems

[0005] The scoring device of the first invention of the present invention includes a rule management unit that stores scoring rules for descriptive questions in Japanese, a response reception unit that receives response sheet information including a descriptive response, which is an answer to a descriptive question in Japanese and is handwritten, gives the descriptive response included in the response sheet information received by the response reception unit and the scoring rules to a generation AI, and a scoring unit that obtains a scoring result from the generation AI, and a scoring output unit that outputs the scoring result.

[0006] This configuration allows for appropriate automatic or semi-automatic scoring of answers to descriptive questions in Japanese language exams, thereby reducing the burden on graders and enabling fair scoring.

[0007] Furthermore, the scoring device of this second invention, compared to the first invention, has a rule management unit that stores common rules for two or more written questions and individual rules for each of the two or more written questions; an answer reception unit that receives answer sheet information containing written answers for each of the two or more written questions, or written answers contained in the information of two or more answer sheets containing written questions; and a scoring unit that, for each written answer for each of the two or more written questions, provides the generated AI with the written answer contained in the answer sheet information received by the answer reception unit, the common rules, and the individual rules corresponding to the written questions, and obtains a scoring result from the generated AI.

[0008] This configuration allows for accurate and semi-automatic scoring of answers to descriptive questions in Japanese language exams by applying appropriate rules, thereby reducing the burden on graders and enabling fair and highly accurate scoring.

[0009] Furthermore, the scoring device of this third invention, in addition to the first or second invention, has a rule management unit that stores two or more scoring rules, including scoring rules and scoring rules for written questions, and a scoring unit that provides written answers, scoring rules and scoring rules to a generating AI and obtains scoring results from the generating AI.

[0010] This configuration allows for appropriate automatic or semi-automatic scoring of answers to descriptive questions in Japanese language exams, thereby reducing the burden on graders and enabling fair scoring.

[0011] Furthermore, the scoring device of the fourth invention is a scoring device that, in relation to any one of the first to third inventions, comprises a scoring unit which includes a proofreading means that acquires a written answer contained in the answer sheet information received by the answer reception unit and acquires a revised written answer which is a sentence in which linguistic errors in the sentence shown in the written answer have been corrected, and a scoring means that provides the revised written answer and scoring rules to a generating AI and acquires a scoring result from the generating AI.

[0012] This configuration allows for appropriate automatic or semi-automatic scoring of answers to descriptive questions in Japanese language exams, thereby reducing the burden on graders and enabling fair scoring.

[0013] Furthermore, the scoring device of this fifth invention is a scoring device in which, for any one of the first to fourth inventions, the scoring unit provides the written answer and scoring rules to the generating AI and causes the generating AI to output the scoring result and the reason for scoring, and the scoring output unit outputs the scoring result and reason in association with the written question.

[0014] With this configuration, answers to descriptive questions in Japanese language can be appropriately graded automatically or semi-automatically, and the grading results and reasons for the grading can be output.

[0015] Furthermore, the scoring device of this sixth invention, in contrast to the fifth invention, is a scoring device in which the scoring output unit outputs the scoring result and reason in correspondence with the written question and into the answer sheet information.

[0016] With this configuration, answers to descriptive questions in Japanese language can be appropriately graded automatically or semi-automatically, and the grading results and reasons for the grading can be output.

[0017] Furthermore, the scoring device of the seventh invention further comprises an instruction management unit which stores lenient scoring information that instructs to give lenient scoring to any one of the first to sixth inventions, and a flag management unit which stores a lenient flag which identifies whether or not to give lenient scoring. The scoring unit, when the lenient flag indicates that lenient scoring should be given, also provides the lenient scoring information to the generating AI and obtains the scoring result.

[0018] This structure allows for appropriate scoring of answers to descriptive questions in Japanese language exams, depending on the context.

[0019] In addition, for the scoring device of the eighth invention, for any one of the first to seventh inventions, in the rule management unit, different scoring rules are stored in association with school identification information that identifies two or more schools each. The answer reception unit receives answer sheet information associated with the school identification information. The scoring unit acquires, from the rule management unit, the scoring rule associated with the school identification information associated with the answer sheet information, provides the scoring rule and the written answer to the generation AI, and acquires the scoring result from the generation AI.

[0020] With such a configuration, it is possible to perform appropriate scoring for the answers to the descriptive questions in Japanese according to the schools to which the applicant wishes to apply or the schools where the test is conducted.

[0021] In addition, the scoring device of the ninth invention further includes a recognition unit that acquires a descriptive answer, which is a character recognition result for the written answer included in the answer sheet information received by the answer reception unit, for any one of the first to eighth inventions.

[0022] With such a configuration, for the handwritten answers to the descriptive questions in Japanese, by appropriately performing automatic or semi-automatic scoring, the burden on the scorers can be reduced and fair scoring can be achieved.

[0023] In addition, for the scoring device of the tenth invention, with respect to the ninth invention, the descriptive answer is a vertical writing having two or more lines, and it is an instruction to acquire the sentence indicated by the descriptive answer. The instruction management unit stores recognition instruction information indicating an instruction to acquire a sentence from a handwritten vertical writing having two or more lines. The recognition unit provides the handwritten descriptive answer and the recognition instruction information included in the answer sheet information received by the answer reception unit to the generation AI, and acquires the descriptive answer, which is the character recognition result, from the generation AI.

[0024] With such a configuration, it is possible to appropriately score a handwritten vertical writing having two or more lines. [[ID=​In addition, the scoring device of the eleventh invention further includes a layout management unit that stores layout information, which is information specifying the layout of the answer sheet specified by the answer sheet information and is information specifying the area for descriptive answers, and the recognition unit includes a cutting-out means for obtaining a descriptive answer from the answer sheet information using the layout information, and a recognition means for obtaining a character-recognized descriptive answer for the descriptive answer obtained by the cutting-out means.

[0026] With such a configuration, based on the layout information of the answer sheet, the descriptive answer can be appropriately obtained, and as a result, appropriate scoring can be performed.

[0027] In addition, the scoring device of the twelfth invention, with respect to the eleventh invention, the answer sheet information is information obtained by photographing an answer sheet on which corner marks, which are marks for specifying the positions of the four corners, are printed, and the recognition unit further includes a distortion correction means for correcting the distortion of the answer sheet information using the corner marks, and the cutting-out means obtains a descriptive answer from the distortion-corrected answer sheet information using the layout information.

[0028] With such a configuration, corresponding to the distortion of the read answer sheet, etc., the descriptive answer can be appropriately obtained, and as a result, appropriate scoring can be performed.

[0029] In addition, the scoring device of the thirteenth invention, with respect to any one of the ninth to twelfth inventions, further includes a correction unit having a result output means for outputting the descriptive answer obtained by the recognition unit, and a correction reception means for receiving a correction to the descriptive answer output by the result output means, and the scoring unit gives the descriptive answer corrected by the correction unit and the scoring rules to the generation AI and obtains a scoring result from the generation AI.

[0030] With such a configuration, the recognition result can be appropriately corrected, and as a result, appropriate scoring can be performed.

[0031] Furthermore, the scoring device of the fourteenth invention is a scoring device that, in addition to any one of the nineth to thirteenth inventions, has an answer sheet information which includes a handwritten respondent identifier, and the recognition unit acquires the respondent identifier from the answer sheet information and further comprises a storage unit which stores the respondent identifier in association with the answer sheet information.

[0032] This configuration allows for the automatic association of respondents with their answer sheet information.

[0033] Furthermore, the scoring device of the fifteenth invention, in contrast to the ninth invention, is a scoring device in which the scoring unit provides the written answer before character recognition and the written answer as a result of character recognition to a generating AI, and obtains a scoring result from the generating AI.

[0034] This structure allows for more accurate scoring of answers to descriptive questions in Japanese language exams.

[0035] Furthermore, the scoring device of the sixteenth invention is a scoring device that, with respect to any one of the first to fifteenth inventions, has answer sheet information which includes a written answer and a general answer which is an answer to a correct answer identification problem which is a problem in which the correct answer is identified, and the scoring unit comprises a written scoring means which provides the written answer included in the answer sheet information and scoring rules to a generating AI and obtains a written scoring result from the generating AI, a general scoring means which determines whether the general answer included in the answer sheet information is correct and obtains a general scoring result which is the scoring result for the correct answer identification problem, and a scoring acquisition means which uses the written scoring result and the general scoring result to obtain a scoring result for the answer sheet information.

[0036] This configuration allows for the automatic or semi-automatic scoring of answer sheets that include both written answers and non-written answers.

[0037] Furthermore, the scoring device of the seventeenth invention is a scoring device in which, for any one of the first to sixteenth inventions, the answer receiving unit receives answer sheet information for two or more students, the scoring unit obtains a scoring result for each of the two or more students, and the scoring output unit outputs sorted written answers and scoring results using the scoring result as a key.

[0038] This structure allows for the proper verification of the answers of multiple respondents to descriptive questions in Japanese language, thereby enabling the explanation to proceed appropriately. [Effects of the Invention]

[0039] The scoring device according to the present invention reduces the burden on graders and enables fair scoring by appropriately scoring answers to descriptive questions in Japanese language automatically or semi-automatically.

[0040] Furthermore, the effects described above are not necessarily limited, and any of the effects described herein, or any other effects that can be inferred from this specification, may be achieved in conjunction with or in lieu of the effects described above. [Brief explanation of the drawing]

[0041] [Figure 1] Conceptual diagram of information system A in Embodiment 1 [Figure 2] Block diagram of information system A [Figure 3] Block diagram of the scoring device 1 [Figure 4] A flowchart illustrating an example of the operation of the scoring device 1. [Figure 5] A flowchart illustrating an example of the scoring process. [Figure 6] A flowchart illustrating an example of the same distortion correction process. [Figure 7] A flowchart illustrating an example of the description recognition process. [Figure 8] A flowchart illustrating an example of the correction process. [Figure 9] A flowchart explaining an example of the scoring process. [Figure 10] A diagram showing an example of the same answer sheet information. [Figure 11] A diagram showing an example of the scoring prompt. [Figure 12] A diagram illustrating an example of the scoring rules. [Figure 13] A diagram showing an example of the same handwritten answer key. [Figure 14] Figure showing an example of the same output. [Figure 15] Block diagram of the computer system [Modes for carrying out the invention]

[0042] The following describes embodiments of the scoring device and the like with reference to the drawings. Note that components denoted by the same reference numerals in the embodiments perform similar operations, and therefore, further explanation may be omitted.

[0043] (Embodiment 1) This embodiment describes a scoring device that provides scoring rules for descriptive questions in Japanese language to a generating AI, obtains the scoring results, and outputs them. The scoring rules may include, for example, individual rules for each question and common rules shared by multiple questions. The scoring rules may include, for example, rules for adding points and rules for deducting points. The answers to descriptive questions may be written vertically and consist of one or more lines.

[0044] This embodiment describes a scoring device that obtains and outputs scoring results using answers written by respondents that have been grammatically corrected. The respondent is, for example, a student. The respondent is, for example, a student of a cram school or regular school. The respondent is, for example, a student preparing for entrance exams to a junior high school, high school, or university. However, the respondent's position is not relevant.

[0045] In this embodiment, a scoring device that acquires and outputs scoring results and reasons will be described. In this embodiment, it is preferable to output the scoring results and reasons within the answer sheet information.

[0046] In this embodiment, a scoring device that obtains lenient scoring results when in lenient mode will be described.

[0047] In this embodiment, a scoring device capable of scoring using different scoring criteria for each school will be described. "Each school" refers to, for example, each individual school or each class within a school.

[0048] In this embodiment, we describe a scoring device that can recognize the handwritten answers of respondents to descriptive questions in Japanese language and score them using the character recognition results.

[0049] In this embodiment, a scoring device capable of cutting out the answer area from the answer sheet based on layout information will be described.

[0050] In this embodiment, a scoring device that corrects distortion in answer sheets, such as those scanned using a scanner, by using a marker located in the corner of the answer sheet will be described.

[0051] In this embodiment, a scoring device that assists in correcting the recognized answer when the character recognition result satisfies inappropriate conditions will be described.

[0052] In this embodiment, a scoring device that associates respondent identifiers with answer sheet information will be described.

[0053] This embodiment describes a scoring device that provides a generating AI with two or more of the extracted descriptive answers, character recognition results, and corrected answers, and obtains a scoring result.

[0054] In this embodiment, a scoring device capable of scoring a test that includes both written response questions and non-written response questions will be described.

[0055] In this embodiment, a scoring device is described that extracts written answers from two or more students, sorts them in order of grade, constructs a list, and outputs it.

[0056] In this specification, information X being associated with information Y means that information Y can be obtained from information X, or information X can be obtained from information Y, and the method of association is irrelevant. Information X and information Y may be linked, may reside in the same buffer, may information X be contained in information Y, or information Y may be contained in information X, and so on.

[0057] Furthermore, in this specification, selecting or determining information Z means obtaining information Z, obtaining a pointer to information Z, obtaining the ID of information Z, setting a flag on information Z, etc., and it is sufficient to be able to access information Z.

[0058] Figure 1 is a conceptual diagram of information system A in this embodiment. Information system A comprises a scoring device 1, one or more terminal devices 2, and one or more generation AI devices 3.

[0059] Scoring device 1 is a device that provides scoring rules for descriptive questions in Japanese language to a generating AI, obtains the scoring results, and outputs them. Scoring device 1 is usually a server, but it may also be a terminal. If scoring device 1 is a server, it may be, for example, a cloud server or an ASP server, but the type is not specified. If scoring device 1 is a terminal, it may be, for example, a smartphone, a tablet device, or a so-called personal computer, but the type is not specified. If scoring device 1 is a terminal device, terminal device 2 is not required in information system A.

[0060] Terminal device 2 is the device used by the user. The user may be, for example, the instructor, but could also be a respondent, etc. Terminal device 2 can be, for example, a smartphone, tablet device, or so-called personal computer; the type is not limited.

[0061] The generation AI device 3 is a device that has the function of a generation AI. In this context, the generation AI device 3 usually has the function of a text generation AI. The generation AI device 3 may also have an image generation function. The generation AI is, for example, ChatGPT (registered trademark) or Gemini, but is not limited to that. The generation AI device 3 is, for example, a cloud server or an ASP server, but is not limited to that type. The generation AI device 3 will be referred to as the generation AI as appropriate. Furthermore, the scoring device 1 may also have the function of a generation AI. In such a case, the generation AI device 3 is not necessary for information system A. If there are two or more generation AIs, for example, the generation AI used for character recognition and the generation AI used for scoring are different generation AIs.

[0062] Figure 2 is a block diagram of information system A in this embodiment. Figure 3 is a block diagram of scoring device 1.

[0063] The scoring device 1 comprises a storage unit 11, a reception unit 12, a processing unit 13, and an output unit 14. The storage unit 11 comprises an instruction management unit 111, a rule management unit 112, a layout management unit 113, and a flag management unit 114. The reception unit 12 comprises an answer reception unit 121. The processing unit 13 comprises a recognition unit 131, a correction unit 132, a scoring unit 133, and a storage unit 134. The recognition unit 131 comprises a distortion correction means 1311, an extraction means 1312, and a recognition means 1313. The correction unit 132 comprises a result output means 1321 and a correction reception means 1322. The scoring unit 133 comprises a calibration means 1331 and a scoring means 1332. The output unit 14 comprises a scoring output unit 141.

[0064] The terminal device 2 comprises a terminal storage unit 21, a terminal receiving unit 22, a terminal processing unit 23, a terminal transmission unit 24, a terminal receiving unit 25, and a terminal output unit 26.

[0065] The storage unit 11, which constitutes the scoring device 1, stores various types of information. These types of information include, for example, various instruction information, various scoring rules, layout information, tolerance flags, and answer sheet information, which will be described later.

[0066] The instruction management unit 111 stores various types of instruction information. Instruction information is information given to the generating AI. Instruction information may be so-called prompts. Instruction information may also be information used to train the generating AI. Examples of instruction information include OCR prompts, recognition instruction information, lenient scoring information, and corrected answer acquisition prompts.

[0067] The instruction information in the instruction management unit 111 may differ depending on the school or the respondent. In other words, the instruction information in the instruction management unit 111 may be associated with, for example, school-specific information or respondent conditions.

[0068] School-specific information refers to information that identifies one or more schools. Examples of school-specific information include school name, school identifier, and school attribute value conditions. School attribute value conditions refer to conditions related to one or more school attribute values. Examples of school attribute values ​​include school rank, academic achievement score, and location.

[0069] Respondent conditions are conditions related to the respondent. Respondent conditions include respondent attribute value conditions and respondent identifiers. Respondent attribute value conditions are conditions related to one or more respondent attribute values. Respondent attribute values ​​are, for example, the respondent's class, respondent's level, respondent's gender, respondent's age, or grade. Respondent conditions are, for example, the respondent identifier and respondent name. Note that the respondent name can also be considered a respondent identifier.

[0070] An OCR prompt is a prompt that instructs the system to obtain the character recognition results from the answer sheet information. An OCR prompt might look like this: "Your role is to copy the characters exactly as they appear on the answer sheet for the Japanese language test. Output the answers for each question, paired with the question number. Also, output the answers according to the following instructions: [Instructions]<Recognition Instructions>" Another example of an OCR prompt might look like this: "Recognize the characters on the answer sheet for the Japanese language test according to the following instructions, and output the answers for each question, paired with the question number. [Instructions]<Recognition Instructions>" In the prompt, strings enclosed in "<" and ">" are variables.

[0071] Answer sheet information refers to information obtained by photographing a test answer sheet. For example, answer sheet information might be information obtained by scanning a Japanese language test answer sheet. The answer sheet is typically a Japanese language test answer sheet. The answer sheet has one or more written answers. Preferably, the answer sheet has one or more written answers and one or more general answers. The answer sheet information is usually an image. For example, answer sheet information is information obtained by scanning an answer sheet. The data type of the answer sheet information is not limited to jpeg, gif, or pdf. Preferably, the answer sheet information is information obtained by photographing an answer sheet with corner markers printed on it, which are markers used to identify the position of the four corners of the answer sheet. The corner markers are preferably cross-shaped, but not required. The answer sheet information may include, for example, a handwritten respondent identifier.

[0072] A descriptive answer is the answer to a written response question in Japanese language. A descriptive answer is, for example, an answer written by hand. A descriptive answer is, for example, the result of character recognition of a handwritten answer. It is preferable for descriptive answers to be written vertically. It is preferable for descriptive answers to have two or more lines. A descriptive answer written by hand may be called a handwritten descriptive answer, and a descriptive answer resulting from character recognition may be called a recognized descriptive answer.

[0073] A general answer is an answer to a question with a definite correct answer. Examples of general answers include answers to multiple-choice questions (e.g., answers with symbols or numbers, answers with ○ or ×, answers on a mark sheet, etc.) and answers to questions requiring written answers in kanji.

[0074] Recognition instruction information is information that instructs the recognition of a written answer. Preferably, the recognition instruction information includes information indicating how to recognize the written answer. Preferably, the recognition instruction information includes information specifying the reading order of the characters that make up the written answer. For example, the recognition instruction information could be information indicating instructions for retrieving a sentence from a vertically written handwritten text with two or more lines. For example, the recognition instruction information might be: "Read from the rightmost column, moving to the leftmost column. Read from top to bottom in each column. If the answer has two or more columns, do not jump between columns. Contextual correction is strictly prohibited." Note that an OCR prompt can be considered to include recognition instruction information. The recognition instruction information may also be a prompt, or part of the information that constitutes the prompt.

[0075] A scoring prompt is a prompt that instructs the user to score according to the scoring rules. A scoring prompt is a prompt that instructs the user to score the answer sheet indicated by the answer sheet information. For example, a scoring prompt might be: "Score the following answer according to the scoring rules below, and output the score and reason for scoring. [Scoring Rules]<Scoring Rules> [Answer]<Written Answer>" Another example of a scoring prompt might be: "Score the following answer according to the scoring rules below, and output the score and reason for scoring. The model answer is below. [Scoring Rules]<Scoring Rules> [Answer]<Written Answer> [Model Answer]<Model Answer>"

[0076] Tolerant grading information is information that instructs for lenient grading. For example, lenient grading information might be, "Correct grammatical, stylistic, and formal errors, and calculate a fair and minimal breakdown of deductions that takes into account the elementary school level." Tolerant grading information can be a prompt, or part of the information that makes up a prompt.

[0077] The corrected answer acquisition prompt is a prompt to obtain a corrected written answer. A corrected written answer is a sentence that corrects the linguistic errors in the sentence shown in the acquired written answer. The corrected written answer is also referred to as the written answer as appropriate. An example of the corrected answer acquisition prompt is, "Output the sentence below with the linguistic errors corrected!"

[0078] The rule management unit 112 stores one or more scoring rules. These scoring rules are the rules for scoring Japanese language questions. Typically, these scoring rules are for scoring written response questions. The questions themselves may also be referred to as "questions" or "assignments."

[0079] The scoring rules of the rule management unit 112 may differ depending on the school or the respondent. In other words, each of the one or more scoring rules of the rule management unit 112 corresponds to, for example, school-specific information or respondent conditions.

[0080] It is preferable for two or more scoring rules to include both common rules and individual rules. Common rules are rules that apply to two or more written response questions. Examples of common rules include, "Points will be deducted for kanji mistakes," and "Points will be deducted for answers that do not fill 80% of the answer box." Individual rules are specific rules for each written response question. An example of an individual rule is, "Two points will be deducted if the protagonist's feelings are not expressed." Common rules are often deduction rules, as described later, but they may also be rules that correspond to bonus rules. Individual rules are often bonus rules, as described later, but they may also be rules that correspond to deduction rules.

[0081] It is preferable for two or more scoring rules to include both bonus rules and deduction rules. Bonus rules are information that specifies when points are added. Examples of bonus rules include: "Add 5 points if the protagonist's feelings are described," "Add points if A's feelings at that time are described," and "Add 1 point for beautiful and legible handwriting." Deduction rules are information that specifies when points are deducted. Examples of deduction rules include: "Deduction for kanji mistakes," "Deduction of 2 points for sentences that do not fill 80% of the answer box," and "Deduction of 2 points if the protagonist's feelings are missing."

[0082] The layout management unit 113 stores one or more layout information. The layout information in the layout management unit 113 is associated with, for example, a test identifier. A test identifier is information that identifies a test. A test identifier is, for example, the test ID or test name.

[0083] Layout information is information that identifies the layout of the answer sheet as specified by the answer sheet information. Layout information includes, for example, respondent identifier area information. Respondent identifier area information is area information where the respondent identifier is written. Area information is information that identifies an area. Layout information includes area information for written answers. Layout information includes area information for answers to one or more questions. Area information is, for example, the coordinate values ​​(x,y) of two points on the diagonal of a rectangular area. Area information is, for example, the coordinate values ​​(x1,y1) of the upper right area and (x2,y2) of the lower left area, with the upper right area of ​​the answer sheet as the origin.

[0084] The flag management unit 114 stores tolerance flags. Tolerance flags are information that identifies whether or not to perform lenient scoring. For example, tolerance flags can be "ON" to indicate that lenient scoring will be performed, or "OFF" to indicate that lenient scoring will not be performed.

[0085] The reception unit 12 receives various instructions and information. These instructions and information may include, for example, information on one or more answer sheets, scoring instructions, comparison instructions, instruction information, scoring rules, layout information, and tolerance flags.

[0086] A grading instruction is an instruction to grade one or more answer sheet entries. A grading instruction may, for example, contain one or more answer sheet entries. A grading instruction may also contain information for obtaining one or more answer sheet entries (for example, information indicating the location where the grading information is stored). The location where the grading information is stored may, for example, be a folder name or a URL.

[0087] A comparison instruction is an instruction that outputs a response in a manner that allows for comparison of the written answers of two or more respondents. A comparison instruction, for example, has a question identifier. A question identifier is information that identifies a question.

[0088] Here, "reception" typically refers to the reception of information transmitted via wired or wireless communication lines, but it may also be a concept that includes the reception of information input from input devices such as keyboards, mice, and touch panels, as well as the reception of information read from recording media such as optical discs, magnetic discs, and semiconductor memory.

[0089] The answer reception unit 121 receives one or more answer sheet information. For example, the answer reception unit 121 receives answer sheet information from the terminal device 2. The answer sheet information is associated with, for example, school-specific information.

[0090] The processing unit 13 performs various processes. These processes include, for example, those performed by the recognition unit 131, the correction unit 132, the scoring unit 133, or the storage unit 134.

[0091] The recognition unit 131 obtains character recognition results for one or more answers included in the answer sheet information received by the answer reception unit 121. The recognition unit 131 obtains character recognition results for written answers included in the answer sheet information received by the answer reception unit 121. The recognition unit 131 may also obtain character recognition results for one or more general answers included in the answer sheet information received by the answer reception unit 121. Handwritten written answers included in the answer sheet, and the character recognition results for those written answers, may both be referred to as written answers. Handwritten general answers included in the answer sheet, and the character recognition results for those general answers, may both be referred to as general answers. The recognition unit 131 obtains, for example, a respondent identifier from the answer sheet information. The recognition unit 131 obtains, for example, a respondent identifier from the character recognition results from the answer sheet information. Handwritten respondent identifiers and respondent identifiers from character recognition results may both be referred to as respondent identifiers.

[0092] The recognition unit 131 preferably outputs the descriptive answer, which is the result of character recognition for the descriptive answer contained in the answer sheet information received by the answer reception unit 121, and the character recognition score. The character recognition score is the score of character recognition, and is, for example, a score output by a generation AI or a character recognition module.

[0093] The recognition unit 131, for example, provides the handwritten written answer and recognition instruction information contained in the answer sheet information received by the answer reception unit 121 to the generation AI, and obtains the written answer, which is the character recognition result, from the generation AI.

[0094] The recognition unit 131, for example, provides the handwritten general answer and recognition instruction information contained in the answer sheet information received by the answer reception unit 121 to the generation AI, and obtains the general answer, which is the character recognition result, from the generation AI.

[0095] The recognition unit 131, for example, provides the handwritten respondent identifier and recognition instruction information contained in the answer sheet information received by the answer reception unit 121 to the generation AI, and obtains the respondent identifier, which is the character recognition result, from the generation AI.

[0096] The recognition unit 131, for example, provides the answer sheet information and recognition instruction information received by the answer reception unit 121 to the generation AI, and obtains two or more character recognition results and a respondent identifier from the generation AI. The two or more character recognition results include, for example, a descriptive answer and a general answer.

[0097] The distortion correction means 1311 corrects the distortion of the answer sheet information using one or more corner markers in the answer sheet information. The distortion correction means 1311 usually corrects the distortion of the answer sheet information using four corner markers. For example, the distortion correction means 1311 acquires the coordinate information of each of the four corner markers in the answer sheet information and corrects the answer sheet information so that each coordinate information becomes a predetermined value (x,y) relative to the coordinate information.

[0098] The extraction means 1312 uses layout information to obtain written answers from the answer sheet information. It is preferable for the extraction means 1312 to extract and obtain written answers from the distortion-corrected answer sheet information using layout information. For example, the extraction means 1312 uses layout information to obtain one or more written answers and one or more general answers from the distortion-corrected answer sheet information. For example, the extraction means 1312 uses layout information to obtain a respondent identifier from the distortion-corrected answer sheet information. The written answers obtained by the extraction means 1312 are usually handwritten strings.

[0099] The recognition means 1313 obtains the character-recognized descriptive answer for the descriptive answer acquired by the extraction means 1312.

[0100] The recognition means 1313 provides, for example, the descriptive answer and recognition instruction information acquired by the extraction means 1312 to the generating AI, and obtains the descriptive answer of the character recognition result from the generating AI. It is preferable that the recognition means 1313 provides, for example, the descriptive answer and recognition instruction information acquired by the extraction means 1312 to the generating AI, and outputs the descriptive answer of the character recognition result and the character recognition score from the generating AI.

[0101] The recognition means 1313, for example, provides the general answer and recognition instruction information acquired by the extraction means 1312 to the generating AI, and obtains the general answer of the character recognition result from the generating AI. The recognition means 1313, for example, provides the general answer and recognition instruction information acquired by the extraction means 1312 to the generating AI, and obtains the general answer of the character recognition result and the character recognition score from the generating AI.

[0102] The recognition means 1313, for example, provides the respondent identifier and recognition instruction information acquired by the extraction means 1312 to the generating AI, and obtains the respondent identifier of the character recognition result from the generating AI. The recognition means 1313, for example, provides the respondent identifier and recognition instruction information acquired by the extraction means 1312 to the generating AI, and obtains the respondent identifier of the character recognition result and the character recognition score from the generating AI.

[0103] The modification unit 132 assists in modifying the descriptive answers obtained by the recognition unit 131. The modification unit 132 may also assist in modifying the general answers obtained by the recognition unit 131 or modifying the responder identifier.

[0104] The result output means 1321 outputs the descriptive answer acquired by the recognition unit 131. The result output means 1321 transmits the descriptive answer acquired by the recognition unit 131 to the terminal device 2. Note that output here is a concept that includes, for example, display on a screen, projection using a projector, printing with a printer, sound output, transmission to an external device, storage on a recording medium, and transfer of processing results to other processing devices or other programs.

[0105] The result output means 1321 is preferably configured to output the descriptive answer acquired by the recognition unit 131 when the character recognition result from the recognition unit 131 satisfies the inappropriate conditions.

[0106] An inappropriate condition is when the character recognition result of a written answer is inappropriate. For example, an inappropriate condition is when the character recognition score is below or below a threshold.

[0107] The correction receiving means 1322 accepts corrected descriptive answers to those output by the result output means 1321. For example, the correction receiving means 1322 receives corrected descriptive answers from the terminal device 2 to those output by the result output means 1321. The correction receiving means 1322 may also accept uncorrected descriptive answers.

[0108] The scoring unit 133 provides the descriptive answers and scoring rules contained in the answer sheet information received by the answer reception unit 121 to the generating AI, and obtains the scoring results from the generating AI. The scoring unit 133 may also provide the model answers from the storage unit 11 to the generating AI.

[0109] A scoring result is information that shows the result of scoring. A scoring result has a score. The score may be, for example, the score for a single question. A scoring result may have, for example, a score for each of 1 or more answers. A scoring result may have, for example, a score and reason for each of 1 or more answers. A scoring result may have, for example, the score for the entire test.

[0110] For example, for each of the two or more written questions, the scoring unit 133 provides the generated AI with the written answers included in the answer sheet information received by the answer reception unit 121, the common rules, and the individual rules corresponding to the written question, and obtains the scoring result from the generated AI.

[0111] The scoring unit 133 provides, for example, the written answer, the scoring rules, and the scoring rules to the generating AI, and obtains the scoring result from the generating AI. Normally, for each question, the points allocated (maximum score) are stored in the storage unit 11, associated with the question identifier, and the scoring unit 133 obtains a score that does not exceed the maximum score even if, for example, multiple scoring rules are applied.

[0112] It is preferable for the scoring unit 133 to provide the written answer and scoring rules to the generating AI, and to obtain the scoring result and the reason for the scoring from the generating AI.

[0113] The scoring unit 133 also provides, for example, lenient scoring information to the generating AI and obtains the scoring result. The scoring unit 133 also provides, for example, lenient scoring information to the generating AI and obtains the scoring result when the lenient flag indicates that lenient scoring will be performed.

[0114] The scoring unit 133 obtains, for example, scoring rules associated with school-specific information associated with answer sheet information from the rule management unit 112, provides the scoring rules and written answers to the generating AI, and obtains the scoring results from the generating AI. The scoring unit 133 obtains, for example, scoring rules associated with school-specific information associated with answer sheet information and scoring rules common to all respondents from the rule management unit 112, provides the scoring rules and written answers to the generating AI, and obtains the scoring results from the generating AI.

[0115] The scoring unit 133, for example, provides the descriptive answer obtained by the correction unit 132 and the scoring rules to the generating AI, and obtains the scoring result from the generating AI.

[0116] The scoring unit 133, for example, provides the written answer before character recognition and the written answer as a result of character recognition to the generating AI, and obtains the scoring result from the generating AI.

[0117] The scoring unit 133 obtains the scoring result for each of the two or more respondents. The scoring unit 133 obtains the scoring result and the reason for each of the two or more respondents.

[0118] The proofreading means 1331 obtains the written answer contained in the answer sheet information received by the answer receiving unit 121 and obtains a corrected written answer corresponding to that written answer. For example, the proofreading means 1331 obtains the written answer contained in the answer sheet information received by the answer receiving unit 121, provides the written answer and a corrected answer acquisition prompt to the generating AI, and obtains the corrected written answer from the generating AI. If there are no linguistic errors in the written answer, the proofreading means 1331 obtains the original written answer or does not obtain a corrected written answer.

[0119] The scoring means 1332 may, for example, provide a corrected written answer and scoring rules to a generating AI and obtain a scoring result from the generating AI. Alternatively, the scoring means 1332 may provide an uncorrected written answer and scoring rules to a generating AI and obtain a scoring result from the generating AI. It is also preferable for the scoring means 1332 to obtain the reason for the scoring.

[0120] The storage unit 134 stores the respondent identifier and the answer sheet information in association. For each of the one or more answer sheet entries, the storage unit 134 stores the respondent identifier and the answer sheet information in association. The storage unit 134 stores the respondent identifier and the answer sheet information in the storage unit 11, for example, but the storage location is not specified. It is preferable that the respondent identifier here is a character-recognized respondent identifier.

[0121] The storage unit 134 stores, for example, respondent identifiers, answer sheet information, and scoring results in association. The storage unit 134 stores, for example, one or more answer sheet information entries, storing respondent identifiers, answer sheet information, and scoring results in association. The storage unit 134 stores, for example, scoring results for each answer sheet information entry and for each question (for each answer).

[0122] The output unit 14 outputs various types of information. Here, output is a concept that includes, for example, display on a screen, projection using a projector, printing with a printer, sound output, transmission to an external device, storage on a recording medium, and transfer of processing results to other processing devices or other programs.

[0123] The scoring output unit 141 outputs the scoring results. The scoring output unit 141 outputs the scoring results and reasons in association with the written questions. It is preferable for the scoring output unit 141 to output the scoring results and reasons in association with the written questions within the answer sheet information. It is preferable for the scoring output unit 141 to output answer sheet information in which the score and reasons for each of the one or more questions are written. It is preferable for the scoring output unit 141 to output sorted written answers and scoring results using the scoring results of two or more respondents as a key.

[0124] The terminal storage unit 21, which constitutes the terminal device 2, stores various types of information. These types of information include, for example, the respondent identifier.

[0125] The terminal reception unit 22 receives various types of information and instructions. These types of information and instructions include, for example, information on one or more answer sheets, scoring instructions, comparison instructions, instruction information, scoring rules, layout information, and tolerance flags.

[0126] Any means of inputting information and instructions can be used, such as a scanner, touch panel, keyboard, mouse, menu screen, camera, microphone, etc.

[0127] The terminal processing unit 23 performs various processes. These processes include, for example, converting received information and instructions into information and instructions for transmission. Other processes include, for example, converting received information into information for output.

[0128] The terminal transmission unit 24 transmits various information and instructions to the scoring device 1. These various information and instructions include, for example, information on one or more answer sheets, scoring instructions, comparison instructions, instruction information, scoring rules, layout information, and tolerance flags.

[0129] The terminal receiving unit 25 receives various types of information. These types of information include, for example, scoring results, answer sheet information, and written answers and scoring results sorted using the scoring results of two or more respondents as the key. Preferably, the answer sheet information here is answer sheet information that includes scoring results and reasons associated with the written answers.

[0130] The terminal output unit 26 outputs various types of information. These types of information include, for example, information received by the terminal receiving unit 25.

[0131] Here, "output" is a concept that includes display on a screen, projection using a projector, printing with a printer, sound output, transmission to an external device, storage on a recording medium, and transfer of processing results to other processing devices or other programs.

[0132] The storage unit 11, instruction management unit 111, rule management unit 112, layout management unit 113, flag management unit 114, or terminal storage unit 21 are preferably made of a non-volatile recording medium, but can also be made of a volatile recording medium.

[0133] The process by which information is stored in the storage unit 11, etc. is not relevant. For example, information may be stored in the storage unit 11, etc. via a recording medium, information transmitted via a communication line, etc. may be stored in the storage unit 11, etc., or information input via an input device may be stored in the storage unit 11, etc.

[0134] The reception unit 12 or the answer reception unit 121 is preferably implemented by wireless or wired communication means, but may also be implemented by means of receiving broadcasts, device drivers for input means such as touch panels or keyboards, or control software for menu screens.

[0135] The processing unit 13, recognition unit 131, correction unit 132, scoring unit 133, storage unit 134, distortion correction means 1311, cutting means 1312, recognition means 1313, result output means 1321, correction acceptance means 1322, calibration means 1331, or scoring means 1332 can usually be implemented using a processor, memory, etc. The processing procedures of the processing unit 13, etc., are usually implemented in software, and this software is recorded on a recording medium such as ROM. However, it may also be implemented in hardware (dedicated circuitry). The processor can be a CPU, MPU, GPU, etc., and the type is not limited.

[0136] The output unit 14, or the scoring output unit 141, is usually implemented by wireless or wired communication means, but it may also be implemented by driver software for an output device such as a display or speaker, or by driver software for an output device and the output device itself.

[0137] The terminal reception unit 22 can be implemented using device drivers for input means such as touch panels and keyboards, or control software for menu screens, etc.

[0138] The terminal transmission unit 24 is usually implemented by wireless or wired communication means, but it may also be implemented by broadcasting means.

[0139] The terminal receiving unit 25 is usually implemented by wireless or wired communication means, but it may also be implemented by means of receiving broadcasts.

[0140] The terminal output unit 26 may or may not be considered to include output devices such as a display or speakers. The terminal output unit 26 can be implemented using driver software for an output device, or driver software for an output device and an output device.

[0141] Next, an example of the operation of the scoring device 1 will be explained using the flowchart in Figure 4.

[0142] (Step S401) The reception unit 12 determines whether or not it has received the scoring instruction. If it has received the scoring instruction, it proceeds to step S402; otherwise, it proceeds to step S408.

[0143] (Step S402) The recognition unit 131 assigns 1 to counter i.

[0144] (Step S403) The recognition unit 131 determines whether or not the i-th respondent exists. If the i-th respondent exists, the process proceeds to step S404; otherwise, it returns to step S401. Whether or not the i-th respondent exists is determined, for example, by whether or not the i-th answer sheet information exists, or whether or not the i-th respondent identifier exists in the storage unit 11.

[0145] (Step S404) The recognition unit 131 obtains the answer sheet information of the i-th respondent.

[0146] (Step S405) The recognition unit 131, etc., performs a scoring process on the answer sheet information acquired in step S404 and obtains the scoring result. An example of the scoring process will be explained using the flowchart in Figure 5.

[0147] (Step S406) The scoring output unit 141 outputs the scoring result obtained in step S405. The scoring output unit 141 or the storage unit 134 stores the scoring result, for example, in association with the i-th answer sheet information.

[0148] (Step S407) The recognition unit 131 increments the counter i by 1. Return to step S403.

[0149] (Step S408) The reception unit 12 determines whether or not it has received the answer sheet information. If it has received the answer sheet information, it proceeds to step S409; otherwise, it proceeds to step S412.

[0150] (Step S409) The recognition unit 131 acquires the answer sheet information received in step S408.

[0151] (Step S410) The recognition unit 131, etc., performs a scoring process on the answer sheet information acquired in step S409 and obtains the scoring result. An example of the scoring process will be explained using the flowchart in Figure 5.

[0152] (Step S411) The scoring output unit 141 outputs the scoring result obtained in step S410. Return to step S401.

[0153] (Step S412) The reception unit 12 determines whether or not it has received the comparison instruction. If it has received the comparison instruction, it proceeds to step S409; otherwise, it proceeds to step S416.

[0154] (Step S413) The processing unit 13 obtains the question identifier included in the comparison instruction.

[0155] (Step S414) The processing unit 13 retrieves two or more descriptive answers corresponding to the question identifier from the storage unit 11. The processing unit 13 sorts the two or more descriptive answers corresponding to the question identifier using the score corresponding to the descriptive answer as the key. The processing unit 13 usually sorts the descriptive answers in descending order, but it is also acceptable to sort them in ascending order.

[0156] (Step S415) The output unit 14 outputs the descriptive answers etc. sorted in step S414. The process returns to step S401. The descriptive answers etc. include, for example, a descriptive answer, a descriptive answer and score, or a descriptive answer, score and reason.

[0157] (Step S416) The reception unit 12 determines whether or not it has received the information. If it has received the information, it proceeds to step S417; otherwise, it returns to step S401. The information may include, for example, instruction information, scoring rules, layout information, and tolerance flags.

[0158] (Step S417) The storage unit 134 stores the information received in step S416 in the storage unit 11. Return to step S401. The storage unit 134 stores, for example, one or more scoring rules in the rule management unit 112, associated with school-specific information.

[0159] In the flowchart shown in Figure 4, processing is terminated by power-off or processing termination interrupts.

[0160] Next, an example of the scoring process in steps S405 and S410 will be explained using the flowchart in Figure 5.

[0161] (Step S501) The distortion correction means 1311 performs distortion correction processing on the answer sheet information to be graded. An example of distortion correction processing will be explained using the flowchart in Figure 6.

[0162] (Step S502) The cutting means 1312 obtains layout information from the layout management unit 113.

[0163] (Step S503) The cutting means 1312 cuts out the area indicated by the respondent identifier area information contained in the layout information acquired in step S502 from the answer sheet information after distortion correction. The recognition means 1313 performs character recognition processing on the cut-out area and obtains the respondent identifier.

[0164] (Step S504) The storage unit 134 stores the answer sheet information in the storage unit 11, associating it with the character-recognized respondent identifier.

[0165] (Step S505) The scoring unit 133 obtains school identification information corresponding to the answer sheet information. For example, the scoring unit 133 obtains school identification information from the storage unit 11 that is paired with the character-recognized respondent identifier.

[0166] (Step S506) The scoring unit 133 obtains one or more common rules from the rule management unit 112.

[0167] (Step S507) The scoring unit 133 assigns 1 to counter i.

[0168] (Step S508) The scoring unit 133 determines whether or not the i-th question exists. If the i-th question exists, it proceeds to step S509; otherwise, it returns to the higher-level process.

[0169] (Step S509) The cutting means 1312 obtains the region information of the i-th question from the layout information.

[0170] (Step S510) The extraction means 1312 extracts the region indicated by the region information of the i-th question from the answer sheet information and obtains the handwritten answer for the i-th region. The handwritten answer here is usually an image.

[0171] (Step S511) The recognition means 1313 determines whether the answer to the i-th question is a descriptive answer or a general answer. If it is a descriptive answer, proceed to step S512; if it is a general answer, proceed to step S517. The recognition means 1313 obtains, for example, the question type that corresponds to the question identifier of the i-th question from the storage unit 11. The question type is, for example, "Descriptive answer (e.g., "1")" or "General answer (e.g., "0")". The recognition means 1313 obtains, for example, the length (size) of the answer to the i-th question, and determines that it is a descriptive answer if the length (size) is equal to or greater than a threshold or longer than a threshold, and determines that it is a general answer if it is less than or equal to a threshold. Note that there may be various methods by which the recognition means 1313 determines whether the answer to a question is a descriptive answer or a general answer, and this method is not limited.

[0172] (Step S512) The recognition means 1313 obtains the recognition result for the descriptive answer to the i-th question. An example of such descriptive recognition processing will be explained using the flowchart in Figure 7.

[0173] (Step S513) The scoring unit 133 performs scoring on the written answer to the i-th question. An example of the scoring process will be explained using the flowchart in Figure 9.

[0174] (Step S514) The storage unit 134 stores the handwritten answers and scores etc. in the storage unit 11, associating them with the respondent identifier. The handwritten answers and scores etc. may be, for example, the handwritten answers and scores, or the handwritten answers, scores and reasons.

[0175] (Step S515) The storage unit 134 adds the score, etc., to the position corresponding to the i-th area of ​​the answer sheet information. The score, etc., is, for example, the score, or the score and reason.

[0176] (Step S516) The scoring unit 133 increments counter i by 1. Return to step S508.

[0177] (Step S517) The recognition means 1313 obtains the recognition result for the i-th handwritten answer. The recognition means 1313, for example, provides the i-th handwritten answer to a character recognition module and obtains the recognition result from the character recognition module. The recognition means 1313 obtains the recognition result by, for example, the process described using the flowchart in Figure 7. It is preferable that the recognition means 1313 perform different recognition processing for written answers and general answers.

[0178] (Step S518) The scoring unit 133 retrieves the correct answer to the i-th question from the storage unit 11.

[0179] (Step S519) The scoring unit 133 determines whether the recognition result obtained in step S517 matches the correct answer obtained in step S518. If they match, the unit proceeds to step S520; otherwise, the unit proceeds to step S521.

[0180] (Step S520) The scoring unit 133 retrieves the score for the i-th question from the storage unit 11. Proceed to step S514.

[0181] (Step S521) The scoring unit 133 obtains a score of "0". Proceed to step S514.

[0182] Note that in the flowchart of Figure 5, the distortion correction process in step S501 is optional.

[0183] Next, an example of the distortion correction process in step S501 will be explained using the flowchart in Figure 6.

[0184] (Step S601) The distortion correction means 1311 recognizes corner markers around the four corners (for example, a cross shape) from the answer sheet information and obtains the coordinate values ​​((x1,y1)(x2,y2)(x3,y3)(x4,y4)) of each of the four corner markers in the answer sheet information. The coordinate values ​​are information that indicates the position of the point where the lines of the cross of the cross-shaped corner marker intersect.

[0185] (Step S602) The distortion correction means 1311 obtains the correct coordinate information ((X1,Y1)(X2,Y2)(X3,Y3)(X4,Y4)) of the four corner markers from the storage unit 11.

[0186] (Step S603) The distortion correction means 1311 corrects the distortion of the answer sheet information, which is an image, so that the coordinate values ​​((x1,y1)(x2,y2)(x3,y3)(x4,y4)) of each of the four corner markers in the answer sheet information become the positions indicated by the correct coordinate information ((X1,Y1)(X2,Y2)(X3,Y3)(X4,Y4)). It then returns to the higher-level processing. Note that this processing can be implemented by known technology.

[0187] Next, an example of the description recognition process in step S512 will be explained using the flowchart in Figure 7.

[0188] (Step S701) The recognition means 1313 acquires recognition instruction information from the instruction management unit 111.

[0189] (Step S702) The recognition means 1313 obtains the handwritten answer.

[0190] (Step S703) The recognition means 1313 provides the recognition instruction information and the handwritten answer to the generating AI.

[0191] (Step S704) The recognition means 1313 determines whether or not an answer has been obtained from the generating AI. If an answer has been obtained, the process proceeds to step S705; otherwise, it returns to step S704.

[0192] (Step S705) The recognition means 1313 obtains a score, etc. from the answer. The score, etc. is the character recognition score and the recognition result.

[0193] (Step S706) The recognition means 1313 determines whether the character recognition score satisfies the inappropriate conditions. If the inappropriate conditions are met, the process proceeds to step S707; otherwise, it returns to the higher-level processing.

[0194] (Step S707) The recognition means 1313 performs a correction process on the recognition result. An example of such a correction process will be explained using the flowchart in Figure 8.

[0195] Note that steps S706 to S707 are optional in the flowchart shown in Figure 8.

[0196] Next, an example of the correction process in step S707 will be explained using the flowchart in Figure 8.

[0197] (Step S801) The result output means 1321 outputs the recognition result and the handwritten answer. The result output means 1321 transmits, for example, the acquired recognition result and the handwritten answer to the terminal device 2.

[0198] (Step S802) The correction reception means 1322 determines whether or not it has received the corrected recognition result from the user of terminal device 2 (for example, the instructor). If the recognition result has been received, the process proceeds to step S803; otherwise, it returns to step S802.

[0199] (Step S803) The correction receiving means 1322 obtains the recognition result received in step S802. It returns to the higher-level processing. Note that there does not need to be any correction to the recognition result at this point. The correction receiving means 1322 receives, for example, the final recognition result from the terminal device 2.

[0200] Next, an example of the scoring process in step S513 will be explained using the flowchart in Figure 9.

[0201] (Step S901) The calibration means 1331 acquires the recognition result.

[0202] (Step S902) The calibration means 1331 obtains a corrected answer acquisition prompt from the instruction management unit 111.

[0203] (Step S903) The calibration means 1331 provides the recognition result and the corrected answer acquisition prompt to the generating AI.

[0204] (Step S904) The calibration means 1331 determines whether or not it has obtained an answer from the generating AI. If an answer has been obtained, it proceeds to step S905; otherwise, it returns to step S904.

[0205] (Step S905) The proofreading means 1331 obtains a corrected written answer from the answer. Note that the corrected written answer may be the same as the original written answer. If there are no linguistic errors in the original written answer, the corrected written answer is the same as the original written answer.

[0206] (Step S906) The scoring means 1332 obtains the individual rules for the question from the rule management unit 112. The scoring means 1332 may also obtain the individual rules corresponding to school-specific information from the rule management unit 112.

[0207] (Step S907) The scoring means 1332 determines whether the tolerance flag is ON or OFF. If the tolerance flag is ON, proceed to step S908; otherwise, proceed to step S909.

[0208] (Step S908) The scoring means 1332 obtains lenient scoring information from the instruction management unit 111.

[0209] (Step S909) The scoring means 1332 provides the recognition result and scoring rules, etc. to the generating AI. The recognition result and scoring rules, etc. include, for example, the recognition result, one or more individual rules, and one or more common rules. The recognition result and scoring rules, etc. include, for example, the recognition result, one or more individual rules, one or more common rules, and a scoring prompt. The recognition result and scoring rules, etc. include, for example, the recognition result, one or more individual rules, one or more common rules, and tolerance scoring information.

[0210] (Step S910) The scoring means 1332 determines whether or not it has obtained an answer from the generating AI. If an answer has been obtained, it proceeds to step S911; otherwise, it returns to step S910.

[0211] (Step S911) The scoring means 1332 obtains the score and reason included in the answer obtained in step S910. It returns to the higher-level processing.

[0212] Note that in the flowchart of Figure 9, the calibration process, which is the process of calibration means 1331, does not need to be performed.

[0213] The following describes a specific example of the operation of information system A in this embodiment.

[0214] Currently, the storage unit 11 of the scoring device 1 contains the questions for a Japanese language test. The question for Question 1 is: "The underlined part (2) says 'It pierces the heart when you remember it.' Why is that? Explain in 80 characters or less."

[0215] Furthermore, the storage unit 11 stores answer sheet information from a large number of students aiming for junior high school entrance exams. This answer sheet information is a file accumulated by instructors scanning the answer sheets of numerous students. An example of the answer sheet information is shown in Figure 10. It should be noted that the answer sheet information in the storage unit 11 is associated with school-specific information, "XXX Junior High School." In other words, the answer sheet information in the storage unit 11 is the answer sheet information for the pass / fail judgment test of "XXX Junior High School."

[0216] Furthermore, the instruction management unit 111 stores the OCR prompt "Your role is to copy the text exactly as you see it on the answer sheet for the Japanese language test. Output the answers for each question, paired with the question number. Also, output the answers according to the following instructions: [Instructions] Read from the rightmost column, moving to the leftmost column. Read from top to bottom in each column. If there are two or more columns in the answer, do not jump between columns. Contextual correction is strictly prohibited..." Furthermore, the instruction management unit 111 stores the scoring prompt shown in Figure 11. Note that 1101 in Figure 11 is the answer deficiency criteria (which can also be called the answer deficiency rule). The answer deficiency criteria is a scoring rule that identifies an answer that is deficient. The answer deficiency criteria is a type of scoring rule. Here, the answer deficiency criteria can be considered a type of common rule. The answer deficiency criteria here is information that identifies an answer deficiency that will result in a "0" score. 1102 is the scoring criterion (or scoring rule) for question identifier "Question 1". The scoring rule may also be included in the scoring prompt.

[0217] Furthermore, the storage unit 11 stores the answers or model answers to the test, associated with the question identifier. For example, the model answer "I was deceived by my mother regarding my father's will, and since she showed no remorse and I had no intention of following the will, I was told I had no right to pretend to be a victim, and I couldn't argue back." is stored as the question identifier "Question 1".

[0218] Furthermore, the rule management unit 112 stores the scoring rules shown in Figure 12. The scoring rules shown in Figure 12 correspond to the question identifier "Question 1". In other words, the scoring rules shown in Figure 12 are the scoring rules for Question 1. The scoring rules shown in Figure 12 are associated with the school-specific information "XXX Junior High School". In other words, the scoring rules shown in Figure 12 are the scoring rules for students aiming for "XXX Junior High School".

[0219] Furthermore, the layout management unit 113 stores layout information, including area information for each question on the answer sheet.

[0220] Furthermore, the flag management unit 114 stores the leniency flag "ON". The storage unit 11 stores the leniency scoring information: "Correct grammatical, expressive, and formal errors, and calculate a fair and minimal breakdown of deductions that takes elementary school level into consideration."

[0221] In light of the above situation, the following two specific examples will be explained. Specific example 1 is an example that explains the process of grading answer sheet information from two or more students. Specific example 2 is an example that explains the process of extracting written answers to questions from two or more students, sorting them by grade, and outputting them as a list.

[0222] (Specific example 1) The instructor enters a grading instruction into terminal device 2. Next, terminal device 2 receives the grading instruction and transmits it to grading device 1.

[0223] Next, the scoring device 1 receives a scoring instruction from the terminal device 2. Then, the scoring device 1 performs the operations described using the flowchart in Figure 4 and scores the answer sheet information of each student.

[0224] First, the recognition unit 131 obtains the answer sheet information 1001 of the first student from the storage unit 11. Next, the distortion correction means 1311 performs distortion correction processing on the answer sheet information. Next, the extraction means 1312 obtains layout information from the layout management unit 113. The extraction means 1312 extracts the area indicated by the respondent identifier area information contained in the layout information from the distortion-corrected answer sheet information. Next, the recognition means 1313 performs character recognition processing on the extracted area and obtains the respondent identifier (in this case, the membership number) "01675362". The recognition means 1313 also obtains the examination number "84". Next, the storage unit 134 stores the respondent identifier "01675362" and the examination number "84" in the storage unit 11, associating them with the answer sheet information.

[0225] Next, the scoring unit 133 retrieves the school-specific information "XXX Junior High School" associated with the answer sheet information from the storage unit 11. The scoring unit 133 also retrieves one or more common rules from the rule management unit 112.

[0226] Next, the extraction means 1312 extracts the region indicated by the region information of the first question from the answer sheet information and obtains the handwritten answer for the first region. This handwritten answer is shown in Figure 13.

[0227] Next, the recognition means 1313 determines that the answer to the first question is a written answer. Then, using the written recognition process explained with the flowchart in Figure 7, the recognition means 1313 obtains the recognition result for the written answer to the first question: "My mother wants to use me for her own convenience... so I got angry."

[0228] Next, the scoring unit 133 performs the scoring process described using the flowchart in Figure 9. Specifically, the scoring unit 133 obtains the scoring prompt in Figure 11. The scoring unit 133 also obtains the scoring rules in Figure 12. Furthermore, the scoring unit 133 detects that the tolerance flag in the storage unit 11 is "ON" and obtains the tolerance scoring information "Correct grammatical, expression, and formal errors, and calculate a fair and minimal breakdown of deductions considering the elementary school level." from the storage unit 11.

[0229] Next, the scoring unit 133 provides the AI ​​with the recognition result "My mother wants to use me for her own convenience... so I got angry," a scoring prompt, scoring rules, and lenient scoring information.

[0230] Next, the scoring unit 133 obtains the answer from the generating AI. Then, the scoring unit 133 obtains the scoring result "<Score> 5 <Reason> Lacking the phrase "Regarding my father's will," ..." from the answer. Note that this scoring result is the scoring result for the written answer to the first question.

[0231] Next, the storage unit 134 associates the respondent identifier "01675362" with the question identifier "Question 1" and stores the handwritten answer and the scoring result for "Question 1" "<Score> 5 <Reason> Lacking the phrase "Regarding my father's will,"..." in the storage unit 11.

[0232] From this point forward, the scoring device 1 will score the student's answer sheet information, including the answers to question 2 and beyond, according to the process explained using the flowchart in Figure 5, obtain the scoring results for the entire answer sheet information, and output them.

[0233] The scoring results shall include the information from the scored answer sheet. The scored answer sheet information shall include the score in a manner corresponding to the answer to each question on the answer sheet. Preferably, the scored answer sheet information shall include the score and the reason in a manner corresponding to the written answer to the written question on the answer sheet.

[0234] Furthermore, the scoring results will include handwritten or character-recognized answers corresponding to each question, along with the score or the score and its explanation.

[0235] Furthermore, the scoring device 1 also performs the scoring process for the answer sheet information of the second and subsequent students, as explained using Figures 4 and 5. Then, the scoring device 1 stores the scoring results in the storage unit 11, associating each student's membership number and examination number with the answer sheet information of the second and subsequent students.

[0236] (Specific example 2) Assume the instructor is giving a lecture explaining the results of a Japanese language test given to two or more students. Then, the instructor enters "Comparison instruction <question identifier>Question 1" into terminal device 2. Next, terminal device 2 receives the comparison instruction and transmits it to scoring device 1.

[0237] Next, the scoring device 1 receives the comparison instruction. Next, the processing unit 13 obtains the question identifier "Question 1" included in the comparison instruction. Next, the processing unit 13 obtains each student's written answer corresponding to the question identifier "Question 1" from the storage unit 11. Next, the processing unit 13 sorts the two or more written answers corresponding to the question identifier in descending order using the score corresponding to the written answer as the key. Next, the output unit 14 transmits the sorted written answers, etc., to the terminal device 2.

[0238] The instructor's terminal device 2 receives and outputs written answers sorted in descending order. An example of such output is shown in Figure 14. In Figure 14, the answers of multiple students are also sorted in descending order by score. Furthermore, in Figure 14, the list includes not only the score but also the reason, which facilitates student understanding and makes it easier for the instructor to conduct lectures explaining the test answers. Note that in Figure 14, the list may also include the results of character recognition of handwritten answers.

[0239] As described above, according to this embodiment, by appropriately grading the answers to descriptive questions in Japanese language automatically or semi-automatically, the burden on graders is reduced and fair grading becomes possible.

[0240] Furthermore, according to this embodiment, by applying appropriate rules to the answers to descriptive questions in Japanese language, scoring can be performed automatically or semi-automatically, reducing the burden on the grader and enabling fair and highly accurate scoring.

[0241] Furthermore, according to this embodiment, by appropriately grading the answers to descriptive questions in Japanese language automatically or semi-automatically, the burden on graders is reduced and fair grading becomes possible.

[0242] Furthermore, according to this embodiment, answers to descriptive questions in Japanese language can be appropriately graded automatically or semi-automatically, and the grading results and reasons for grading can be output.

[0243] Furthermore, according to this embodiment, appropriate scoring can be performed on answers to descriptive questions in Japanese language, depending on the situation.

[0244] Furthermore, according to this embodiment, it becomes possible to appropriately grade the answers to descriptive questions in Japanese language according to the student's desired school and the school administering the test.

[0245] Furthermore, according to this embodiment, by appropriately grading handwritten answers to descriptive questions in Japanese language automatically or semi-automatically, the burden on graders is reduced, and fair grading becomes possible.

[0246] The processing in this embodiment may be implemented by software. This software may be distributed by software download or the like. Alternatively, this software may be recorded on a recording medium such as a CD-ROM and distributed. This also applies to other embodiments in this specification. The software that implements the scoring device 1 in this embodiment is the following program. In other words, this program causes a computer that can access a rule management unit where scoring rules for descriptive questions in Japanese language are stored to function as an answer receiving unit that receives answer sheet information including descriptive answers that are answers to descriptive questions in Japanese language and are handwritten answers; a scoring unit that provides the descriptive answers included in the answer sheet information received by the answer receiving unit and the scoring rules to a generating AI and obtains scoring results from the generating AI; and a scoring output unit that outputs the scoring results.

[0247] Figure 15 is a block diagram of a computer system 300 that executes the program described herein to realize the scoring device 1 and other devices of the various embodiments described above.

[0248] In Figure 15, the computer system 300 includes a computer 301 with a CD-ROM drive, a keyboard 302, a mouse 303, and a monitor 304.

[0249] In Figure 15, the computer 301 includes, in addition to the CD-ROM drive 3012, an MPU 3013, a bus 3014 connected to the CD-ROM drive 3012, a ROM 3015 for storing programs such as boot-up programs, a RAM 3016 connected to the MPU 3013 for temporarily storing application program instructions and providing temporary storage space, and a hard disk 3017 for storing application programs, system programs, and data. Although not shown here, the computer 301 may further include a network card for providing connectivity to a LAN.

[0250] The program that causes the computer system 300 to execute the functions of the scoring device 1, etc., as described above, may be stored on CD-ROM 3101, inserted into CD-ROM drive 3012, and then transferred to hard disk 3017. Alternatively, the program may be transmitted to computer 301 via a network (not shown) and stored on hard disk 3017. The program is loaded into RAM 3016 during execution. The program may also be loaded directly from CD-ROM 3101 or the network.

[0251] The program does not necessarily have to include an operating system (OS) or third-party program that causes the computer 301 to execute functions such as the scoring device 1 of the above-described embodiment. The program only needs to include the instruction portion that calls appropriate functions (modules) in a controlled manner to obtain the desired result. How the computer system 300 operates is well known, so a detailed explanation is omitted.

[0252] In the above program, steps such as sending information and receiving information do not include hardware-based processing, such as processing performed by a modem or interface card in the transmission step (processing that can only be performed by hardware).

[0253] Furthermore, the computer running the above program may be a single computer or multiple computers. In other words, it may perform centralized processing or distributed processing.

[0254] Furthermore, it goes without saying that in each of the above embodiments, two or more communication means present in a single device may be physically implemented in a single medium.

[0255] Furthermore, in each of the above embodiments, each process may be implemented by centralized processing by a single device, or by distributed processing by multiple devices.

[0256] It goes without saying that the present invention is not limited to the embodiments described above, and various modifications are possible, all of which are also included within the scope of the present invention. [Industrial applicability]

[0257] As described above, the scoring device 1 according to the present invention has the effect of reducing the burden on graders and enabling fair scoring by appropriately scoring answers to descriptive questions in Japanese language automatically or semi-automatically, and is useful, for example, as a server for scoring Japanese language tests for a large number of students. [Explanation of Symbols]

[0258] A Information Systems 1. Scoring device 2 Terminal devices 3 Generation AI device 11 Storage Unit 12 Reception Department 13 Processing Unit 14 Output section 21 Terminal storage section 22 Terminal Reception Section 23 Terminal Processing Unit 24 Terminal transmission unit 25 Receiving part of the terminal 26 Terminal output section 111 Instruction Management Department 112 Rule Management Department 113 Layout Management Department 114 Flag Management Department 121 Answer Submission Department 131 Recognition part 132 Correction section 133 Grading Department 134 Storage section 141 Scoring Output Unit 1311 Correction means 1312 Cutting means 1313 Recognition means 1321 Result output means 1322 Correction Request Method 1331 Calibration means 1332 Scoring Method

Claims

1. A rule management unit that stores scoring rules for descriptive questions in Japanese language, The answer reception department accepts answer sheet information, including handwritten answers to descriptive questions in Japanese language, and A scoring unit provides the descriptive answers included in the answer sheet information received by the answer reception unit and the scoring rules to a generating AI, and obtains the scoring results from the generating AI. The system comprises a scoring output unit that outputs the aforementioned scoring results, The aforementioned rule management unit includes: School-specific information that identifies the desired school or two or more schools that administer the test is associated with different scoring rules, which are then stored. The aforementioned answer reception unit, The answer sheet information corresponding to the school-specific information is received. The scoring unit is, A scoring device that acquires school-specific information corresponding to the answer sheet information, acquires scoring rules corresponding to the school-specific information from the rule management unit, provides the scoring rules and the written answers to the generating AI, and acquires the scoring results from the generating AI.

2. A rule management unit that stores scoring rules for descriptive questions in Japanese language, The answer reception department accepts answer sheet information, including handwritten answers to descriptive questions in Japanese language, and A scoring unit provides the descriptive answers included in the answer sheet information received by the answer reception unit and the scoring rules to a generating AI, and obtains the scoring results from the generating AI. The system comprises a scoring output unit that outputs the aforementioned scoring results, An instruction management unit stores lenient scoring information that instructs the system to perform lenient scoring, It further comprises a flag management unit that stores a tolerance flag that determines whether or not to perform lenient scoring, The scoring unit is, A scoring device that determines whether the tolerance flag is information indicating that lenient scoring will be performed, and if the tolerance flag is information indicating that lenient scoring will be performed, provides the lenient scoring information to the generating AI and obtains the scoring result.

3. A rule management unit that stores scoring rules for descriptive questions in Japanese language, The answer reception department accepts answer sheet information, including handwritten answers to descriptive questions in Japanese language, and A scoring unit provides the descriptive answers included in the answer sheet information received by the answer reception unit and the scoring rules to a generating AI, and obtains the scoring results from the generating AI. The system comprises a scoring output unit that outputs the aforementioned scoring results, The aforementioned answer reception unit, The answer sheet information of two or more students is received, The system further comprises an extraction means for extracting the handwritten written answers from the answer sheet information of each of the two or more students, The scoring unit is, For each of the two or more students mentioned above, obtain the scoring result, The system further comprises a processing unit that sorts the written answers of each of the two or more students using the scoring result of the written answers as the key. The scoring output unit is, A scoring device that outputs the sorted two or more descriptive answers and the scoring results.

4. A rule management unit that stores scoring rules for descriptive questions in Japanese language, The answer reception department accepts answer sheet information, including handwritten answers to descriptive questions in Japanese language, and A scoring unit provides the descriptive answers included in the answer sheet information received by the answer reception unit and the scoring rules to a generating AI, and obtains the scoring results from the generating AI. The system comprises a scoring output unit that outputs the aforementioned scoring results, The aforementioned rule management unit includes: It stores common rules, which are the common scoring rules for two or more written questions, and individual rules, which are the scoring rules for each of the two or more written questions. The aforementioned answer reception unit, We accept answer sheet information containing written answers to two or more written questions, or written answers contained in two or more answer sheet information containing written questions. The scoring unit is, A scoring device that provides a generating AI with the descriptive answers included in the answer sheet information received by the answer reception unit, the common rules, and the individual rules for the descriptive questions, and obtains the scoring result from the generating AI for each of the two or more descriptive questions.

5. The scoring unit is, The scoring device according to claim 4, wherein the device obtains the common rules from the rule management unit once, obtains the individual rules for each descriptive question from the rule management unit, provides the common rules and the individual rules for each descriptive question to the generating AI, and obtains the scoring result from the generating AI for each of the two or more descriptive questions.

6. The aforementioned rule management unit includes: Two or more scoring rules, including rules for adding points and rules for deducting points for the aforementioned written questions, are stored. The scoring unit is, A scoring device according to any one of claims 1 to 4, which provides a written answer, the scoring rules and the scoring rules to the generating AI, and obtains the scoring result from the generating AI.

7. The scoring unit is, A proofreading means that obtains a written answer contained in the answer sheet information received by the answer receiving unit, and obtains a corrected written answer which is a sentence in which the linguistic errors in the sentence indicated by the written answer have been corrected, A scoring device according to any one of claims 1 to 4, comprising: a scoring means for providing the aforementioned revised written answer and the aforementioned scoring rules to a generating AI and obtaining the aforementioned scoring result from the generating AI.

8. The scoring unit is, The descriptive answer and the scoring rules are provided to the generating AI, and the generating AI is made to output the scoring result and the reason for the scoring. The scoring output unit is, A scoring device according to any one of claims 1 to 4, which outputs the scoring result and the reason.

9. The scoring output unit is, The scoring device according to claim 8, which outputs the scoring result and the reason in the answer sheet information in a manner corresponding to the written question.

10. The scoring device according to any one of claims 1 to 4, further comprising a recognition unit that acquires the written answer, which is the result of character recognition for the written answer contained in the answer sheet information received by the answer receiving unit.

11. The aforementioned written answer is in vertical writing format with two or more lines. The system further comprises an instruction management unit which stores recognition instruction information indicating an instruction to obtain the sentence shown in the aforementioned descriptive answer, and an instruction to obtain the sentence from a vertically written handwritten text having two or more lines, The aforementioned recognition unit, The scoring device according to claim 10, wherein the handwritten written answers and the recognition instruction information included in the answer sheet information received by the answer receiving unit are provided to a generating AI, and the written answers, which are character recognition results, are obtained from the generating AI.

12. The system further comprises a layout management unit which stores layout information that identifies the layout of the answer sheet identified by the answer sheet information, and information that identifies the area of ​​the written answer, The aforementioned recognition unit, An extraction means for obtaining the written answers from the answer sheet information using the layout information, The scoring device according to claim 10, further comprising: recognition means for acquiring the characterized written answer for the written answer acquired by the extraction means.

13. The aforementioned answer sheet information is information obtained by photographing an answer sheet on which corner markers, which are markers used to identify the positions of the four corners, are printed. The aforementioned recognition unit, The system further comprises distortion correction means for correcting distortion of the answer sheet information using the aforementioned corner marks, The cutting means is, The scoring device according to claim 12, which obtains the written answer using the layout information from the distortion-corrected answer sheet information.

14. The recognition unit provides a result output means for outputting the descriptive answer acquired by the recognition unit, The correction unit further comprises a correction receiving means for receiving corrections to the descriptive answer output by the result output means, The scoring unit is, The scoring device according to claim 10, which provides the descriptive answer corrected by the correction unit and the scoring rules to a generating AI and obtains a scoring result from the generating AI.

15. The aforementioned answer sheet information includes a handwritten respondent identifier, The aforementioned recognition unit, The respondent identifier is obtained from the aforementioned answer sheet information, The scoring device according to claim 10, further comprising a storage unit that stores the respondent identifier and the answer sheet information in association.

16. The scoring unit is, The scoring device according to claim 10, wherein the descriptive answer before character recognition and the descriptive answer as a result of character recognition are provided to a generating AI, and the scoring result is obtained from the generating AI.

17. The aforementioned answer sheet information includes descriptive answers and general answers which are answers to correct answer identification questions, which are questions in which the correct answer is identified. The scoring unit is, A descriptive scoring means provides the descriptive answers included in the answer sheet information and the scoring rules to a generating AI, and obtains a descriptive scoring result from the generating AI, A general scoring means that determines whether the general answer included in the answer sheet information is correct and obtains a general scoring result which is the scoring result of the correct answer identification problem, A scoring device according to any one of claims 1 to 4, comprising scoring acquisition means for acquiring a scoring result for the answer sheet information using the descriptive scoring result and the general scoring result.

18. A scoring method that causes a computer to perform all the processing that a scoring device according to any one of claims 1 to 4 would perform.

19. Computers, A program for causing a scoring device to function as described in any one of claims 1 to 4.

Images