Program, information processing device, and information processing method.

Abstract

To provide a program capable of detecting incorrect grades which may have been incorrectly graded by graders from among grades by graders and the like on correctness of answers in a test and the like, an information processing device, and an information processing method.SOLUTION: A program causes a computer to achieve a classification function, a comparison function, a calculation function, and a determination function. The classification function classifies correctness of answers indicated by answer data a prescribed number of times, which is a plurality of times, using a classification model learned from the answer data. The comparison function compares grades on correctness of answers with each of classification results a prescribed number of times by the classification function. The calculation function calculates a certainty factor of a grade error on the basis of the comparison results. The determination function determines whether or not a grade is incorrect on the basis of the certainty factor.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a program, an information processing device, and an information processing method. [Background technology]

[0002] Conventionally, there are known techniques for supporting the grading of answer sheets in examinations, etc. For example, Patent Document 1 discloses an information processing device that extracts marking marks on each answer sheet from an image representing an answer sheet, identifies the extracted marking marks, classifies them into "◯ (correct answer mark)" "× (incorrect answer mark)", etc., and determines marking marks that may have been erroneously identified based on the similarity between marking marks in the same classification. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-86297 Summary of the Invention [Problem to be solved by the invention]

[0004] Here, there is a need to detect marks that may have been incorrectly marked by a grader, etc., among the marks made by a grader, etc. regarding the correctness of answer sheets in an examination, etc. However, the technology of Patent Document 1 identifies and judges marking marks on the assumption that the marks made by the grader, etc. are correct, making it difficult to detect marks that may have been incorrectly marked by the grader, etc.

[0005] Therefore, the present invention aims to provide a program, an information processing device, and an information processing method that can detect any marks made by a grader regarding the correctness of answers in an exam, etc., that may have been made incorrectly by a grader, etc.

[0006] A program according to one aspect of the present invention enables a computer to implement a classification function that classifies the correctness of an answer indicated by answer data a predetermined number of times using a classification model learned from the answer data; a comparison function that compares the score for correctness of the answer with each of the results of the classification performed by the classification function a predetermined number of times; a calculation function that calculates the degree of certainty that the score is incorrect based on the results of the comparison; and a judgment function that determines whether the score is incorrect or not based on the degree of certainty.

[0007] An information processing device according to one embodiment of the present invention includes a classification unit that classifies the correctness of an answer indicated by answer data a predetermined number of times using a classification model learned from the answer data, a comparison unit that compares the score for correctness of the answer with each of the results of the classification by the classification unit for the predetermined number of times, a calculation unit that calculates a degree of certainty that the score is incorrect based on the results of the comparison, and a judgment unit that determines whether the score is incorrect or not based on the degree of certainty.

[0008] In one aspect of the information processing method of the present invention, a computer classifies the correctness of an answer indicated by answer data a predetermined number of times using a classification model learned from the answer data, compares the score for the correctness of the answer with each of the classification results for the predetermined number of times, calculates a certainty that the score is incorrect based on the results of the comparison, and determines whether the score is incorrect based on the certainty. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a program, an information processing device, and an information processing method that can detect answers that may have been incorrectly marked by a grader, etc., from among the marks made by a grader, etc. regarding the correctness of the answer sheets. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of the system configuration of a scoring support system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram illustrating an example of an overview of a scoring system according to an embodiment of the present invention. [Figure 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a server device according to the present embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a classification model according to the present embodiment. [Figure 5] 10 is a table illustrating an example of a classification result according to the present embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of the operation of the server device according to the embodiment. [Figure 7] FIG. 2 is a diagram illustrating an example of a hardware configuration of a server device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] A preferred embodiment of the present invention (hereinafter referred to as "the present embodiment") will be described with reference to the accompanying drawings. In the drawings, components with the same reference numerals have the same or similar configurations.

[0012] In this invention, the terms "unit," "means," "device," and "system" do not simply mean physical means, but also include cases where the functions of the "unit," "means," "device," and "system" are realized by software. Furthermore, the functions of one "unit," "means," "device," or "system" may be realized by two or more physical means or devices, and the functions of two or more "units," "means," "device," or "system" may be realized by one physical means or device.

[0013] <1. System configuration> An example of the system configuration of a scoring support system 1 according to this embodiment will be described with reference to FIG.

[0014] The scoring support system 1 is a system that supports users in scoring answer sheets (hereinafter simply referred to as "answer sheets") of examinees for exams administered at school, cram school, home, etc. The scoring support system 1 can be used not only for exams but also for scoring answer sheets for everyday study questions. The scoring support system 1 detects possible incorrect markings by a grader (hereinafter simply referred to as "marking errors") from among the marks (hereinafter simply referred to as "markings") made by the grader regarding the correctness of the answer sheets, and presents the detected possible marking errors to the user on a screen, etc.

[0015] 1, the scoring support system 1 includes a server device 100 used by a provider of the scoring support service, a user terminal 200 used by a user, and a scanner 300. The server device 100, the user terminal 200, and the scanner 300 are connected to each other via a network N so that they can communicate with each other.

[0016] The network N is composed of a wireless network and a wired network. Examples of the network N include a mobile phone network, a PHS (Personal Handy-phone System) network, a wireless LAN (Local Area Network), 3G (3rd Generation), LTE (Long Term Evolution), 4G (4th Generation), 5G (5th Generation), WiMax (registered trademark), infrared communication, Bluetooth (registered trademark), a wired LAN, a telephone line, a power line network, and a network conforming to IEEE1394 or the like.

[0017] The server device 100 is an information processing device for providing a marking support service. By executing a predetermined program, the server device 100 realizes various functions for determining whether the marking of an answer sheet data input by a user is incorrect or not and for providing the result of this determination to the user.

[0018] The answer data is data related to an answer. The answer data may be, for example, data including an image of an answer created by scanning an answer sheet (hereinafter simply referred to as an "answer sheet") on which an examinee has filled out an answer. As another example, the answer data may be data including text indicating the content of the question and the content of the answer to that question.

[0019] The answer data may be, for example, a label with the result of grading by a grader for each answer. Alternatively, as another example, the answer data may be provided separately, and grading data indicating the result of grading each answer may be associated with the answer data.

[0020] The answer data may represent answers from multiple subjects and / or multiple people, for example, it may represent answers from 1,000 people to test questions in three subjects: Japanese, science, and social studies.

[0021] The user terminal 200 is an information processing device used by a user, such as a smartphone, laptop, etc. By executing a predetermined program, the user terminal 200 cooperates with the server device 100 to send and receive information (hereinafter also referred to as "output information") about a screen for outputting detected scoring error candidates to the user, displays this screen, and accepts setting input of a threshold value, which will be described later, from the user.

[0022] The scanner 300 is an image input device that converts an answer sheet into answer data, which is digital image data. Note that the scanner 300 may be any device that can read a paper answer sheet and convert it into digital image data, and may be a terminal device such as a smartphone or laptop with a camera function.

[0023] <2. Overview> An overview of the scoring support system 1 will be described with reference to Fig. 2. As shown in Fig. 2, the scoring support system 1 first reads answer data. Next, the scoring support system 1 uses machine learning technology to calculate the degree of certainty that the marking of an answer in the answer data is incorrect (hereinafter also referred to as "mistake certainty T"). Specifically, the scoring support system 1 compares the result of marking the answer by the grader regarding the correctness of the answer with each of the correctness / incorrectness results of the answer classified multiple times by the server device 100, and calculates the mistake certainty T based on the result of the comparison. Next, the scoring support system 1 compares the calculated mistake certainty T with a preset threshold T c As a result of this comparison, the scoring support system 1 determines whether the error certainty T is greater than or equal to the threshold T c If it is equal to or greater than the threshold value T (Yes), the marking support system 1 determines that the marking error is a candidate. c If it is less than the threshold T (No), it is determined that it is not a candidate for scoring error. c may be a value that is set according to at least one of the type of question for the answer sheet and the format of the answer sheet, for example.

[0024] In the past, it was necessary for a long time for a person to manually check whether there were any errors in the marking of marked answer data. With the above configuration, the marking support system 1 can detect marks that may have been marked incorrectly by a marking person, etc., from marks made by a marking person, etc., on answer sheets in the answer data. Therefore, the marking support system 1 can reduce the amount of manual checking work and accurately detect marks that may have been marked incorrectly.

[0025] <3. Functional configuration> The functional configuration of the server device 100 according to this embodiment will be described with reference to Fig. 3. As shown in Fig. 3, the server device 100 includes a control unit 110, a communication unit 120, and a storage unit .

[0026] [Control Unit] The control unit 110 includes an acquisition unit 111, a classification unit 112, a comparison unit 113, a calculation unit 114, and a determination unit 115. The control unit 110 may also include, for example, a setting unit 116 and / or an output unit 117.

[0027] [Acquisition Department] The acquisition unit 111 acquires answer data from the user terminal 200, the scanner 300, or the like. The acquisition unit 111 may acquire, for example, answer data that has been read by the scanner 300 via the communication unit 120 and converted into paper answer sheets. The acquisition unit 111 may also acquire answer data via the communication unit 120 from, for example, the user terminal 200 or another external device (not shown). The acquisition unit 111 stores the acquired answer data in the storage unit 130.

[0028] [Classification Department] The classification unit 112 refers to the model storage unit 131 and the answer data storage unit 132, and classifies the correctness or incorrectness of the answer indicated by the answer data a predetermined number of times using a classification model learned from the answer data. The predetermined number of times is the number of times the correctness or incorrectness of the answer is classified, and is multiple times. The predetermined number of times is also expressed as N times (N: natural number). Classifying the correctness or incorrectness of this answer means classifying each answer in the answer data into either a correct answer or an incorrect answer. For example, if the predetermined number of times is 10 times (N=10), the classification unit 112 classifies the correctness or incorrectness of each answer in the answer data 10 times. For example, if the answer data is image data, the classification unit 112 may use OCR (Optical Character Recognition) to analyze an image of an answer sheet read by a scanner 300 or the like and recognize character strings of the questions and answers.

[0029] The classification model is a model for classifying whether an answer is correct or incorrect, and is trained by a machine learning technique using training data, for example.

[0030] The classification unit 112 may train a classification model using, for example, answer sheet data. The classification model may include, for example, a recurrent neural network (RNN) as a model using a deep learning technique. Furthermore, the classification model may include, for example, a long short-term memory (LSTM) among RNNs that can take into account long-term time series.

[0031] An example of classification processing using the classification model will now be described with reference to Fig. 4. In this example, an answer to a written question will be used as an example. Note that, as a premise, when the answer data is image data, the classification unit 112 is assumed to recognize the character string of the answer in the answer data.

[0032] (1) As shown in Figure 4, the classification model performs morphological analysis on the answer sentences (character strings of the answer) of the answer data, and divides the sentences into word units (in other words, into parts of speech). This morphological analysis process may be realized using known morphological analysis software such as Mecab or JUMAN++.

[0033] (2) The classification model includes BERT (Bidirectional Encoder Representations from Transformers), which vectorizes the text of words decomposed in (1) above. Another example of a natural language processing model for vectorization in this way may be Word2Vec.

[0034] (3) The classification model includes an LSTM that constitutes an encoder and a decoder, and inputs the word group vectorized in (2) above into the LSTM.

[0035] (4) The classification model includes an attention mechanism that weights words contained in the answer sheet and classifies the answer sheet as correct or incorrect based on these weighted words. Specifically, the attention mechanism weights the output of the LSTM decoder based on the intermediate representation output from the LSTM encoder. With this configuration, the attention mechanism can weight important words in each answer sheet and perform learning.

[0036] (5) The classification model normalizes the values ​​output from the attention mechanism using a softmax function and outputs a binary value ('0' or '1') for each answer indicating whether it is correct or incorrect.

[0037] Returning to FIG. 3, the explanation will be continued. The classification unit 112 may train the classification model using, for example, a k-fold cross-validation technique. Specifically, the classification unit 112 first divides the answer sheet data. Next, the classification unit 112 performs a combination in which a portion of the divided data is used as training data and the remaining portion of the divided data is used as validation data. This divided data will hereinafter also be referred to as "divided data." Then, the classification unit 112 trains the classification model multiple times by switching the divided data used as training data and the divided data used as validation data in each combination (in other words, shuffling each time).

[0038] For example, when using 4-fold cross-validation, the classification unit 112 randomly divides the answer sheet data into four (k=4) pieces. The classification unit 112 performs a combination in which one piece of the divided data is used as validation data (also referred to as test data) and the remaining three pieces of divided data are used as training data (also referred to as learning data). The classification unit 112 trains a classification model using this combination the first time. In the second time, the classification unit 112 replaces the validation data from the first divided data with another piece of divided data. The classification unit 112 also combines this replaced validation data with the divided data that was the validation data in the first time as training data, and trains a classification model using this combination. In this way, the classification unit 112 trains a classification model by switching the divided data used as training data and the divided data used as validation data each time so that all four pieces of divided data are used as validation data once each time. The classification unit 112 then generates four classification models based on the results of this training, averages these four generated classification models, and finally integrates them into a single classification model.

[0039] According to the above configuration, the classification unit 112 can efficiently use answer sheet data even if the amount of data is small. In addition, over-learning can be suppressed, and generalization performance can be improved.

[0040] [Comparison section] The comparison unit 113 compares the marking of the answer data as to whether the answer is correct or incorrect with each of the classification results for a predetermined number of times by the classification unit 112. For example, if the predetermined number of times is 10 times and the marking by the grader as to whether answer A is correct or incorrect is "correct," the comparison unit 113 compares this marking "correct answer" with each of the results of the classification unit 112 classifying answer A as correct or incorrect 10 times.

[0041] [Calculation section] Based on the result of the comparison by the comparison section 113, the calculation section 114 calculates the degree of certainty that the marking is incorrect (that is, the mistake certainty T).

[0042] The calculation unit 114 may calculate, for example, the rate at which the scoring and classification results in the comparison result by the comparison unit 113 differ (hereinafter also referred to as "rate") as the error certainty T. With this configuration, the calculation unit 114 can calculate the rate at which the scoring and classification results differ, and obtain the error certainty T. Therefore, the degree of likelihood (certainty) of a scoring error can be obtained depending on the degree to which the scoring and classification results differ.

[0043] The calculation unit 114 may calculate, as the error certainty T, the number of times that the scores (a) in the comparison result and the classification result (d) by the classification unit 112 differ (C a ≠ d ) and divide the counted number by a predetermined number to calculate the ratio. With this configuration, the calculation unit 114 can calculate the ratio (miss certainty T) by a simple calculation, and since the ratio is calculated by dividing by a constant rather than a variable, the calculation cost can be reduced.

[0044] [Judgment section] The determination unit 115 determines whether or not the marking is incorrect based on the error certainty T calculated by the calculation unit 114. For example, for marking that is determined to be incorrect, the determination unit 115 labels the answer sheet data as a candidate for a marking error and stores it in the storage unit 130.

[0045] According to the above configuration, the determination unit 115 can determine whether or not the marking is incorrect using a statistical index that indicates the degree of certainty of the marking error. Therefore, the determination unit 115 can detect marks that may have been incorrectly marked by the grader from among the marks made by the grader on the answer sheets.

[0046] The determination unit 115 determines whether the ratio calculated by the calculation unit 114 as the error certainty T is equal to or exceeds a threshold T c If the score exceeds 0, the score may be determined to be incorrect.

[0047] <Classification, Comparison, Calculation, and Judgment Processes> Here, an example of processing by each of the classification unit 112, comparison unit 113, calculation unit 114, and determination unit 115 will be described with reference to FIG. 5. In this example, a correct answer on an answer sheet is represented by "◯" and an incorrect answer is represented by "×". In addition, the predetermined number of times is set to 10 times (N=10), and the threshold T c is set to "0.5".

[0048] An example will be given using answers 1 and 2. As shown in FIG. 5, first, for answer 1, the grader marks the answer as "O" and the classification unit 112 classifies the answer ten times as "O, O, O, X, O, O, O, O, O, O." The calculation unit 114 compares the marking of answer 1 with each of the ten classification results, and calculates the number of times the marking and classification results differ (C a ≠ d ) (in this example, the number of times that they differ is 1). The calculation unit 114 calculates the percentage of times that the scoring and classification results differ as the error certainty T by dividing the counted number (1) by a predetermined number (10 in this example) to obtain "0.1". The determination unit 115 calculates the error certainty T (0.1) by dividing the calculated error certainty T by a threshold value T c Since the difference is less than (0.5), it is determined that the marking of answer 1 is not an error (i.e., it is not a marking error). Next, for answer 2, the marking by the marking officer is the correct answer "X", while the results of the 10 classifications by the classification unit 112 are "O, O, O, O, O, O, O, O, O, O, O". The calculation unit 114 calculates the number of times that the marking and classification results differ for answer 2 (C a ≠ d ) is counted as 10 times, and this counted number is divided by a predetermined number to calculate "1.0". The determination unit 115 determines whether this calculated mistake certainty T(1.0) is equal to or lower than the threshold T c Since the score exceeds (0.5), the marking of Answer 2 is judged to be incorrect (i.e., a marking error).

[0049] [Settings section] The setting unit 116 sets the threshold T c The setting unit 116 sets the threshold T c In other words, the threshold Tc may be changed according to at least one of the type of question and the format of the answer. The "type of question" may be, for example, the subject to which the question belongs (for example, the test subject such as science, social studies, or Japanese). The "format of answer" may be, for example, a fill-in-the-blank format or a free description format (in other words, the question format). As another example, the setting unit 116 may change the threshold T c may be set.

[0050] According to the above configuration, the threshold Tc of The threshold Tc can be set taking into consideration the type of question and the subject to which the question belongs. For example, if the content is essentially the same, answers to science and social studies questions tend to be written in similar ways, while answers to Japanese language questions tend to be written in a variety of ways. Also, for example, free-form questions tend to be written in a more varied way than fill-in-the-blank questions. In this way, the threshold Tc can be set taking into consideration the fact that variations in expression vary depending on the type of question and the format of the answer.

[0051] [Output section] The output unit 117 generates output information to be output to the screen or the like of the user terminal 200. The output unit 117 transmits this generated output information to the destination user terminal 200 via the communication unit 120. For example, the output unit 117 generates output information to output, as a candidate for scoring error, a score determined to be incorrect by the determination unit 115, and transmits this generated output information to the user terminal 200 via the communication unit 120.

[0052] [Communications Department] The communication unit 120 transmits and receives various information including answer data and output information to and from the user terminal 200, the scanner 300, and the like via the network N.

[0053] [Storage] The storage unit 130 stores classification models, answer data, output information, etc. The storage unit 130 may store each piece of information using a database management system (DBMS), or may store each piece of information using a file system. When using a DBMS, a table may be provided for each piece of information, and each piece of information may be managed by associating these tables.

[0054] The storage unit 130 may include, for example, a model storage unit 131. The model storage unit 131 stores the classification model. The storage unit 130 may also include, for example, an answer sheet data storage unit 132. The answer sheet data storage unit 132 stores answer sheet data.

[0055] <4. Example of operation> An example of the operation of the server device 100 according to this embodiment will be described with reference to Fig. 6. Note that the order of the processes in the flow chart shown below is an example and may be changed as appropriate.

[0056] 6, the acquisition unit 111 of the server device 100 acquires answer data (step S11). Next, in loop 1, the classification unit 112 repeatedly executes the process of step S11 N times. The classification unit 112 classifies the answer indicated by the answer data into correct or incorrect answers using a classification model learned from the answer data (step S11).

[0057] The comparison unit 113 of the server device 100 compares the score on the correctness of the answer sheet with each of the N classification results by the classification unit 112 (step S12). The calculation unit 114 calculates an error certainty T indicating that the score is incorrect based on the result of this comparison (step S13). The determination unit 115 determines whether the score is incorrect or not based on the calculated error certainty T (step S14).

[0058] The output unit 117 causes the user terminal 200 to output the result of the determination made by the determination unit 115 (step S15).

[0059] <5. Hardware Configuration> 7, an example of a hardware configuration in which the above-described server device 100 is realized by a computer 800 will be described. Note that the functions of each device can also be realized by dividing them into multiple devices.

[0060] As shown in FIG. 7, the computer 800 includes a processor 801, a memory 803, a storage device 805, an input I / F unit 807, a data I / F unit 809, a communication I / F unit 811, and a display device 813.

[0061] The processor 801 controls various processes in the computer 800 by executing programs stored in the memory 803. For example, each functional unit included in the control unit 110 of the server device 100 can be realized by the processor 801 executing a program temporarily stored in the memory 803.

[0062] The memory 803 is a storage medium such as a RAM (Random Access Memory), etc. The memory 803 temporarily stores the program code of the program executed by the processor 801 and data required when the program is executed.

[0063] The storage device 805 is a non-volatile storage medium such as a hard disk drive (HDD) or flash memory. The storage device 805 stores an operating system and various models and programs for implementing the above-mentioned configurations. In addition, the storage device 805 can also store a table for registering answer data and a DB for managing this table. Such programs and data are loaded into the memory 803 as needed and can be referenced by the processor 801.

[0064] The input I / F unit 807 is a device for receiving input from a user. Specific examples of the input I / F unit 807 include a keyboard, a mouse, a touch panel, various sensors, and a wearable device. The input I / F unit 807 may be connected to the computer 800 via an interface such as a USB (Universal Serial Bus).

[0065] The data I / F unit 809 is a device for inputting data from outside the computer 800. A specific example of the data I / F unit 809 is a drive device for reading data stored in various storage media. The data I / F unit 809 may be provided outside the computer 800. In this case, the data I / F unit 809 is connected to the computer 800 via an interface such as a USB.

[0066] The communication I / F unit 811 is a device for performing data communication via the Internet N, either wired or wirelessly, with devices external to the computer 800. The communication I / F unit 811 may be provided outside the computer 800. In this case, the communication I / F unit 811 is connected to the computer 800 via an interface such as a USB.

[0067] The display device 813 is a device for displaying various types of information. Specific examples of the display device 813 include a liquid crystal display, an organic EL (Electro-Luminescence) display, and a display of a wearable device. The display device 813 may be provided outside the computer 800. In this case, the display device 813 is connected to the computer 800 via, for example, a display cable. Furthermore, when a touch panel is adopted as the input I / F unit 807, the display device 813 can be configured as an integral part of the input I / F unit 807.

[0068] The above-described embodiments are merely examples for explaining the present invention, and are not intended to limit the present invention to these embodiments. Furthermore, the present invention can be modified in various ways without departing from the spirit of the invention. Furthermore, those skilled in the art can adopt embodiments in which the above-described elements are replaced with equivalents, and such embodiments are also within the scope of the present invention.

[0069] The components of the server device described in the above embodiment are assumed to realize predetermined processing in cooperation with other hardware by the processor 801 executing a program stored in the storage device 805. In other words, these components are assumed to be software or firmware, as well as corresponding hardware, and in both of these concepts, they are also referred to as "functions," "means," "parts," "processing circuits," "units," or "modules," and can be interpreted as such.

[0070] [Variations] Although the present invention has been described based on the above embodiment, the following cases are also included in the present invention.

[0071] [Variation 1] At least a part of the components of the server device 100 according to the above embodiment may be included in the user terminal 200. For example, the determination unit 115 and the output unit 117 of the server device 100 may be implemented in the user terminal 200.

[0072] [Variation 2] In the above embodiment, an example has been described in which the scorer is a human, but the entity that performs the scoring according to the present invention is not limited to this. The scoring may be, for example, automatically performed by a machine. [Explanation of symbols]

[0073] 1...scoring support system, 100...server device, 110...control unit, 111...acquisition unit, 112...classification unit, 113...comparison unit, 114...calculation unit, 115...determination unit, 116...setting unit, 117...output unit, 120...communication unit, 130...storage unit, 200...user terminal, 00...computer, 801...processor, 803...memory, 805...storage device, 807...input I / F unit, 809...data I / F unit, 811...communication I / F unit, 813...display device.

Claims

1. On the computer, A classification function that classifies the correctness or incorrectness of the answer indicated by the answer data a predetermined number of times using a classification model learned from the answer data; a comparison function that compares the score of the answer sheet as to whether it is correct or incorrect with each of the results of the classification by the classification function for the predetermined number of times; a calculation function for calculating a degree of certainty that the scoring is incorrect based on the result of the comparison; a determination function for determining whether the scoring is an error or not based on the certainty factor, The calculation function calculates, as the confidence level, a rate at which the scores and the classification results differ from each other in the comparison result; The determining function determines that the scoring is incorrect if the ratio exceeds a threshold value; The threshold value is set according to at least one of the type of question for the answer sheet and the format of the answer sheet. program.

2. the calculation function counts the number of times that the scores in the comparison results differ from the classification results, and calculates the ratio by dividing the counted number by the predetermined number. The program according to claim 1.

3. The classification function divides the answer data, performs a combination in which a part of the divided data is used as training data and the remaining part of the divided data is used as verification data, and in the combination, the data used as training data and the data used as verification data are switched each time, thereby training the classification model multiple times. The program according to claim 1 or 2.

4. The answer sheet is an answer sheet to a written question, The classification model includes an attention mechanism that weights words included in the answer sheet, and classifies the answer sheet as correct or incorrect based on the weighted words. The program according to any one of claims 1 to 3.

5. a classification unit that classifies the correctness or incorrectness of the answer indicated by the answer data a predetermined number of times using a classification model trained by the answer data; a comparison unit that compares the score of the answer sheet as to whether it is correct or incorrect with each of the results of the classification by the classification unit for the predetermined number of times; a calculation unit that calculates a degree of certainty that the scoring is incorrect based on the result of the comparison; a determination unit that determines whether the scoring is an error based on the confidence level, the calculation unit calculates, as the confidence level, a rate at which the scoring and the classification results differ from each other in the comparison result; The determination unit determines that the scoring is incorrect when the ratio exceeds a threshold value; The threshold value is set according to at least one of the type of question for the answer sheet and the format of the answer sheet. Information processing device.

6. The computer Classifying the correctness of the answer indicated by the answer data a predetermined number of times using a classification model trained using the answer data; comparing the score of the answer sheet as to whether it is correct or incorrect with each of the predetermined number of classification results; calculating a confidence level that the scoring is incorrect based on the results of the comparison; determining whether the scoring is an error based on the confidence level; The calculating step calculates, as the confidence level, a percentage of the score and the classification result that are different from each other in the comparison result; determining that the scoring is incorrect if the ratio exceeds a threshold; The threshold value is set according to at least one of the type of question for the answer sheet and the format of the answer sheet. Information processing methods.

Images