Information processing device and program

Abstract

To provide an information processing device capable of evaluating an answer to a plotting question and assisting a marking job and the like by presenting the evaluation results.SOLUTION: An information processing device acquires an answer to a plotting question as answer result data first. The information processing device then acquires and outputs evaluation information on evaluation of targeted answer result data using an evaluation model which has already learned the relationship between the answer result data to the plotting question and the evaluation in advance. This enables the information processing device to present the evaluation results of the answer to the plotting question to an instructor and so on in a comprehensive way on the basis of the evaluation information.SELECTED DRAWING: Figure 18

Description

Technical Field

[0001] The present invention relates to a technique for grading answers to drawing problems.

Background Art

[0002] Conventionally, a system for automatically grading tests has been known. Patent Document 1 discloses a system that performs automatic grading by comparing an image of an answer sheet read by a scanner with an answer collection stored in a storage device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For drawing problems where the answer is a drawing such as a figure, since there are many variations in the answers, the load of grading and evaluation is high, which has been a great burden on instructors. Also, there has been a problem that it is difficult to automatically grade the answers to such drawing problems by a system.

[0005] The present invention has been made, for example, to solve the above problems, and an object thereof is to provide an information processing apparatus that evaluates an answer to a drawing problem and presents an evaluation result to assist in grading work and the like.

Means for Solving the Problems

[0006] In one aspect of the present invention, an information processing apparatus includes an answer result data acquisition unit that acquires an answer to a drawing problem as answer result data, an evaluation information acquisition unit that acquires evaluation information regarding evaluation of target answer result data using an evaluation model in which the relationship between answer result data and evaluation for the drawing problem has been learned in advance, and an evaluation information output unit that outputs the evaluation information. The system includes a storage unit that stores multiple evaluation models corresponding to multiple drawing problems and information relating to the drawing problems on which each evaluation model was constructed, and an evaluation model selection unit that selects an evaluation model from the storage unit based on information relating to the drawing problems corresponding to the target answer result data. The evaluation information acquisition unit acquires evaluation information using the selected evaluation model, and the evaluation model selection unit calculates the similarity between the information relating to the drawing problems corresponding to the target answer result data and the information relating to the drawing problems on which the evaluation model was constructed, and selects an evaluation model whose similarity is equal to or greater than a threshold.According to this embodiment, the information processing device can evaluate the answers to drawing problems and output evaluation results based on the evaluation information. This can assist instructors in evaluation and grading, and reduce their workload. Furthermore, according to this embodiment, the information processing device can select an appropriate evaluation model from the storage unit based on information related to the drawing problem and obtain evaluation information for the answer result data. Furthermore, according to this embodiment, the information processing device can select an appropriate evaluation model from the storage unit based on the similarity of information related to the drawing problem.

[0009] In one embodiment of the information processing device described above, the information relating to the drawing problem includes the text of the drawing problem statement, the image of the problem, and the image of the model answer. According to this embodiment, the information processing device can calculate the similarity between the text of the drawing problem statement, the image of the problem, and the image of the model answer, and select an appropriate evaluation model from the storage unit based on the calculated similarity.

[0010] In one embodiment of the information processing device described above, the target answer result data is evaluated as either a perfect score, a deduction, or zero points. According to this embodiment, the information processing device can present the evaluation of the answer result data to instructors, etc., as either a perfect score, a deduction, or zero points, based on the evaluation information. This makes it possible to present the evaluation to instructors, etc., in an easy-to-understand manner.

[0011] In one embodiment of the above-described information processing device, if the target answer result data is evaluated as a point deduction, the evaluation information acquisition unit acquires information about the areas that affected the evaluation, along with the evaluation information, as point deduction area information. According to this embodiment, the information processing device can output information about the areas that affected the evaluation, namely the point deduction, along with the evaluation.

[0012] In another aspect of the present invention, a program executed by an information processing device equipped with a computer, comprising: an answer result data acquisition unit that acquires the answer to a drawing problem as answer result data; an evaluation information acquisition unit that acquires evaluation information relating to the evaluation of the target answer result data using an evaluation model that has been previously trained on the relationship between the answer result data and evaluation for the drawing problem; and an evaluation information output unit that outputs the evaluation information. A storage unit that stores multiple evaluation models corresponding to multiple drawing problems and information about the drawing problems on which each evaluation model was constructed, and an evaluation model selection unit that selects the evaluation model from the storage unit based on information about the drawing problems corresponding to the target answer result data. The computer is made to function as follows: The evaluation information acquisition unit acquires evaluation information using the selected evaluation model, and the evaluation model selection unit calculates the similarity between the information regarding the drawing problem corresponding to the target answer result data and the information regarding the drawing problem on which the evaluation model was constructed, and selects an evaluation model whose similarity is equal to or greater than a threshold.By installing and running this program on a computer, the information processing apparatus according to the present invention can be configured.

Advantages of the Invention

[0013] According to the information processing apparatus of the present invention, it is possible to assist in scoring operations and the like by evaluating the answers to drawing problems and presenting the evaluation results.

Brief Description of the Drawings

[0014] [Figure 1] Shows the configuration of a learning support system to which the server of the present invention is applied. [Figure 2] An example of a question paper. [Figure 3] An example of an answer sheet. [Figure 4] An example of a model paper. [Figure 5] An example of data extracted from each sheet by region detection. [Figure 6] An example of the data configuration of the answer DB. [Figure 7] A diagram for explaining the evaluation criteria for drawing problems. [Figure 8] An example of grouping answer result images for each scoring result. [Figure 9] A diagram for explaining the learning of an evaluation model. [Figure 10] An example of the data configuration of the evaluation model DB. [Figure 11] A block diagram showing the hardware configuration of the server. [Figure 12] A block diagram showing the functional configuration of the server. [Figure 13] A diagram for explaining the method of calculating text similarity. [Figure 14] A diagram for explaining the method of calculating similarity using CNN. [Figure 15] A diagram for explaining the method of selecting an evaluation model based on similarity. [Figure 16] An example of an image of a deduction area. [Figure 17] This is an example of an evaluation result screen. [Figure 18] This is a flowchart of the evaluation process. [Figure 19] This shows the configuration of a learning support system to which a server of a modified example is applied.

Mode for Carrying Out the Invention

[0015] Hereinafter, embodiments of the present invention will be described while referring to the drawings. <Embodiment> [Overall Configuration] FIG. 1 shows the configuration of a learning support system to which the server of the present invention is applied. The learning support system 100 is a system that assists in scoring operations and the like by evaluating answers to drawing problems and presenting evaluation results. The learning support system 100 is configured such that a plurality of learner terminals 9, an instructor terminal 10, and a server 20 can communicate with each other via a network 5 such as the Internet. In the present embodiment, a drawing problem is a problem that requires drawing a figure or the like as an answer. Also, a learner is a student or the like who works on drawing problems, and an instructor is a teacher or the like who grades answers to drawing problems.

[0016] The learner terminal 9 is used by a learner and is an information processing device such as a tablet or a personal computer (PC). Also, the instructor terminal 10 is used by an instructor and is an information processing device such as a tablet or a PC, similar to the learner terminal 9. Specifically, the learner terminal 9 is a terminal device that displays a test question paper including drawing problems and allows a learner to input answers to the question paper. On the other hand, the instructor terminal 10 is a terminal device that transmits information regarding the question paper and the answer paper on which the learner has input answers to the server 20 and receives and displays screen information regarding the evaluation result screen from the server 20.

[0017] Server 20 is an information processing device that processes, stores, and transmits various types of information, and can be, for example, a server device, a PC, or a general-purpose tablet. Server 20 is also connected to the answer database (hereinafter referred to as "DB") 31 and the evaluation model DB 32, which will be described later.

[0018] [Question Paper] Here, the test paper used in this embodiment will be described. Figure 2 is an example of a test paper that includes drawing problems. In this embodiment, the test paper is a digital teaching material and is displayed on a tablet or the like used by each learner. As shown in Figure 2(a), the test paper consists of a test subject section 22, a name input section 23, and an answer input section 24. The test subject section displays information such as the test date, target grade level, test subject, and test time. The name input section 23 has multiple fields for inputting information such as the grade level, class, number, and name of the learner who took the test. The answer input section 24 displays the question text of the problems that make up the test and has multiple fields for inputting the answer to each problem.

[0019] In this embodiment, the problem includes calculation problems in which numbers or symbols are entered as answers, and drawing problems in which figures or the like are created as answers. Specifically, as shown in Figure 2(b), the answer input section 24 of the problem sheet includes problem text areas for calculation problems (hereinafter also referred to as "calculation problem text areas") 60-64, shown by coarse dotted lines, and answer areas for calculation problems (hereinafter also referred to as "calculation answer areas") 65a, 66a, 67a, and 68a, shown by thin solid lines. The answer input section 24 also includes a problem text area for drawing problems (hereinafter also referred to as "drawing problem text area") 69, shown by a thick solid line, and an answer area for drawing problems (hereinafter also referred to as "drawing answer area") 70, shown by fine dotted lines. For the sake of explanation, the solid and dotted lines representing each area are shown in the diagram, but they are not actually displayed.

[0020] Furthermore, it is assumed that nothing is entered in the name input section 23 and the answer input section 24 of the question paper. In this embodiment, a question paper in which information about the learner is entered in the name input section 23 and the learner's answers are entered in the answer input section 24 is referred to as the "answer sheet." In addition, a question paper in which a model answer is entered in the answer input section 24 is referred to as the "model sheet."

[0021] [Answer sheet] Next, the test answer sheet used in this embodiment will be described. Figure 3 is an example of a test answer sheet that includes drawing problems. The test answer sheet is a digital teaching material. The answer sheet is a question sheet into which predetermined information has been entered by the learner, and as shown in Figure 3(a), it consists of a test subject section 22, a name input section 23, and an answer input section 24. The learner uses the learner terminal 9 to enter their grade, class, number, and name in each item of the name input section 23. The learner also enters the answers to the problems corresponding to each item of the answer input section 24. In other words, the learner's grade, class, number, and name are entered in each item of the name input section 23 of the answer sheet. The learner's answers to the corresponding problems are entered in each item of the answer input section 24.

[0022] As shown in Figure 3(b), the answer sheet, like the question sheet, includes calculation problem statement areas 60-64, calculation answer areas 65b, 66b, 67b, and 68b, drawing problem statement area 69, and drawing answer area 70b.

[0023] [Model form] Next, a model test sheet used in this embodiment will be described. Figure 4 is an example of a model test sheet that includes drawing problems. The model test sheet is a digital teaching material. The model sheet is a question sheet with model answers for the test entered, and as shown in Figure 4(a), it consists of a test subject section 22, a name input section 23, and an answer input section 24. Each item in the answer input section 24 has the model answer for the corresponding problem entered.

[0024] As shown in Figure 4(b), the model paper, like the problem paper, includes calculation problem text areas 60-64, calculation answer areas 65c, 66c, 67c, and 68c, drawing problem text area 69, and drawing answer area 70c.

[0025] [Region detection] Next, we will explain the detection of areas on each sheet of paper by the server 20. The server 20 can apply any area detection method to the question paper, answer sheet, or model sheet. Here, the area detection method is a method that detects the positional information of each area contained in a given sheet of paper based on information about that sheet of paper, and assigns a label to each area.

[0026] For example, a region detection method can be applied that uses deep learning, such as R-CNN (Rich feature hierarchies for accurate object detection and semantic segmentation), to detect objects in an image as rectangular regions.

[0027] Specifically, let's explain the region detection for the answer input section 24 of the answer sheet shown in Figure 3(a). In this case, the server 20 first detects the positional information for the calculation problem text regions 60-64, the calculation answer regions 65a, 66a, 67a, and 68a, the drawing problem text region 69, and the drawing answer region 70a, as shown in Figure 3(b). Then, the server 20 assigns labels to the calculation problem text regions 60-64 to indicate that they are regions of characters and symbols representing calculation problems. Similarly, the server 20 assigns labels to the calculation answer regions 65a, 66a, 67a, and 68a to indicate that they are regions of answers to calculation problems, labels to the drawing problem text region 69 to indicate that it is a region of characters and symbols representing a drawing problem, and labels to the drawing answer region 70b to indicate that it is a region of answers to a drawing problem.

[0028] Figure 5 shows an example of data extracted from each sheet by region detection. By detecting the region of the answer sheet, the server 20 obtains, for example, the coordinates indicating the region 69 of the drawing problem statement on the answer sheet as location information. Based on the obtained coordinates, the server 20 obtains the text of the drawing problem statement (hereinafter also referred to as "problem statement text") as problem statement data from the information on the answer sheet, as shown in Figure 5(a). Similarly, by detecting the region of the answer sheet, the server 20 obtains the location information of the drawing answer region 70b and obtains the image of the learner's answer to the drawing problem (hereinafter also referred to as "answer result image") as answer result data, as shown in Figure 5(b).

[0029] Server 20 performs region detection on the question paper and the model paper, similar to region detection on the answer paper. By performing region detection on the question paper, Server 20 obtains location information of the drawing answer area 70a and, from the information on the question paper, obtains an image pre-printed in the answer to the drawing problem (hereinafter also referred to as the "problem image") as shown in Figure 5(c) as problem image data. Furthermore, by performing region detection on the model paper, Server 20 obtains location information of the drawing answer area 70c and obtains an image of the model answer to the drawing problem (hereinafter also referred to as the "model answer image") as model answer data, as shown in Figure 5(d).

[0030] [Answer DB] Next, we will explain the answer database 31. The answer database 31 stores information regarding the answers and scoring of the drawing problems. Figure 6 shows an example of the data structure of the answer database 31. As shown in Figure 6, the answer database 31 stores the following in association: a problem ID that identifies the drawing problem, the name of the learner who answered the drawing problem, the answer result data which is an image of the learner's answer, the points allocated to the drawing problem, and the scoring result of the learner's answer.

[0031] Here, we will explain the evaluation criteria for the construction problems included in the problem sheet shown in Figure 2. Figure 7 is a diagram illustrating the evaluation criteria for construction problems. In such construction problems, as shown in Figure 7, first, draw a curve that is the arc of a circle centered at A using a compass. Similarly, draw a curve that is the arc of a circle centered at B using a compass. Then, draw a straight line passing through the two points that are the intersections of the two curves. This straight line is the perpendicular bisector of line AB. Draw a circle using a compass, with the intersection of line AB and the perpendicular bisector as the center, passing through A and B. This is the model answer to the construction problem, "Construct a circle with line segment AB as its diameter," and the criteria for evaluation and grading are two points: "Is the perpendicular bisector drawn accurately?" and "Is the circle passing through AB drawn accurately?"

[0032] When an instructor grades a drawing problem, they should determine whether the answer meets the two criteria. For example, if both criteria are met, the answer receives full marks; if only one criterion is met, the score is reduced by 50%; and if neither criterion is met, the answer receives 0 points. Alternatively, if the answer roughly meets the two criteria but not perfectly, the score may be reduced by 20%; if it roughly meets the one criterion, the score may be reduced by 80%, and so on. The grading method is at the instructor's discretion.

[0033] The storage configuration of the solution DB31 described above is just one example; other storage configurations may be used as long as the relationships between the data are maintained. Furthermore, the solution DB31 can be implemented using recording media such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0034] [Building an evaluation model] Next, we will explain how to build an evaluation model. Training data is used to build (generate) an evaluation model. Training data is data that associates the input data used in training the evaluation model with the corresponding ground truth data.

[0035] In this embodiment, the input data consists of various learners' answer result images for a given drawing problem, and the correct answer data consists of the scoring results of the answer result images. At this time, one evaluation model is constructed for each drawing problem. However, it is not limited to constructing one evaluation model for one drawing problem. For example, if the wording of the drawing problem is different but the meaning is the same, such as "Draw a circle with line segment AB as its diameter" and "Let's draw a circle with line segment AB as its diameter," or if the problem image and model answer image of the drawing problem are similar, one evaluation model may be constructed for multiple similar drawing problems. Whether or not the problem image and model answer image are similar can be determined by quantifying the similarity using a similarity calculation method based on perfect match or a similarity calculation method using CNN (Convolutional Neural Network), or it can be determined by visual inspection by an instructor. Here, the similarity calculation method based on perfect match is, for example, a method in which the sizes of the target images are matched, the difference between pixels is calculated, and the sum of the differences is taken as the similarity. For convenience, the method for calculating similarity using CNNs will be explained in detail later.

[0036] Specifically, the generation of training data based on the answer DB31 will be explained. Figure 8 shows an example of answer result images being grouped by scoring result. As shown in Figure 8, Server 20 retrieves answer result images from the answer DB31 where the scoring result is the same as the point allocation, i.e., a perfect score, and groups them into the perfect score group. Server 20 then associates the correct answer data for "perfect score" with the answer result images in the perfect score group. Server 20 also retrieves answer result images from the answer DB31 where the scoring result is 0 points, and groups them into the 0-point group. Server 20 then associates the correct answer data for "0 points" with the answer result images in the 0-point group.

[0037] Furthermore, Server 20 retrieves answer result images from Answer DB 31 where the scoring result is 20% less than the allocated score, and groups them into the 20% deduction group. Server 20 then associates the correct answer data for "20% deduction" with the answer result images in the 20% deduction group. Server 20 also retrieves answer result images from Answer DB 31 where the scoring result is 50% less than the allocated score, and groups them into the 50% deduction group. Server 20 then associates the correct answer data for "50% deduction" with the answer result images in the 50% deduction group. In this way, Server 20 generates training data based on Answer DB 31.

[0038] In this embodiment, for the sake of explanation, the 20% deduction group and the 50% deduction group will be combined into a single deduction group, and the correct "deduction" data will be associated with them. In this way, grouping other than the perfect score group and the 0-point group can be set arbitrarily.

[0039] Using the generated training data, Server 20 trains an evaluation model to estimate the probabilities of "perfect score," "deduction," and "zero score." Examples of machine learning methods include CNN deep learning. Figure 9 illustrates the training of the evaluation model. Specifically, as shown in Figure 9, when Server 20 receives an image of the perfect score group's answer results, it trains the evaluation model to estimate the probability of a perfect score as "100%", the probability of a deduction as "0%", and the probability of a zero score as "0%". Similarly, when Server 20 receives an image of the deduction group's answer results, it trains the evaluation model to estimate the probability of a perfect score as "0%", the probability of a deduction as "100%", and the probability of a zero score as "0%". Furthermore, when Server 20 receives an image of the zero score group's answer results, it trains the evaluation model to estimate the probability of a perfect score as "0%", the probability of a deduction as "0%", and the probability of a zero score as "100%".

[0040] In this embodiment, the server 20 constructs the evaluation model, but the present invention is not limited to this. The evaluation model may be constructed on another server, and the server 20 may store the constructed evaluation model.

[0041] [Evaluation Model Database] Next, we will describe the evaluation model DB32. The evaluation model DB32 stores information about the evaluation model. Figure 10 shows an example of the data structure of the evaluation model DB32. As shown in Figure 10, the evaluation model DB32 stores the model ID that identifies the evaluation model, the evaluation model itself, and the model construction data in association with each other. The model construction data is information about the drawing problem corresponding to the evaluation model, and is information about the drawing problem corresponding to the answer result data used to train the evaluation model.

[0042] Specifically, in this embodiment, the problem statement data, problem image data, and model answer data that constitute the model construction data are the problem statement text, problem image, and model answer image of a drawing problem, respectively. For example, as shown in Figure 10, the evaluation model DB32 stores the problem statement text "Draw a circle with line segment AB as its diameter," problem image 35a, and model answer image 36a, and the problem statement text "Let's draw a circle with line segment AB as its diameter," problem image 35b, and model answer image 36b, associated with the evaluation model with model ID "XXX". It also stores the problem statement text "Draw a triangle similar to triangle A," problem image 37, and model answer image 38, associated with the evaluation model with model ID "XXY".

[0043] The storage configuration of the evaluation model DB32 described above is just one example; other storage configurations may be used as long as the relationships between data are maintained. Furthermore, the evaluation model DB32 can be implemented using recording media such as HDDs or SSDs.

[0044] [Server Configuration] Figure 11 is a block diagram showing the hardware configuration of server 20. Server 20 acquires and outputs evaluation information of target answer result data using an evaluation model, and comprises a communication unit 11, a control unit 12, a storage unit 13, a recording medium 14, a display unit 15, and an input unit 16. These components, the answer DB 31, and the evaluation model DB 32 are interconnected via a bus 19.

[0045] Server 20 can run on a single computer, or it can run in a distributed manner across multiple computers, or it can run in a distributed manner across virtual machines.

[0046] The communication unit 11 is a communication unit that communicates with the learner terminal 9 and the instructor terminal 10 via the network 5. Specifically, the communication unit 11 receives information about the question paper, the answer sheet, and the model sheet from the instructor terminal 10, and transmits screen information related to the evaluation result screen to the instructor terminal 10.

[0047] The control unit 12 includes arithmetic processing units such as a CPU (Central Processing Unit), MPU (Micro-Processing Unit), and GPU (Graphics Processing Unit), and performs various information processing and control processing related to the server 20 by reading and executing programs stored in the memory unit 13. The programs can be deployed on a single computer or site, or distributed across multiple sites and executed on multiple computers interconnected by a communication network. Although Figure 11 describes the control unit 12 as a single processor, it may also be a multi-processor system.

[0048] The storage unit 13 includes memory elements such as RAM (Random Access Memory) and ROM (Read Only Memory), and stores programs or data necessary for the control unit 12 to execute processing. The storage unit 13 also temporarily stores data necessary for the control unit 12 to execute arithmetic processing.

[0049] The recording medium 14 is a non-volatile, non-temporary recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from the server 20. The recording medium 14 stores various programs that the control unit 12 executes. When the server 20 performs evaluation processing, the programs stored on the recording medium 14 are loaded into the storage unit 13 and executed by the control unit 12.

[0050] The display unit 15 is a liquid crystal display or an organic EL (electroluminescence) display, etc., and displays various information according to the instructions of the control unit 12. The input unit 16 is an input device such as a mouse, keyboard, touch panel, or buttons, and outputs the received operation information to the control unit 12.

[0051] In this embodiment, the storage unit 13 and the various databases may be configured as a single storage device, or they may be separate storage devices. Furthermore, the various databases may be external storage devices connected to the server 20, and their configuration can be set arbitrarily.

[0052] Figure 12 is a block diagram showing the functional configuration of server 20. Functionally, server 20 comprises an answer DB 31, an evaluation model DB 32, an input information acquisition unit 41, an area detection unit 42, a problem statement data acquisition unit 43, an answer result data acquisition unit 44, a problem image data acquisition unit 45, a model answer data acquisition unit 46, an evaluation model selection unit 47, an evaluation information acquisition unit 48, a deduction area information acquisition unit 49, an evaluation result screen creation unit 50, and a screen output unit 51.

[0053] The input information acquisition unit 41, region detection unit 42, problem statement data acquisition unit 43, answer result data acquisition unit 44, problem image data acquisition unit 45, model answer data acquisition unit 46, evaluation model selection unit 47, evaluation information acquisition unit 48, deduction area information acquisition unit 49, evaluation result screen creation unit 50, and screen output unit 51 are realized by the control unit 12 executing a program.

[0054] The input information acquisition unit 41 acquires input information from the instructor terminal 10 along with a request for the evaluation results screen. The input information includes question paper information, answer sheet information, and model sheet information. Specifically, the instructor uses the instructor terminal 10 to acquire each student's answer sheet information from each student's student terminal 9. The instructor then sends the acquired question paper information, along with the corresponding answer sheet information and model sheet information, as input information to the server 20 along with a request for the evaluation results screen. As a result, the server 20 acquires the input information from the instructor terminal 10. In this embodiment, the instructor terminal 10 transmits the input information to the server 20, but the present invention is not limited to this, and the method of acquiring input information by the server 20 can be arbitrarily set.

[0055] The region detection unit 42 detects the location information of each region included in the answer sheet using an arbitrary region detection method based on the answer sheet information and assigns a corresponding label. The region detection unit 42 also detects the location information of each region included in the question paper using an arbitrary region detection method based on the question paper information and assigns a corresponding label. Furthermore, the region detection unit 42 detects the location information of each region included in the model paper using an arbitrary region detection method based on the model paper information and assigns a corresponding label.

[0056] The problem statement data acquisition unit 43 acquires the problem statement text from the answer sheet information based on the position information indicating the drawing problem statement area 69 on the answer sheet. The problem statement text acquired by the problem statement data acquisition unit 43 is used to select an evaluation model.

[0057] The answer result data acquisition unit 44 acquires an answer result image from the answer sheet information based on the position information indicating the drawing answer area 70b on the answer sheet. The answer result image acquired by the answer result data acquisition unit 44 is subject to evaluation by the evaluation model.

[0058] The problem image data acquisition unit 45 acquires a problem image from the problem paper information based on position information indicating the drawing answer area 70a on the problem paper. The problem image acquired by the problem image data acquisition unit 45 is used to select an evaluation model.

[0059] The model answer data acquisition unit 46 acquires a model answer image from the model sheet information based on positional information indicating the drawing answer area 70c on the model sheet. The model answer image acquired by the model answer data acquisition unit 46 is used to select an evaluation model.

[0060] The evaluation model selection unit 47 selects and retrieves an evaluation model from among multiple evaluation models stored in the evaluation model DB 32 to be used to evaluate the target answer result data. Specifically, the evaluation model selection unit 47 selects and retrieves an evaluation model from the evaluation model DB 32 based on information about the drawing problem corresponding to the target answer result data. Here, the information about the drawing problem includes the problem statement data, the problem image data, and the model answer data. The evaluation model selection unit 47 includes a problem statement similarity calculation unit 55, a problem image similarity calculation unit 56, and a model answer similarity calculation unit 57.

[0061] The problem statement similarity calculation unit 55 calculates the similarity between the problem statement text acquired by the problem statement data acquisition unit 43 and the problem statement text of each evaluation model stored in the evaluation model DB 32.

[0062] Text similarity is expressed in a feature space such that the closer the problems are, the higher the similarity and the closer the value is to 1, and the further apart the problems are, the lower the similarity and the closer the value is to 0. Figure 13 illustrates the method for calculating text similarity. As shown in Figure 13, the problem statement similarity calculation unit 55 treats the problem statement text to be used for similarity calculation, for example, "Draw a circle with line segment AB as its diameter," as individual characters such as "line" and "segment," and uses machine learning methods capable of processing time-series information and text information, such as LSTM (Long Short Term Memory), to quantify and extract the features of the problem statement text. Specifically, the problem statement similarity calculation unit 55 takes "Draw a circle with line segment AB as its diameter" as input to the LSTM and trains it to output the same text. After training is complete, the problem statement similarity calculation unit 55 extracts and utilizes the features of the intermediate layer of the LSTM. Similarly, the problem statement similarity calculation unit 55 quantifies and extracts the feature quantities of the problem statement texts on which each evaluation model was constructed, based on all the problem statement texts stored in the evaluation model DB 32. The problem statement similarity calculation unit 55 then calculates the similarity as the difference between the feature quantities of the target problem statement text and the feature quantities of the problem statement texts on which each evaluation model was constructed.

[0063] The problem image similarity calculation unit 56 calculates the similarity between the problem image acquired by the problem image data acquisition unit 45 and the problem images of each evaluation model stored in the evaluation model DB 32.

[0064] For example, the problem image similarity calculation unit 56 may apply a similarity calculation method based on exact matches, matching the size of the problem image to be used for similarity calculation and the problem images on which each evaluation model is constructed, calculating the difference between pixels, and using the sum of the differences as the similarity. Alternatively, the problem image similarity calculation unit 56 may apply a similarity calculation method using a CNN. Figure 14 illustrates the similarity calculation method using a CNN. As shown in Figure 14, the problem image similarity calculation unit 56 inputs the problem image to be used for similarity calculation into a feature extraction CNN and extracts feature vectors. Specifically, the problem image similarity calculation unit 56 trains the feature extraction CNN to output the same image as the input image. After training is complete, the problem image similarity calculation unit 56 extracts the values ​​of the intermediate layer as feature vectors and uses them. Similarly, the problem image similarity calculation unit 56 inputs the problem images on which each evaluation model is constructed into a feature extraction CNN and extracts feature vectors. The problem image similarity calculation unit 56 then calculates the difference between the two extracted feature vectors and uses this as the similarity score.

[0065] The model answer similarity calculation unit 57 calculates the similarity between the model answer images acquired by the model answer data acquisition unit 46 and the model answer images of each evaluation model stored in the evaluation model DB 32. Note that the method for calculating the similarity of the model answer images is the same as the method for calculating the similarity of the problem images, so for convenience, an explanation is omitted.

[0066] The evaluation model selection unit 47 normalizes the similarity of the problem statement text, problem image, and model answer image, calculated by the problem statement similarity calculation unit 55, problem image similarity calculation unit 56, and model answer similarity calculation unit 57, respectively, to a range of 0 to 100. Specifically, the evaluation model selection unit 47 normalizes the calculated similarity values ​​by assigning values ​​from 0 to 100 to each of a certain range. In this case, the certain range and the numerical values ​​assigned as similarity can be arbitrarily set, for example, if the similarity before normalization is "0.0 to 10.0", the similarity after normalization will be "100", and if the similarity before normalization is "10.0 to 20.0", the similarity after normalization will be "99". Then, the evaluation model selection unit 47 sets thresholds for each of the normalized similarities of the problem statement text, problem image, and model answer image, and determines whether or not there is an evaluation model in the evaluation model DB 32 that satisfies all the threshold conditions.

[0067] Figure 15 illustrates the selection of an evaluation model based on similarity. For example, the normalized similarity thresholds for the problem statement text, problem image, and model answer image are set to 90%, 95%, and 95%, respectively. In this case, as shown in Figure 15, the evaluation model with model ID "XXX" has normalized similarity values ​​of the problem statement text, problem image, and model answer image all above the threshold. Therefore, the evaluation model selection unit 47 selects and obtains the evaluation model with model ID "XXX" from the evaluation model DB 32 as one that satisfies all the threshold conditions. On the other hand, as shown in Figure 15, the evaluation model with model ID "XXY" has normalized similarity values ​​of the problem statement text, problem image, and model answer image all below the threshold. Therefore, the evaluation model selection unit 47 determines that the evaluation model with model ID "XXY" is not an evaluation model that satisfies all the threshold conditions.

[0068] Furthermore, the evaluation model selection unit 47 will not select the answer result image as an evaluation model if even one of the normalized similarities of the problem statement text, problem image, and model answer image is below the threshold, not just when the normalized similarity of all of them is below the threshold.

[0069] The evaluation information acquisition unit 48 acquires evaluation information regarding the evaluation of the target answer result data using the evaluation model selected by the evaluation model selection unit 47. For example, the evaluation information acquisition unit 48 inputs the answer result image as the target answer result data to the selected evaluation model and acquires the evaluation information, which is output as "100%" for a perfect score, "0%" for a deduction, and "0%" for a score of zero. In this case, the evaluation information acquisition unit 48 evaluates the answer result image as having the highest probability among a perfect score, a deduction, and zero. Therefore, in this case, the evaluation information acquisition unit 48 evaluates the answer result image as having a perfect score. In addition to the probabilities of the answer result image receiving a perfect score, a deduction, and zero, the evaluation information may also include an evaluation result indicating whether the answer result image was evaluated as a perfect score, a deduction, or zero.

[0070] The deduction area information acquisition unit 49 acquires information about the areas that influenced the evaluation when the target answer result data is evaluated as having a deduction. Specifically, the deduction area information acquisition unit 49 uses an evaluation model to create and acquire a deduction area image, which visualizes the areas evaluated as having a deduction from the input answer result image (hereinafter, "areas evaluated as having a deduction" are also called "deduction areas"). Methods for identifying deduction areas in the answer result image include, for example, identifying the location of images that influenced the output result in image classification using CNNs such as Grad-CAM. Another method is to identify the location of images that influenced the judgment when the evaluation model determines that "the probability of a deduction is high". Figure 16 is an example of a deduction area image. As shown in Figure 16, by applying these methods, the deduction area information acquisition unit 49 can create and acquire a deduction area image.

[0071] The evaluation result screen creation unit 50 creates an evaluation result screen that includes the evaluation results of the target answer result data. Figure 17 is an example of an evaluation result screen. As shown in Figure 17, the evaluation result screen includes an answer area 80, a question text area 81, a learner area 82, an evaluation result area 83, a deduction area area 84, an instructor area 85, change buttons 86a and 86b, a score input area 87, and a confirmation button 88.

[0072] The answer area 80 is the area that displays the answer result image, which is the answer result data subject to evaluation. The problem statement area 81 is the area that displays the problem statement for the drawing problem corresponding to the answer result image. The learner area is the area that displays the name of the learner who entered the answer for the answer result image. The evaluation result area 83 is the area that displays the evaluation result for the answer result image. The deduction area area 84 is the area that displays the deduction area image if the evaluation result is a deduction. The deduction area area 84 does not display anything if the evaluation result is a perfect score or 0 points. The instructor area 85 is the area that displays the name of the instructor who requested or logged in to the evaluation result screen.

[0073] The change buttons 86a and 86b are buttons that change the answer result image displayed in the answer area 80. For example, if the instructor presses the change button 86a, the answer area 80 will display the answer result image of the student with the number preceding the currently displayed student. On the other hand, if the instructor presses the change button 86b, the answer area 80 will display the answer result image of the student with the number following the currently displayed student. The score input area 87 is an area where the instructor enters the score for the student's answer. The instructor can check the evaluation results and the deduction area image and enter the score for the student's answer. The confirm button 88 is a button that confirms the score entered by the instructor.

[0074] In the above configuration, the answer result data acquisition unit 44, evaluation model selection unit 47, evaluation information acquisition unit 48, screen output unit 51, and evaluation model DB 32 of the server 20 are examples of the answer result data acquisition unit, evaluation model selection unit, evaluation information acquisition unit, output unit, and storage unit of the present invention, respectively. Furthermore, the deduction area information acquisition unit 49 of the server 20 is an example of the evaluation information acquisition unit of the present invention.

[0075] [Evaluation process] Next, we will explain the evaluation process, which involves obtaining and outputting evaluation information for the target answer result data using an evaluation model. Figure 18 is a flowchart of the evaluation process. This process is achieved by the server 20 executing a pre-prepared program.

[0076] The instructor transmits input information using the instructor terminal 10. When the server 20 receives the input information (step S11), it first detects the areas contained in the question paper, answer paper, and model paper, respectively, based on the question paper information, answer paper information, and model paper information contained in the input information (step S12). Then, the server 20 obtains the question text of the drawing question area 69 and the answer result image of the drawing answer area 70b from the answer paper information (step S13). The server 20 also obtains the question image of the drawing answer area 70a from the question paper information (step S14). The server 20 also obtains the model answer image of the drawing answer area 70c from the model paper information (step S15).

[0077] Server 20 selects and retrieves an evaluation model from the evaluation model DB32 based on information about the drawing problem corresponding to the target answer result image (step S16). Specifically, Server 20 calculates the similarity between the problem statement text, problem image, and model answer image of the drawing problem and the problem statement text, problem image, and model answer image for which each evaluation model has been constructed. Then, Server 20 determines whether there is an evaluation model in the evaluation model DB32 that satisfies the condition that the similarity of the problem statement text, problem image, and model answer image is all above a threshold (step S17). If no evaluation model satisfies the condition (step S17; No), the evaluation of the answer result image cannot be performed, and Server 20 terminates the evaluation process.

[0078] On the other hand, if an evaluation model that meets the conditions exists (Step S17; Yes), the server 20 selects that evaluation model to be used to evaluate the answer result image and retrieves it from the evaluation model DB32. The server 20 then inputs the target answer result image into the retrieved evaluation model to output an evaluation of the answer result image and obtains evaluation result information (Step S18). At this time, the server 20 determines whether the evaluation result of the answer result image is a deduction or not (Step S19). If the evaluation result is a deduction (Step S19; Yes), the server 20 obtains deduction area information (Step S20). On the other hand, if the evaluation result is a perfect score or 0 points and not a deduction (Step S19; No), the server 20 proceeds to the process in Step S21. The server 20 then creates an evaluation result screen including the evaluation result and deduction area image based on the evaluation result information and deduction area information, and sends screen information related to the evaluation result screen to the instructor terminal 10 (Step S21). The instructor's terminal 10 displays the evaluation results screen, allowing the instructor to easily check the learner's answer results images and information on areas where points were deducted, which can be used to help grade the drawing problems. This completes the evaluation process.

[0079] In this embodiment, the question paper, answer sheet, and model sheet are used as digital teaching materials, but the present invention is not limited to this, and paper teaching materials may also be used. In this case, the question paper, answer sheet, and model sheet are transmitted to the server 20 as scanned data. Therefore, the server 20 obtains the text of the drawing problem by performing character recognition on the image of the drawing problem text area 69 using OCR (Optical Character Reader) based on the scanned data of the answer sheet.

[0080] Furthermore, in this embodiment, the problem image and the model answer image are obtained from the problem paper information and the model answer sheet information, respectively, by region detection. However, the present invention is not limited to this, and the problem image and the model answer image may be pre-associated with the answer sheet. Alternatively, a problem ID to identify a problem, the problem text, the problem image, and the model answer image may be stored in a problem database in advance, and the problem image and the model answer image may be extracted from the problem database based on the problem text of a drawing problem. In this case, the input information does not need to include the problem paper information and the model answer sheet information. Also, region detection of the problem paper and the model answer sheet does not need to be performed.

[0081] Furthermore, in this embodiment, the answer result image is obtained by region detection from the answer sheet of a test consisting of multiple questions, but the present invention is not limited to this, and the answer result image may be obtained directly from the instructor's terminal 10. In this case, the question ID is associated with the answer result image, and the server 20 can perform evaluation processing by referring to a question database that associates the question ID, the question text, the question image, and the model answer image.

[0082] Furthermore, in this embodiment, the evaluation results are given as "perfect score," "deduction," and "0 points," but they could also be given as, for example, "○," "△," and "×." Additionally, "deduction" could be further subdivided into "20% deduction," "50% deduction," "80% deduction," etc. In this case, instead of the instructor manually entering the scores on the evaluation results screen, if the point distribution for the drawing problems is known in advance, the score for the answer result image could be automatically calculated based on the percentage of deductions.

[0083] The learning support system 100 of this embodiment presents the evaluation results of the answers to drawing problems to instructors and others in an easy-to-understand manner. Furthermore, if the evaluation result is a deduction, it presents the instructors and others with an image visualizing the areas where points were deducted. In this way, the learning support system 100 can assist instructors in evaluation and scoring. Thus, it can reduce the burden on instructors when evaluating and scoring answers to drawing problems, which have many variations.

[0084] <Variation> In the above embodiment, the instructor uses an instructor terminal 10, but the present invention is not limited thereto, and the instructor may use an instructor terminal 90 that has the functions of a server 20. The instructor terminal 90, like the server 20, is, for example, a PC or a general-purpose tablet.

[0085] Figure 19 shows an example configuration of the learning support system 200 in this case. The instructor terminal 90 is connected to the answer database 91 and the evaluation model database 92, and is configured to communicate with the learner terminal 9 via the network 5. The instructor terminal 90 can perform the evaluation processing that was performed by the server 20, and can acquire and output evaluation information of the target answer result data using the evaluation model. In this case, the instructor terminal 90 is an example of the information processing device of the present invention. [Explanation of symbols]

[0086] 5 Network 9. Learner terminals 10, 90 Instructor terminals 20 servers 31, 91 Answer DB 32, 92 Evaluation Model DB 41 Input Information Acquisition Unit 42 Area detection unit 43 Problem statement data acquisition unit 44. Answer Result Data Acquisition Unit 45 Problem Image Data Acquisition Unit 46 Model Answer Data Acquisition Unit 47 Evaluation Model Selection Section 48 Evaluation Information Acquisition Department 49. Unit for acquiring information on areas where points are deducted. 50 Evaluation Result Screen Creation Section 51 Screen output section 55 Question sentence similarity calculation part 56 Problem Image Similarity Calculation Unit 57 Model answer similarity calculation unit 100, 200 Learning Support Systems

Claims

1. A solution result data acquisition unit that obtains the solution to the drawing problem as solution result data, An evaluation information acquisition unit acquires evaluation information regarding the evaluation of the target answer result data using an evaluation model that has been previously trained on the relationship between the answer result data and evaluation for the aforementioned drawing problem, An evaluation information output unit that outputs the aforementioned evaluation information, A memory unit that stores multiple evaluation models corresponding to multiple drawing problems, and information about the drawing problems on which each evaluation model was constructed, in association with each other. The system includes an evaluation model selection unit that selects the evaluation model from the storage unit based on information related to the drawing problem corresponding to the target answer result data. The aforementioned evaluation information acquisition unit acquires evaluation information using the selected evaluation model, The evaluation model selection unit calculates the similarity between information about the drawing problem corresponding to the target answer result data and information about the drawing problem on which the evaluation model was constructed, and selects an evaluation model whose similarity is equal to or greater than a threshold.

2. The information processing apparatus according to claim 1, characterized in that the information relating to the drawing problem includes the text of the problem statement, the image of the problem, and the image of the model answer.

3. The information processing device according to claim 1, wherein the target answer result data is evaluated as either a perfect score, a deduction, or zero points.

4. The information processing apparatus according to claim 3, wherein, if the target answer result data is evaluated as a point deduction, the evaluation information acquisition unit acquires information on the area that affected the evaluation as point deduction area information, along with the evaluation information.

5. A program executed by an information processing device equipped with a computer, A solution result data acquisition unit that obtains the solution to a drawing problem as solution result data. An evaluation information acquisition unit acquires evaluation information regarding the evaluation of the target answer result data using an evaluation model that has been previously trained on the relationship between the answer result data and evaluation for the aforementioned drawing problem. Evaluation information output unit that outputs the aforementioned evaluation information, A memory unit that stores multiple evaluation models corresponding to multiple drawing problems, and information about the drawing problems on which each evaluation model was constructed, in association with each other. The computer functions as an evaluation model selection unit, which selects the evaluation model from the storage unit based on information related to the drawing problem corresponding to the target answer result data. The aforementioned evaluation information acquisition unit acquires evaluation information using the selected evaluation model, The evaluation model selection unit is a program that calculates the similarity between information about the drawing problem corresponding to the target answer result data and information about the drawing problem on which the evaluation model was constructed, and selects an evaluation model whose similarity is equal to or greater than a threshold.

Images