Information processing device, information processing method, and program

Abstract

This system provides a mechanism for appropriately scoring problems that have multiple possible solutions. [Solution] For each problem, information on example solutions with multiple solutions and information on the answers of the respondents are obtained. Based on the obtained information on example solutions and the respondents' answers, the first solution among the multiple solutions and the content of the respondents' answers are displayed side by side.

Description

Technical Field

[0001] This embodiment relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In recent years, as part of the digital transformation (DX) of education, the digitization of tests and scoring has been progressing.

[0003] For example, Patent Document 1 discloses a mechanism that allows a test taker to take a test on an information terminal for a problem with multiple solutions, collect the solutions of the test takers who answered correctly, and provide feedback to the information terminal.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In Patent Document 1, first, the correctness of the final answer of the answer data acquired from the test taker's terminal is judged. Next, the calculation process of the answer data whose final answer is judged to be correct is collected and stored as one of the solutions. Finally, an example is shown in which the collected and stored solutions can be referred to on other user terminals. Therefore, in this method, there is a risk that a solution in which only the final answer is accidentally correctly derived although the calculation process is incorrect will be registered and shared. Also, this method is a mechanism that pre-scores the answers to problems with multiple solutions and shares the intermediate expressions of the answer data with the correct final answer as answer examples. Therefore, with this method, it is not possible to appropriately score problems with multiple solutions.

[0006] [[ID=4X]] Note: There seems to be a duplicate tag in the original text which might be a formatting error. I've translated as is but it should be corrected in the original source if possible. Also, I've added some "X" in the tag numbering in the translation for the duplicate tag situation.The purpose of this embodiment is to provide a mechanism for appropriately scoring problems that have multiple possible solutions. [Means for solving the problem]

[0007] This embodiment is characterized by comprising: a first acquisition means for acquiring information on example solutions having multiple solutions for a given problem, and information on the answer of a respondent for the given problem; and a generation means for generating first image data that displays the first solution among the multiple solutions and the content of the respondent's answer side by side, based on the information on example solutions and the answer of the respondent acquired by the first acquisition means. [Effects of the Invention]

[0008] Thus, according to this embodiment, it is possible to provide a mechanism for appropriately scoring problems that have multiple solutions. [Brief explanation of the drawing]

[0009] [Figure 1] This is a configuration diagram showing an example of a system configuration according to the embodiment. [Figure 2] This is a configuration diagram showing an example of the hardware configuration according to the embodiment. [Figure 3] This is a diagram illustrating an example of a screen that accepts instructions to scan a blank answer sheet. [Figure 4] This is a diagram illustrating an example of a display screen that accepts the acceptance or rejection of acquired image data. [Figure 5] This is a diagram showing an example of a display screen used to define the criteria for extracting answers. [Figure 6] This is a diagram illustrating an example of a display screen used to define interim response criteria, etc. [Figure 7] This is a diagram showing an example of a screen that accepts instructions to scan answer examples. [Figure 8] This is a diagram illustrating an example of a screen for receiving settings related to scoring criteria. [Figure 9]It is a configuration diagram showing an example regarding item setting, change, and deletion of scoring criteria. [Figure 10] It is a configuration diagram showing an example of a screen for receiving an instruction to scan an answer sheet on which answers have been written. [Figure 11] It is a configuration diagram showing an example of a screen for receiving a correction regarding information to be evaluated. [Figure 12] It is a configuration diagram showing an example of a screen for receiving scoring and correction of an evaluation target. [Figure 13] It is a configuration diagram showing an example of automatic adjustment of the position and scale of display of an evaluation target. [Figure 14] It is a configuration diagram showing an example of manual adjustment of the position and scale of display of an evaluation target. [Figure 15] It is a flowchart showing the information processing flow of an embodiment. [Figure 16] It is a flowchart showing the information processing flow of an embodiment. [Figure 17] It is a flowchart showing the information processing flow of an embodiment. [Figure 18] It is a flowchart showing the information processing flow of an embodiment. [Figure 19] It is a flowchart showing the information processing flow of an embodiment. [Figure 20] It is a configuration diagram showing an example of a data table of an embodiment. [Figure 21] It is a configuration diagram showing an example of a data table of an embodiment. [Figure 22] It is a configuration diagram showing an example of another embodiment for appropriately extracting information to be evaluated without using the answer extraction criterion 501. [Figure 23] It is a configuration diagram showing an example of a case where clusters are extracted starting from the position reference origin of the answer extraction criterion 501. [[ID=4l]] [Figure 24] It is a configuration diagram showing an example of a screen for receiving settings related to scoring criteria. [Figure 25] It is a configuration diagram showing an example of a screen for displaying one solution and an answer. [Modes for carrying out the invention]

[0010] The embodiments will be described in detail below with reference to the drawings.

[0011] Figure 1 is a configuration diagram showing an example of a system configuration according to the embodiment.

[0012] The image data providing device 110 is a device for providing image data to the information processing device 120, which will be described later.

[0013] In this embodiment, as an example, the scanner 111 acquires three image data files: a blank answer sheet, an answer sheet including example answers, and an answer sheet with answers written on it. This data is then used to generate image data, which is provided to the information processing device 120 via the communication network 130.

[0014] However, this is merely one example, and you should appropriately select and use image data that contains the information to be evaluated.

[0015] For example, the source images for the image data may include essays written on composition paper, reports written on report paper, or illustrations drawn in the work entry section. In addition, the image data providing device 110 includes various other devices for acquiring, managing, and supplying content to be evaluated.

[0016] In addition to the above, it is possible to receive and implement an invention by including copyrighted works such as text, drawings, videos, and program code that may be subject to evaluation within the scope of image data.

[0017] Let's return to the image data providing device 110.

[0018] Another example of an image data providing device 110 that replaces the scanner 111 is a configuration in which, for example, a mobile terminal 112 is used to provide still images and video image data acquired by the camera function to the information processing device 120.

[0019] In addition, a configuration can also be adopted in which still image and video image data managed within the image management server 113 is provided to the information processing device 120 via the communication network 130.

[0020] Furthermore, the information processing device 120 itself may also function as the image data providing device 110, and may be equipped with a camera function that can acquire still images and video image data, such as a mobile terminal 112.

[0021] Network 130 is a network that enables communication between hardware components according to the embodiment (for example, the image data providing device 110 and the information processing device 120).

[0022] The communication method of the communication network 130 may be wireless or wired. It may also be a WAN (Wide Area Network) or a LAN (Local Area Network).

[0023] Figure 2 is a configuration diagram showing an example of the hardware configuration according to the embodiment.

[0024] An example of hardware according to this embodiment includes at least an image data providing device 110 and an information processing device 120.

[0025] 201 is the CPU, which controls each device connected to the system bus 204 by loading programs stored in the program ROM 202 or external memory 211 into RAM 203 and executing them.

[0026] Furthermore, the font ROM of ROM 202 stores font data, and the data ROM of ROM 202 stores various other data. 203 is RAM, which functions as the main memory and work area of ​​the CPU 201. 202 is a keyboard controller (input controller) that controls input from 209 and pointing devices (not shown), such as a touchpad.

[0027] The input controller 205 controls the input device 209. The input device 209 also includes a touch panel (touchscreen), in which case the input device 209 also functions as the display unit 210 (a display device for showing a display screen for displaying video). The input device 209 is not limited to the above and can utilize any known device. For example, it could be an audio input device or a device that acquires user gestures using a camera or tracker and converts them into input instructions.

[0028] 206 is the video card (VC) which controls the display of the display unit 210. The display unit 210 can utilize various existing output devices. For example, screen displays and retinal projection displays are conceivable, but any output device that allows the user to acquire visual information will suffice.

[0029] 207 is the memory controller, which controls access to external memory 211 (hard disk, etc.). 208 is the network interface card (NIC), which controls communication on the communication network 130.

[0030] The image data acquisition unit controller 220 is a controller that controls the image data acquisition unit 221. The image data acquisition unit 221 is a unit that combines the mechanisms necessary to acquire image data, and can take various forms to suit the characteristics of the device. For example, in the scanner 111, it is a scanning unit that scans images, and in the mobile terminal 112 and the information processing device 120, it is a camera unit that takes still images and videos.

[0031] Furthermore, in the case of an image management server 113 or an information processing device 120, an external memory 211 for acquiring and storing image data and a network interface card 208 for controlling the connection to a communication network 130 for acquiring image data from an external source are envisioned.

[0032] Figure 3 is a diagram showing an example of a screen that accepts instructions to scan a blank (unanswered) answer sheet.

[0033] In Figure 3, the display unit 210 of the information processing device 120 displays a screen with the configuration shown in the figure, and is configured to receive instructions from the user to acquire image information 402.

[0034] Furthermore, the display screens in Figures 4 to 14 are also displayed on the display unit 210 of the information processing device 120, similar to Figure 3, and are configured to receive instructions from the user.

[0035] Furthermore, the following descriptions of embodiments will proceed assuming a system configuration in which a PC (information processing device 120) is connected to a scanner 111 (image data providing device 110) by wired or wireless connection, as an example.

[0036] However, the information processing device 120 is also capable of connecting to various other terminals and acquiring image data. For example, various configurations combining existing mechanisms are envisioned, such as wirelessly connecting a PC (information processing device 120) to a tablet terminal (mobile terminal 112), or connecting an image management server 113 and a PC (information processing device 120) via a network.

[0037] The image data acquisition buttons 301a to 301c are buttons that receive instructions from the user to acquire image data, including the answer fields, from the image data providing device 110, which is connected via communication, to the information processing device 120.

[0038] For example, when the image data acquisition button 301a is pressed by the user, it requests the scanner 111, which is connected to the system for communication, to scan the original document (a blank answer sheet) and transmit the resulting image data, thereby acquiring the image data.

[0039] Furthermore, the image data acquisition button 301b receives a button press from the user and requests the mobile terminal 112, which is connected to the device for communication, to launch the camera application, take a picture of the document (blank answer sheet), and send the resulting image data, thereby acquiring the image data.

[0040] Furthermore, the image data acquisition button 301c receives a button press from the user and requests the image management server 113, which is connected to it via communication, to send image data of the blank answer sheet it manages, thereby acquiring the image data.

[0041] Alternatively, the information processing device 120 itself may be equipped with an image data acquisition unit 221, such as a camera, and images may be acquired by activating this unit (not shown).

[0042] In this embodiment, the information processing device 120 acquires image data obtained by scanning a blank answer sheet (step S1608), assigns the file name "hakushi_0X.png" (where X is the page number), and manages it in the data table shown in Figure 20(a).

[0043] Figure 4 is a diagram showing an example of a display screen that accepts the acceptance or rejection of acquired image data.

[0044] Figure 4 shows the image data of an answer sheet, which consists of 6 pages in total, each containing problem number 401, a student ID number entry field 402, the problem statement 403, a space for intermediate calculations 404, and an answer field 405, displayed on the answer sheet display screen 406.

[0045] In Figure 4, problem number 401 is assumed to consist of a main question enclosed in a square (in this case, 5), and a sub-question (Figure 21) indicated as a (number) as a subordinate part of that main question.

[0046] Furthermore, the example shown in Figure 4 involves reading an answer sheet for a 6-page exam, and page number 407 indicates that page 5 is being displayed. The structure of the questions is as follows: Page 1: Question 1 consisting of two sub-questions; Page 2: Question 2 consisting of two sub-questions; Pages 3-6: Questions 3, 4, 5, and 6 are on each page.

[0047] These problem numbers (major questions, minor questions) are extracted using OCR or image recognition technology in step S1609-1 of Figure 16, and a data table like the ones shown in Figures 20(b) and (c) is generated and stored in the columns for major question 2004 and minor question 2005 for management.

[0048] Returning to Figure 4, the answer sheet display screen 406 is a screen that displays image data obtained by scanning a blank answer sheet, and displays image data corresponding to the page displayed in the original page display 407.

[0049] Therefore, for example, "Page 5 / 6" in Figure 4 indicates that the fifth page of the six-page image data is being displayed on the answer sheet display screen 406.

[0050] The display advance button 408 is a button that receives instructions from the user to switch between the pages of the image data displayed on the manuscript page display 407 and the answer sheet display screen 406.

[0051] In Figure 4, the back button 409 is a button that receives a command from the user to transition to the display screen shown in Figure 3, which accepts a command to scan a blank answer sheet.

[0052] In Figure 4, the displayed "Proceed" button 410 is a button that receives instructions from the user to transition to the display screen shown in Figure 5, which defines the answer extraction criteria.

[0053] Figure 5 is a diagram showing an example of a display screen used to define the criteria for extracting answers.

[0054] The answer extraction criterion 501 is a criterion set in a part of the image area of ​​the blank answer sheet scanned in Figure 4, in order to extract the subject of evaluation (information written in the answer section in this embodiment) from the "answer sheet with answers already written" as shown in Figure 11.

[0055] In this embodiment, the answer extraction criterion 501 identifies the answer field 405 using OCROPICAL Character Recognition (or Reader) or image recognition technology, and defines this area as the default answer extraction criterion 501.

[0056] Here, the subject of evaluation is the handwritten number 30, enclosed by cluster identification display 1101c in Figure 11, with its lower right corner extending beyond the answer field 405.

[0057] The handwritten characters "30" that are subject to evaluation are meaningful clusters or groups of handwritten characters, known as "clusters."

[0058] The clusters can then be extracted from images of answer sheets with answers written on them, using algorithms such as OCR technology and image recognition technology that can identify handwritten characters as clusters, similar to how answer field 405 was identified. Alternatively, clusters can be extracted by using an algorithm that groups the differences obtained from images of answer sheets with answers written on them and images of blank (unanswered) answer sheets into chunks of characters.

[0059] Therefore, unlike the prior art which extracts an image including the answer fields, this embodiment does not extract the answer fields, and only the handwritten answers can be extracted.

[0060] Furthermore, answer selection criterion 501 is a standard set in a part of the answer sheet with the aim of selecting the clusters to be evaluated from among the clusters described above on the answer sheet.

[0061] In this embodiment, the shaded area represents the answer extraction criterion 501, and the system is designed to extract clusters of handwritten characters that are included in at least a portion of this area for evaluation.

[0062] By implementing this method, even in cases where conventional techniques would crop answers that extend beyond the answer field or are too small relative to the answer field, causing problems with scoring, clusters of handwritten characters that are subject to evaluation can be appropriately extracted.

[0063] In this embodiment, the answer extraction criterion 501 is shown as a rectangular area corresponding to the shape of the answer field, but it does not have to be a rectangle; for example, it could be a "point," a circle, or a polygon (not shown).

[0064] For example, if a point (a certain coordinate) in an area of ​​the answer sheet is used as the answer extraction criterion 501, then clusters that are at least partially contained within a 100-pixel wide and 30-pixel high area surrounding that point may be extracted as subjects for evaluation.

[0065] In any case, even if the answer extends beyond the answer box or is too small for the answer box, it is acceptable as long as the target for evaluation can be appropriately selected based on answer selection criterion 501.

[0066] Here, we will add some further details regarding the extraction of evaluation targets using Figure 23.

[0067] In this embodiment, clusters of handwritten characters to be evaluated are extracted using the following procedure.

[0068] First, the answer fields are identified from the image data of the blank answer sheet, and based on the identified answer fields, the answer extraction criterion 501 is set.

[0069] Next, handwritten characters are identified in clusters from the completed answer sheets.

[0070] Then, from the clusters of handwritten characters identified on a cluster basis, the clusters of handwritten characters to be evaluated (handwritten characters enclosed by cluster identification indicator 1101 in Figure 23) are extracted, starting from the answer extraction criterion 501.

[0071] Figure 23 is a diagram illustrating an example of a case where clusters are extracted starting from the positional reference of answer extraction criterion 501.

[0072] In Figure 23, the answer extraction criteria 501, a rectangle measuring 20x10 pixels, is defined using image data from the answer sheet, with the X-axis pointing to the right and the Y-axis pointing downwards.

[0073] Furthermore, the vertices a, b, c, and d of the answer extraction criterion 501 are located at a(0,0), b(20,0), c(20,10), and d(0,10), respectively, and the center point o is o(10,5).

[0074] In Figure 23, (a) through (e) show clusters enclosed by a 20x10 rectangle (vertices a'b'c'd') cluster identification display 1101, while in (f) show clusters enclosed by a 10x6 rectangle cluster identification display 1101, each in a different location.

[0075] The center point o' of the cluster in Figure 23(a) is o'(20,10). The center point o' of the cluster in Figure 23(b) is o'(10,15). The center point o' of the cluster in Figure 23(c) is o'(10,17). The center point o' of the cluster in Figure 23(d) is o'(20,17). The center point o' of the cluster in Figure 23(e) is o'(30,15). The center point o' of the cluster in Figure 23(f) is o'(10,14).

[0076] Based on the above, we will explain which of the handwritten character clusters (a) through (f) in Figure 23 can be extracted according to the method for extracting the handwritten character clusters to be evaluated.

[0077] The extraction method will be explained in the case where "at least a portion of the cluster (cluster identification display 1101) relates to the answer extraction criterion 501 (overlapping, overlapping, matching)."

[0078] Please note that this extraction method is merely one example for illustrative purposes.

[0079] In another embodiment, the extraction method may be set to one method by default. Alternatively, multiple default extraction methods may be provided, and the user may select one of them. Alternatively, the user may be allowed to input a specific extraction method.

[0080] Furthermore, one possible way to accept user-submitted rules, either selectively or specifically, is to provide a mechanism (such as a button, toggle, dropdown, slider, picker, or text field) in the UI (as shown in Figures 8, 12, and 13) for selecting or inputting the extraction method. Alternatively, this could be done on a separate screen (not shown).

[0081] In this example, we will explain a case where the answer extraction criteria 501 and the cluster identification display 1101 each have grid point data, and the extraction of clusters is determined by matching the grid points of both.

[0082] For example, let's consider "there is a match between grid points of 2x2 pixels or more" as a condition for the cluster extraction method.

[0083] In this case, the positional relationship between the answer extraction criterion 501 that meets the conditions and the cluster identification display 1101 is only valid in Figure 23(a).

[0084] This is because, as shown in Figure 23(a), the answer extraction criterion 501 and the cluster identification display 1101 coincide in the region (10,5)(20,5)(20,10)(10,10) with grid points of 5x10 pixels vertically. In other words, the condition "there is a coincidence of grid points of 2x2 pixels or more vertically and horizontally" is satisfied.

[0085] On the other hand, in Figure 23(b), grid points of 1x20 pixels vertically and horizontally meet at (0,10) through (20,10).

[0086] Furthermore, in Figure 23(e), the grid points of 1x1 pixels meet at (20,10).

[0087] In other words, Figures 23(b) and 23(e) do not satisfy the condition that "there is a match of grid points of 2x2 or more pixels vertically and horizontally," and therefore, clusters are not extracted in these cases.

[0088] Therefore, if the user were to input a change in the cluster extraction method to "extract clusters when there is a match of grid points of 1x1 pixels or more," then Figures 23(b) and 23(e) would satisfy the condition. In other words, these would be cases where clusters are extracted.

[0089] Furthermore, in Figures 23(c), 23(d), and 23(f), there are no points where the grid points of the answer extraction criteria 501 and the cluster identification display 1101 coincide, therefore, clusters are not extracted as subjects for evaluation.

[0090] Next, the extraction method can also be defined as "the distance between the center points of each cluster and the answer extraction criterion 501 should be within 11."

[0091] In this case, the distance between the centers of the clusters and the answer extraction criteria 501 is approximately 11.2 in Figure 23(a), 12 in Figure 23(c), approximately 15.6 in Figure 23(d), and approximately 15.6 in Figure 23(e). In these cases, the clusters are not extracted as subjects for evaluation.

[0092] On the other hand, in Figure 23(b), the distance between the centers of the clusters and the answer extraction criteria 501 is 10, and in Figure 23(f), it is 11. In these cases, the clusters are extracted as subjects for evaluation.

[0093] Alternatively, the extraction method can be defined as "the shortest distance between each side or point of the cluster and the answer extraction criterion 501 is 1 or less."

[0094] In this case, since the cluster and the answer extraction criterion 501 overlap in a plane in Figure 23(a), the cluster is extracted as a target for evaluation.

[0095] Furthermore, in Figure 23(b), since the cluster and the answer extraction criterion 501 are touching by an edge, the cluster is extracted as a target for evaluation.

[0096] Furthermore, in Figure 23(e), the clusters and the answer extraction criteria 501 are touching at a single point, so the clusters are extracted as subjects for evaluation.

[0097] Furthermore, in Figure 23(f), the shortest distance between the cluster and the answer extraction criterion 501 is 1, so the cluster is extracted as a target for evaluation.

[0098] On the other hand, in Figures 23(c) and 23(d), the shortest distance between the cluster and the answer extraction criterion 501 is 2, so in these cases, the cluster is not extracted as a target for evaluation.

[0099] As described above, several methods are conceivable for selecting the items to be evaluated based on answer selection criteria 501. Ideally, the system should allow graders to arbitrarily choose which method to adopt.

[0100] Alternatively, in another embodiment, instead of setting the answer extraction criteria 501, it is conceivable to accept an arbitrary range specification from the user for each image data of an answer sheet with answers completed, and extract the clusters to be evaluated. Or, it is conceivable to accept an arbitrary cluster specification from the user and extract the clusters to be evaluated.

[0101] To explain using Figure 23 as an example, the system displays the image data with the answers filled in on the display screen, and allows the user to arbitrarily specify the range corresponding to the dotted line in the cluster identification display 1101.

[0102] By setting it up this way, users can arbitrarily select and extract all of the clusters in Figures 23(a) through (f) as clusters to be evaluated.

[0103] Furthermore, this embodiment is not limited to extracting handwritten answers; similarly, it can accept range and cluster specifications from the user for intermediate calculations and scene diagrams, as described later, and extract clusters to be evaluated.

[0104] Next, we will explain the adjustments to the answer selection criteria 501.

[0105] In the embodiment shown in Figure 5, the answer extraction criteria 501 includes an object for the user to specify an area, with handles for adjusting its size and position. It accepts user input via drag and drop operations, allowing the user to adjust it as they see fit.

[0106] For example, Figure 5 shows what happens when a user tries to adjust an area by moving the mouse cursor over the handles indicated by "○" located at the center of the edge and four vertices of the answer extraction criterion 501 and dragging them.

[0107] The above is an example of a system that accepts user operations such as drag and drop, allowing the user to arbitrarily determine the criteria for answer extraction criterion 501. Furthermore, it is desirable to have a function that can automatically set the initial position appropriately, as this would significantly reduce the workload on the operator.

[0108] Next, we will explain the settings for the default answer extraction criterion 501.

[0109] For example, in the case shown in Figure 5, the system detects textual information that can be acquired by OCR, such as "answer" and "solution," as well as information that is often used as an answer field in exam questions, such as a "square frame" and "underline," and determines the initial position based on the location of the detected information.

[0110] Another approach is to use a mechanism that allows for more flexible setting of the initial position of the answer extraction criterion 501, for example, by using a pre-trained model.

[0111] In Figure 5, the back button 409 is a button that receives a command from the user to transition to the display screen in Figure 4, which accepts confirmation of whether the acquired image data is acceptable or not.

[0112] Furthermore, the "Proceed" button 410 in Figure 5 is a button that accepts instructions from the user to transition to the display screen shown in Figure 6, where answer criteria and other information are defined along the way.

[0113] Figure 6 is a diagram showing an example of a display screen for defining interim response criteria, etc.

[0114] Intermediate calculation extraction criterion 601 is the area that serves as the criterion for extracting information to be evaluated as an intermediate calculation.

[0115] As shown in Figure 6, similar to the answer extraction criterion 501, the intermediate calculation extraction criterion 601 also has handles for adjusting the size and position of the object used by the user to specify the area, allowing for arbitrary adjustment through drag and drop operations.

[0116] Furthermore, if the answer extraction criterion 501 falls within the area of ​​the intermediate calculation extraction criterion 601, that area shall be treated as the answer extraction criterion 501, not the intermediate calculation extraction criterion 601.

[0117] Furthermore, as with the answer extraction criterion 501, the intermediate calculation extraction criterion 601 allows for the use of various methods for extracting morphological information.

[0118] Furthermore, it is desirable that the intermediate calculation extraction criterion 601, like the answer extraction criterion 501, have a function that allows its initial position to be automatically set appropriately.

[0119] Student ID number extraction criterion 602 is the area that serves as the criterion for extracting personal identification numbers such as student ID numbers, which are used by examinees at the schools or cram schools to which they belong, or examination numbers.

[0120] Furthermore, similar to the answer extraction criteria 501, the student ID number extraction criterion 602 should allow for the use of various existing methods for morphological extraction, depending on the circumstances in which the invention is being implemented, and it is desirable that the initial position be automatically set appropriately.

[0121] In the display screen shown in Figure 6, the back button 409 is a button that receives a command from the user to transition to the display screen shown in Figure 5, which defines the answer extraction criteria.

[0122] Furthermore, the "Proceed" button 410 on the display screen shown in Figure 6 is a button that receives a command from the user to transition to the display screen shown in Figure 7, which accepts a scan command for the answer example.

[0123] This button determines the intermediate calculation extraction criterion 601 and accepts instructions from the user to proceed to the next screen (Figure 7).

[0124] Figure 7 is a diagram showing an example of a display screen for receiving instructions to scan answer examples.

[0125] The display screen shown in Figure 7 is similar to the display screen shown in Figure 3 in that it receives instructions from the user to acquire image data using the image data acquisition button 301. However, it differs from Figure 3 in that it accepts instructions to acquire image data including example answers.

[0126] Furthermore, the back button 409 in the display screen shown in Figure 7 is a button that receives instructions from the user to transition to the display screen shown in Figure 6, which defines the criteria for intermediate responses, etc.

[0127] Figure 8 is a diagram showing an example of a screen for receiving settings related to scoring criteria.

[0128] Figure 8 shows a screen displaying image data including the example solution obtained in Figure 7, along with supplementary displays showing the answer extraction criterion 501, the intermediate calculation extraction criterion 601, and the student ID number extraction criterion 602.

[0129] However, it is also possible to omit displaying this auxiliary information, or to provide a switch to turn the auxiliary information on or off.

[0130] Grading criterion 801 is an object that defines "clusters" for assigning partial credit based on the example answer.

[0131] The word "cluster" means "group," "gathering," or "collection." However, in this example, the information enclosed by scoring criterion 801 is treated as a single group of information (=cluster).

[0132] For example, in the display screen shown in Figure 8, scoring criterion 801a encloses the "scene diagram" which is within the area of ​​intermediate calculation extraction criterion 601. This is defined as a single group of information (=cluster) to be evaluated, and if there is a description that conforms to the content of this cluster, it will be given a score of 2 out of 10.

[0133] In this embodiment, "scene diagram" includes all drawings used in the calculation process (line segment diagrams, graphs, etc.), and in Figure 12 described later, the areas labeled "scene diagram" for convenience will display drawings used in the calculation process.

[0134] Furthermore, in the display screen shown in Figure 8, scoring criterion 801b encloses the "intermediate steps" and "the calculation result (calculation)" within the area of ​​intermediate calculation extraction criterion 601, treating this as a single group of information (=cluster) to be evaluated. If the description conforms to this content, it is defined that an evaluation of 3 points for "intermediate steps" and 2 points for "the calculation result (calculation)" out of a maximum of 10 points will be given.

[0135] Furthermore, in the display screen shown in Figure 8, scoring criterion 801c surrounds the "answer numbers" and "answer units" that are within or near the area of ​​answer extraction criterion 501.

[0136] The information within the enclosed area is treated as a single group of information (=cluster) to be evaluated. If the description matches the content of this cluster, it is awarded 2 points out of 10 for the "number in the answer," and an additional point if the "unit in the answer" is also correct.

[0137] The scoring board 802 displayed on the screen is a board that defines the details of the clusters defined in scoring criteria 801a~c. The numbers "Circle 1"* to "Circle 5"* are the cluster numbers.

[0138] (Note: The circled numbers shown in Figure 8 cannot be used in the specification; therefore, they are represented as "Circle 1," "Circle 2," ... "Circle 6" instead. The same applies to the following explanation.) The board allows you to define at least the number, item name, and point allocation, and it is also possible to assign multiple evaluation items to a single cluster. Details on setting, changing, and deleting items will be described later in Figure 9.

[0139] In Figure 8, the back button 409 is a button that receives a command from the user to transition to the display screen shown in Figure 7, which accepts a command to scan the example answer.

[0140] Furthermore, the "Proceed" button 410 in Figure 8 is a button that receives a command from the user to transition to the display screen shown in Figure 10, which accepts a command to scan an answer sheet with the answers already written on it.

[0141] Figure 9 is a diagram illustrating an example of how scoring criteria items can be set, changed, or deleted.

[0142] Figure 9(a) is a diagram showing the configuration extracted from the example in Figure 8, specifically focusing on scoring criteria 801a and the scoring board 802.

[0143] The item deletion button 901 is a button that accepts instructions from the user to delete an item in a cluster defined in scoring criteria 801a.

[0144] Furthermore, if there is only one item, the system may be configured to display an error message indicating that it cannot be deleted because there is only one item, or it may be configured to delete scoring criterion 801a as received.

[0145] The item addition button 902 is a button that accepts instructions from the user to add items to the cluster defined in scoring criterion 801a. Instructions for 801b and 801c can be accepted in the same way.

[0146] In Figure 9(a), when the item addition button 902 is pressed, the transition occurs as shown in Figure 9(b), and the item "Mark 2" linked to scoring criterion 801a is added to both scoring criterion 801a and the scoring board 802. The transition continues as shown in Figure 9(b).

[0147] In this case, the items "Mark 2 Intermediate Steps" and "Mark 3 Calculations" which are linked to scoring criterion 801b, and the items "Mark 4 Numbers" and "Mark 5 Units" which are linked to scoring criterion 801c, will have their numbers shifted down to "Mark 3 Intermediate Steps," "Mark 4 Calculations," "Mark 5 Numbers," and "Mark 6 Units."

[0148] Furthermore, in Figure 9(b), when a click is received on the added item, the selection options will drop down as shown in Figure 9(c), allowing the user to select an item (or inputting an item).

[0149] Furthermore, in Figure 9(c), when a click on the scoring is received, the second cell from the top of the scoring column becomes active, as shown in Figure 9(d), and it becomes possible to input the scoring. In this case, "2" was entered.

[0150] Then, in Figure 9(d), when the item delete button 901 is pressed, item 2 is deleted, and the system transitions to the same state as shown in Figure 9(a).

[0151] In this case, the items "Mark 3 Intermediate Steps" and "Mark 4 Calculations" which are linked to scoring criterion 801b, and the items "Mark 5 Numbers" and "Mark 6 Units" which are linked to scoring criterion 801c, will be renumbered to "Mark 2 Intermediate Steps," "Mark 3 Calculations," "Mark 4 Numbers," and "Mark 5 Units."

[0152] Figure 10 is a diagram showing an example of a screen that receives instructions from examinees to scan their completed answer sheets.

[0153] In the screen shown in Figure 10, similar to the screen shown in Figure 3, the user can input instructions to acquire image data using the image data acquisition button 301. However, it differs from Figure 3 in that it accepts instructions to acquire image data with the answers already written on it.

[0154] Furthermore, the back button 409 is a button that receives instructions from the user to transition to the display screen shown in Figure 8, which accepts settings related to scoring criteria.

[0155] Figure 11 is a diagram showing an example of a screen that accepts corrections to the information being evaluated.

[0156] In Figure 11, the answer extraction criterion 501, the intermediate calculation extraction criterion 601, and the student ID number extraction criterion 602 are displayed as auxiliary indicators. However, these auxiliary indicators may be omitted, or they may be made available to be toggled on or off.

[0157] Cluster identification displays 1101a to c are clusters identified as targets for evaluation based on the answer extraction criterion 501 and the intermediate calculation extraction criterion 601.

[0158] For example, the cluster identification display 1101a is an object used to identify and display clusters that are subject to evaluation, defined as scene diagrams based on the scoring criteria 801a, from among the handwritten information around the intermediate calculation extraction criteria 601 on the blank answer sheet shown in Figure 8.

[0159] Furthermore, the cluster identification display 1101b is a cluster that is subject to evaluation, defined as the intermediate steps and their calculation results, based on the scoring criteria 801b, from the information handwritten (drawn) in the vicinity of the intermediate calculation extraction criteria 601 on the blank answer sheet shown in Figure 8.

[0160] Furthermore, the cluster identification display 1101c is a cluster that is subject to evaluation, defined as the "numbers" and "units" of the answer, based on the scoring criteria 801c, from the information handwritten (drawn) in the vicinity of the answer extraction criteria 501 on the blank answer sheet shown in Figure 8.

[0161] Here, we will further discuss answer field 405, answer extraction criteria 501, and cluster identification display 1101c.

[0162] In this embodiment, the answer extraction criteria 501 are set to overlap with the answer field 405, and in Figure 11, the handwritten answers to be evaluated are written outside the answer field 405.

[0163] Furthermore, if the handwritten answers are extracted from the image data in this state using the method described in Patent Document 1, based on the "answer field shape information," the image of the inside of the answer field 405, i.e., the illegible image shown in Figure 14(b)i), is simply extracted, making evaluation difficult.

[0164] On the other hand, since the answer extraction criterion 501 functions as a "criterion" for extracting information to be evaluated, it is possible to extract clusters to be evaluated from its vicinity.

[0165] Therefore, as shown in Figure 11, even if a portion of the handwritten answer falls outside the answer extraction criteria 501, the cluster identification display 1101c can extract the clusters to be evaluated in a suitable manner that does not hinder the evaluation, based on the answer extraction criteria 501.

[0166] Furthermore, Figure 13 will later describe another example of how clusters to be evaluated can be extracted in a suitable manner that does not hinder the evaluation, based on the answer extraction criteria 501.

[0167] In Figure 11, the scoring board 802 displays the results of calculating the evaluation (score) of the clusters identified by the cluster identification display 1101, based on the answer extraction criteria 501 and the intermediate calculation extraction criteria 601.

[0168] Specifically, for example, the clusters in cluster identification display 1101a are determined to correspond to the content of scoring criterion 801a using technologies such as OCR and machine learning algorithms, and the answer as a scene diagram is judged to be appropriate.

[0169] Furthermore, "Scene Diagram #1" is assigned 2 points.

[0170] Similarly, the clusters in cluster identification display 1101b were determined to correspond to the content of scoring criteria 801b using technologies such as OCR and machine learning algorithms, and the intermediate steps and calculations were judged to be appropriate.

[0171] Furthermore, "Mark 2: Intermediate steps" is worth 3 points, and "Mark 3: Calculation" (correctness of the intermediate calculation) is worth 2 points.

[0172] Similarly, the clusters in cluster identification display 1101c are determined to correspond to the content of scoring criteria 801c through technologies such as character recognition and machine learning algorithms, and "Maru 4 Numeric" (answer num) is determined to be appropriate, while "Maru 5 Unit" (answer unit) is determined to be inappropriate.

[0173] The answers are scored as follows: "4 numbers" are worth 2 points, and "5 units" are worth 0 points.

[0174] More preferably, similar to the scoring criteria 801 in Figure 8, the cluster identification display 1101 is provided so that the user can arbitrarily change the range of the automatically assigned cluster by accepting the operation of a handle.

[0175] Student ID identification display 1102 is a cluster identified as a student ID number based on student ID number extraction criteria 602. The "*" mark in student ID identification display 1102 indicates that the handwritten characters were read as "0002" through video analysis.

[0176] More preferably, as shown in the scoring board 802 in Figure 11, the evaluation of each cluster is pre-reflected in the program of the information processing device, and the scorer can double-check and confirm / correct the content. In this embodiment, this can be implemented as shown in Figure 12.

[0177] This is because it is expected to reduce the workload of graders and decrease the likelihood of grading errors compared to evaluations by graders alone.

[0178] The student ID number display counter 1103 is a counter that displays images of answer sheets showing student ID numbers read by the attendance number identification display 1102, and is linked to these images.

[0179] The counter feed 1104 is a button that, upon being pressed by the user, receives a command to switch the display of the image data of the answer sheet linked to the student ID number read by the attendance number identification display 1102.

[0180] In the state shown in Figure 11, if the "△" button is pressed, the display will switch to the image data of the answer sheet for the previous student ID number, i.e., page 5 / 6 of student ID number 0001.

[0181] Furthermore, if the "▽" button is pressed while the system is in the state shown in Figure 11, the display will switch to the image data of the answer sheet for the next student ID number, i.e., page 5 / 6 of student ID number 0003.

[0182] In Figure 11, the back button 409 is a button that receives a command from the user to transition to the display screen shown in Figure 10, which accepts a command to scan the answer sheet with the answers already written on it.

[0183] Furthermore, the "Proceed" button 410 in Figure 11 is a button that receives instructions from the user to transition to the display screen shown in Figure 12, where scoring and corrections for the evaluation target are accepted.

[0184] Figure 12 is a diagram showing an example of a screen for receiving scores and corrections for the evaluation target.

[0185] Intermediate calculation evaluation area 1201 is an area that displays the clusters extracted for evaluation and the clusters in the example solution side by side.

[0186] In Figure 12, the clusters extracted for evaluation (hand-drawn scene diagrams and intermediate calculation formulas) are the clusters defined by cluster identification displays 1101a and 1101b, which were read in Figure 10 and determined in Figure 11. Furthermore, in Figure 12, the clusters in the example solution (scene diagrams and intermediate calculations) are the clusters defined in Figure 7 and Figure 8 using scoring criteria 801a and 801b.

[0187] Furthermore, as shown in the example of "10+20=30 minutes" in Figure 13, when arranging them side by side, it is preferable to have a function that allows adjustment of the order and scale using technologies such as OCR or machine learning algorithms so that the handwritten answers correspond to the example answers.

[0188] The answer evaluation area 1202, like the intermediate calculation evaluation area 1201, is an area that displays the clusters defined by the cluster identification display 1101c, which were read in Figure 10 and determined in Figure 11, arranged in order of student ID number, and ready for user scoring.

[0189] Note that in the answer evaluation area 1202, multiple "clusters extracted for evaluation" with different student ID numbers corresponding to the same page (page 5 / 6 in Figure 12) are displayed. However, if there are multiple questions on the same page, it is also acceptable to use question number 401 for the main question or as a separator for each sub-question.

[0190] Furthermore, among the multiple clusters of different student ID numbers displayed in the answer evaluation area 1202, the clusters whose answers correspond to the student ID numbers whose handwritten scene diagrams or intermediate calculations are displayed in the intermediate calculation evaluation area 1201 are clearly identifiable. (In this example, the outer frame is highlighted with a thick border, indicating that it is "identifiable".) As a variation of this example, the intermediate calculation evaluation area 1201 does not necessarily have to be integrated as shown in Figure 12; for example, it may be displayed as a pop-up or in a separate window.

[0191] This is because setting it up in this way has the advantage of allowing for a larger answer evaluation area 1202.

[0192] Evaluation check 1203 is an object that receives evaluations from the user for the clusters (handwritten diagrams, intermediate calculations, and answers) that are the target of evaluation and are displayed in the intermediate calculation evaluation area 1201 and the answer evaluation area 1202. By default, evaluation check 1203 reflects the results displayed on the scoring board in Figure 12.

[0193] Note that the evaluation check 1203 in Figure 12 is configured to determine whether an answer is correct or not using checkboxes, but this is only one example of a mechanism for receiving evaluation input from the user and determining the evaluation.

[0194] For example, in cases where more detailed scoring by the grader is required, such as in long-answer questions in Japanese language exams, the system can be configured to accept numerical input within the range of 0 to 10 points. In this way, various existing input methods can be used to match the type of evaluation required for the answer to the question.

[0195] The answer display area 1204 is the area (display area) where clusters (in this case, handwritten answers) corresponding to the student ID numbers assigned to each answer evaluation area 1202 are displayed.

[0196] Furthermore, the clusters displayed in each answer display area 1204 are designed to be automatically or manually adjusted in position and scale so that they fit within their respective answer display areas 1204.

[0197] The automatic position and scale adjustment function for handwritten answers is described later in Figure 13, and the manual position and scale adjustment function (which accepts user instructions and makes adjustments) is described later in Figure 14.

[0198] Evaluation object 1205 is an object that displays the scoring results to the user in a way that allows them to identify them by classifying them into three levels.

[0199] In Figure 12, a complete answer is indicated by "○", a partial answer by "△", and a completely incorrect answer by "×".

[0200] Furthermore, in the case of a "△" where partial credit has been awarded, the score from the evaluation check 1203 is displayed in the lower right corner so that the total of the partial credit can be identified.

[0201] The "Complete Answer" button 1206 is a button that accepts a "Complete Answer" rating from the user for the answer being evaluated. Therefore, when this button is pressed, the evaluation object 1205 becomes "○".

[0202] The strikethrough 1207 is an object that is automatically or manually (installed upon user request) superimposed on the handwritten answer displayed in the answer display area 1204 to identify incorrect parts of the answer.

[0203] For example, in Figure 12, student ID 0001's answer is "30m," but the correct unit is "minutes," not "m," so the incorrect "m" is crossed out.

[0204] Also, student ID number 0004's answer was "25 minutes," but the correct answer is "30 minutes," so the incorrect part, "25," is crossed out.

[0205] The unit correct answer notation 1208 is an object that displays the correct answers regarding units from the incorrect parts of the answer in a way that can be identified.

[0206] The numerical correct answer notation 1209 is also an object that displays the correct numerical answers among the incorrect parts of the answer in a way that makes them identifiable.

[0207] Furthermore, it is preferable that if the answer (units, numbers, descriptions, drawings, etc.) is incorrect, the answer or example answer is included in the area inside the "△" symbol.

[0208] This is because enclosing important sections with triangular objects makes it easier for graders and users who receive returned graded answers to recognize the areas that need correction.

[0209] The complete correct answer notation 1210 is an object that identifies the example answer and is displayed below the answer with a strikethrough 1207 superimposed if both the numbers and units displayed in the answer display area 1204 are incorrect.

[0210] Furthermore, the complete correct answer notation 1210 is an object that can identify the example answer that will be displayed in the blank answer display area 1204 if nothing is written in the answer display area 1204.

[0211] Thus, if an incorrect handwritten answer is provided for unit correct answer notation 1208, numerical correct answer notation 1209, or complete correct answer notation 1210, a strikethrough 1207 will be displayed below the correct answer.

[0212] Furthermore, in the unit correct answer notation 1208, the numerical correct answer notation 1209, and the complete correct answer notation 1210, if no correct answer is provided, an example answer will be displayed in the blank answer display area 1204 or in the margin of the answer display area 1204.

[0213] This feature is implemented so that graders and users who receive graded answers can clearly see the details of the evaluation, such as whether points were not awarded because there were errors in the answers, or because the answers were not answered at all.

[0214] Furthermore, if the numbers are correct but the units are missing, it is preferable to display the clusters left-aligned rather than center-aligned, so as to ensure sufficient space to write the correct unit notation 1208, as shown in the examples of student ID numbers 0002 and 0008.

[0215] By implementing this system, graders and users who receive graded answer sheets can easily and clearly see if the answer was correct but the units were not applied.

[0216] Evaluation object 1205 and strikethrough 1207 are examples of objects that can identify the evaluation being evaluated, while unit correct answer notation 1208, numerical correct answer notation 1209, and complete correct answer notation 1210 are examples of answer examples, that is, additional information that is beneficial to the evaluation.

[0217] These objects can be reused, which also benefits the test takers.

[0218] Specifically, this involves generating "images of graded answer sheets for return."

[0219] First, we identify the "answer fields and clusters" in the image of the completed answer sheet.

[0220] Then, this is replaced with the object displayed in the answer evaluation area 1202 corresponding to the student ID number, generating a "graded answer sheet image for return."

[0221] The objects referred to here are "answer display area 1204 and cluster" and "evaluation object 1205, strikethrough 1207, unit correct answer notation 1208, numerical correct answer notation 1209, and complete correct answer notation 1210".

[0222] An example will be given based on FIGS. 11 and 12. In the image of the completed answer sheet of student ID number 0002 displayed on the answer sheet display screen 406 in FIG. 11, the answer column 405 that overlaps with the answer extraction criterion 501 and the cluster surrounded by the cluster identification display 1101c become the data to be replaced (deleted).

[0223] Then, as replacement data, the answer display area 1204 displayed in the answer evaluation area 1202 of student ID number 0002 in FIG. 12 is arranged so as to overlap the position of the answer column 405.

[0224] Furthermore, in the same arrangement and scale as the answer display area 1204 in the answer evaluation area 1202 of student ID number 0002 in FIG. 12, the handwritten character "30", the unit correct answer notation 1208 "minutes", and the evaluation object 1205 (triangular object) are arranged as replacement data.

[0225] Such image data is generated as a "graded returned answer image" for providing feedback to each examinee as a test result.

[0226] Note that the specific feedback method is not particularly limited. For example, it may be transmitted to the examinee's terminal, displayed on a display device, or printed out and handed over on paper.

[0227] As described above, by providing the information displayed in the answer evaluation area 1202 in FIG. 12 as reusable replacement information, the examinee can also receive feedback with the "graded returned answer image", which is image data with the answers easily visible and corrected.[[ID=C24]]

[0228] In FIG. The back button 409 in 12 is a button that receives an instruction from the user to transition to the display screen in FIG. 11 that accepts corrections regarding the information to be evaluated.

[0229] FIG. 13 is a configuration diagram showing an example of automatic adjustment of the position and scale of the display of the evaluation target.

[0230] In the configuration diagram above the arrow in Figure 13(a), the answer extraction criteria 501 and cluster identification display 1101c are extracted from Figure 11, and the clusters (handwritten answers) to be evaluated, as defined by the cluster identification display 1101c, are displayed in the answer display area 1204.

[0231] Furthermore, the configuration diagram below the arrow in Figure 13(a) is an example of an embodiment, in which the answer display area 1204 for student ID number 0002, the evaluation object 1205, and the unit correct answer notation 1208 have been extracted and shown from Figure 12.

[0232] As shown above, in Figure 13(a), the clusters (handwritten answers) to be evaluated, defined by the cluster identification display 1101c based on the answer extraction criterion 501, are extracted and displayed in the answer display area 1204.

[0233] The handwritten answer defined by cluster identification display 1101c extends beyond the right and bottom edges of the answer extraction criterion 501. Therefore, in order to display this answer clearly in the answer display area 1204, it is necessary to adjust its scale and position.

[0234] Then, through calculations performed by the information processing device, it is identified that the answer does not have units, and the scale is adjusted so that it fits within the frame of the answer display area 1204. The handwritten answer is displayed left-aligned so that there is a margin on the right side, and the unit correct answer notation 1208 is displayed to the right of it. The evaluation object 1205 is also displayed so that the unit correct answer notation 1208 is inside the area.

[0235] Figure 13(b) is a configuration diagram showing an embodiment of student ID number 0004.

[0236] Figure 13(b) shows the configuration diagram above the arrow, where the answer extraction criteria 501 and cluster identification display 1101c are used, and the clusters (handwritten answers) to be evaluated, as defined by cluster identification display 1101c, are extracted and displayed in the answer display area 1204. In other words, it is the same as Figure 13(a).

[0237] Furthermore, the diagram below the arrow in Figure 13(b) is an example of an embodiment, in which the answer display area 1204 for student ID number 0004, the evaluation object 1205, the strikethrough 1207, and the numerical correct answer notation 1209 have been extracted and shown from Figure 12.

[0238] Thus, in Figure 13(b), the cluster (handwritten answer) for student ID number 0004, as defined by cluster identification display 1101c, is extracted based on the answer extraction criterion 501.

[0239] Then, through calculations performed by the information processing device, the answer is identified as having the correct units but incorrect numbers, so that it fits within the frame of the answer display area 1204. The score is then evaluated as 0 points for both the numbers and units (because it is not appropriate to add points even if only the units are correct). The handwritten answer is then displayed centered, and a strikethrough line 1207 is superimposed on the incorrect number "25," with the correct numerical answer 1209 displayed below it.

[0240] Figure 13(c) is a configuration diagram showing an embodiment of student ID number 0005.

[0241] Figure 13(c) shows the configuration diagram above the arrow, where the answer extraction criteria 501 and cluster identification display 1101c are used, and the clusters (handwritten answers) to be evaluated, as defined by cluster identification display 1101c, are extracted and displayed in the answer display area 1204. In other words, the example shown in Figure 13(c) is the same as those in Figures 13(a) and 13(b).

[0242] Furthermore, the configuration diagram below the arrow in Figure 13(c) is an example of an embodiment, in which the answer display area 1204 for student ID number 0005 and the evaluation object 1205 are extracted and shown from Figure 12.

[0243] Thus, in Figure 13(b), the cluster (handwritten answer) for student ID number 0004, as defined by cluster identification display 1101c, is extracted based on the answer extraction criterion 501.

[0244] This handwritten answer, defined by cluster identification display 1101c, is written in such a way that the upper half extends beyond the answer extraction criterion 501. Therefore, this answer also needs to be scaled and positioned to be displayed clearly in the answer display area 1204.

[0245] Then, through calculations performed by the information processing device, the numerical and unit values ​​of the answer are identified as correct so that they fit within the frame of the answer display area 1204. The score is then evaluated as 2 points for the numerical value and 1 point for the unit, and the handwritten answer is displayed centered together with the evaluation object 1205.

[0246] Figure 14 is a diagram illustrating an example of manual adjustment of the position and scale of the display to be evaluated.

[0247] Figure 14(a) is a modified example of the answer evaluation area 1202 for student ID number 0002 in Figure 12.

[0248] The answer evaluation area in Figure 14(a) differs from the answer evaluation area 1202 for student ID number 0002 in Figure 12 in that it has the following two buttons.

[0249] The first number assignment button 1401, when pressed, if there is a handwritten answer in the answer display area 1204, displays a strikethrough line 1207 over the number, as in the example of student ID number 0004 in Figure 12, and displays the correct number notation 1209 below it.

[0250] Furthermore, the number assignment button 1401, when pressed, displays the complete correct answer notation 1210 in the answer display area 1204 if there is no handwritten answer in the answer display area 1204, as shown in the example of student ID number 0003 in Figure 12.

[0251] The second credit awarding button 1402, when pressed, if there is a handwritten answer in the answer display area 1204, displays a strikethrough line 1207 over the credit section, as in the example of student ID number 0001 in Figure 12, and displays the correct credit answer notation 1208 below it.

[0252] Furthermore, the credit awarding button 1402, when pressed, if there are no credits in the handwritten answer in the answer display area 1204, writes the correct credits notation 1208 in the margin to the right of the handwritten answer in the answer display area 1204, as shown in the example of student ID number 0002 in Figure 12.

[0253] Figure 14(b)i) is a diagram showing only the answer display area 1204 of Figure 14(a).

[0254] When the user selects an answer display area 1204 as shown in Figure 14(b)i), the clusters to be evaluated (answer clusters 1403) extracted by the cluster identification display 1101 in Figure 11 are displayed as shown in Figure 14(b)ii).

[0255] In this case, any portion that extends beyond the answer display area 1204 will be displayed in an identifiable manner when the answer cluster 1403 accepts a selection and becomes active. For example, in Figure 14(b)ii), the portion that does not extend beyond the area is represented in a dark color, and the portion that extends beyond the area is represented in a light color.

[0256] This solution cluster 1403 is equipped with a handle that allows the user to manually adjust its scale and position by accepting user input.

[0257] For example, Figure 14(b)iii) shows an example of dragging the answer cluster 1403 so that the upper left corner of the answer cluster 1403 corresponds to the answer display area 1204, as in ii).

[0258] Furthermore, Figures 14(b)iii) to vi) show an example of how the scale is changed by moving the handle so that the bottom edge of the answer display area 1204 corresponds to the bottom right corner of the answer cluster 1403.

[0259] On the other hand, the transition from Figure 14(b)ii) to vii) is achieved by double-clicking. In Figure 14(b)ii), when the user double-clicks on answer cluster 1403, the processes shown in Figures 14(b)ii) through vi) are automatically executed.

[0260] Figure 14(c) is a configuration diagram showing an example of the state in Figure 14(b) vi) or vii), where the evaluation of the answer cluster 1403 is received by the evaluation check 1203, and the user has pressed the number assignment button 1401.

[0261] Figure 15 is a diagram showing an example of the main flow of this embodiment.

[0262] In step S1501, the information processing device 120 obtains image data of a blank answer sheet from the image data providing device 110.

[0263] In step S1502, the information processing device 120 receives from the user whether the image data acquired from the image data providing device 110 in step S1501 is acceptable or not.

[0264] In step S1503, the information processing device 120 uses the image data acquired from the image data providing device 110 in step S1501 to set the answer extraction criteria 501.

[0265] In step S1504, the information processing device 120 uses the image data acquired from the image data providing device 110 in step S1501 to set the intermediate calculation extraction criterion 601 and the student ID number extraction criterion 602.

[0266] In step S1505, the information processing apparatus 120 acquires the image data of the answer sheet including the answer example from the image data providing apparatus 110.

[0267] In step S1506, the information processing apparatus 120 sets a cluster serving as a scoring standard using the image data acquired from the image data providing apparatus 110 in step S1505.

[0268] In step S1507, the information processing apparatus 120 accesses the image data providing apparatus 110 and acquires the image data of the answer sheet with answers filled in.

[0269] In step S1508, the information processing apparatus 120 accepts a correction regarding information (cluster) to be evaluated based on the image data acquired from the image data providing apparatus 110 in step S1507.

[0270] In step S1508, the information processing apparatus 120 accepts scoring and correction of the evaluation target based on the image data acquired from the image data providing apparatus 110 in step S1507.

[0271] FIGS. 16 to 19 are configuration diagrams showing an example of the sub - flow of the present embodiment.

[0272] FIG. 16(a) is a configuration diagram showing an example of the flow of FIG. 3, step S1501.

[0273] In step S1601, the information processing apparatus 120 accepts from the user a device for sending a request to acquire the image data of a blank answer sheet.

[0274] In step S1602, the information processing apparatus 120 determines whether or not the request was accepted in step S1601. If it is determined that the request was accepted, the process proceeds to step S1603; on the other hand, if it is determined that the request was not accepted, the process returns to step S1601.

[0275] In step S1603, the information processing apparatus 120 sends a request to acquire and transmit image data to the device received in step S1601.

[0276] Note that in the flow of this figure, only the case of sending a request to the scanner 111 is described.

[0277] This is because when the information processing apparatus 120 sends a request to acquire and transmit image data to other devices, there is a similar flow in which the receiving device transmits the image data to the information processing apparatus 120 according to the request.

[0278] In step S1604, the scanner 111 receives the request to acquire and transmit the image data sent by the information processing apparatus 120 in step S1603.

[0279] In step S1605, the scanner 111 determines whether the request received in step S1604 was received. If it is determined that the request was received, the process proceeds to step S1606. On the other hand, if it is determined that the request was not received, the process returns to step S1604.

[0280] In step S1606, the scanner 111 acquires the image data of the blank answer sheet based on the request received in step S1604.

[0281] [[ID=2十三条]] In step S1607, the scanner 111 transmits the image data acquired in step S1606 to the information processing apparatus 120.

[0282] In step S1608, the information processing apparatus 120 receives the image data of the blank answer sheet transmitted by the scanner 111 in step S1607.

[0283] Figure 16(b) is a configuration diagram showing an example of the flow of step S1502 in Figure 4.

[0284] In step S1609-1, the information processing device 120 extracts the question numbers (major and minor questions) from the image data of the blank answer sheet received in step S1608, and generates a data table for managing the question numbers (major and minor questions).

[0285] In step S1609-2, the information processing device 120 accepts the pressing of the back button 409.

[0286] In step S1610, the information processing device 120 determines whether or not it received confirmation that the back button 409 was pressed in step S1609. If it determines that it received confirmation, it proceeds to step S1601; otherwise, it proceeds to step S1611.

[0287] In step S1611, the information processing device 120 accepts the pressing of the forward button 410.

[0288] In step S1612, the information processing device 120 determines whether or not it has received confirmation of the button 410 being pressed in step S1611. If it determines that it has received confirmation, the step ends; otherwise, it proceeds to step S1609.

[0289] Figure 16(c) is a diagram showing an example of the flow in Figure 5, step S1503.

[0290] In step S1613, the information processing device 120 uses image recognition to identify the areas of the blank answer sheet image data acquired in step S1608 that should be identified as answer fields, and sets the answer extraction criteria 501.

[0291] Let me elaborate on this step.

[0292] In this embodiment, the answer fields on a blank answer sheet are identified first.

[0293] Then, answer extraction criterion 501 is set in the position corresponding to this answer field.

[0294] Here, these processes are summarized and represented as one step.

[0295] In step S1614, the information processing apparatus 120 receives a request to change the position or range of the answer extraction criterion 501 set in step S1613 through an operation of the user's handle.

[0296] In step S1615, the information processing apparatus 120 determines whether or not it has received a request to change the position or range of the answer extraction criterion 501 in step S1614. If it is determined that the request has been received, the process proceeds to step S1616; on the other hand, if it is determined that the request has not been received, the process proceeds to step S1617.

[0297] In step S1616, the information processing apparatus 120 performs a process of changing the position or range of the answer extraction criterion 501 based on the change request received in step S1615.

[0298] In step S1617, the information processing apparatus 120 receives a depression of the return button 409. [[ID=​​​​​​​​​​​​​ Figure 17(a) is a diagram showing an example of the flow of step S1504 in Figure 6.

[0303] In step S1701, the information processing device 120 uses image recognition to identify the areas of the blank answer sheet image data acquired in step S1608 that should be identified as answer fields, and sets the intermediate calculation extraction criterion 601 and the student ID number extraction criterion 602.

[0304] In step S1702, the information processing device 120 receives requests from the user to change the position or range of the intermediate calculation extraction criteria 601 and the student ID number extraction criteria 602 set in step S1701, through the operation of the user's handle.

[0305] In step S1703, the information processing device 120 determines whether it received a request to change the position or range of the intermediate calculation extraction criterion 601 or the student ID number extraction criterion 602 in step S1702. If it determines that it received the request, it proceeds to step S1704; otherwise, it proceeds to step S1705.

[0306] In step S1704, the information processing device 120 performs a process to change the position and range of the intermediate calculation extraction criterion 601 and the student ID number extraction criterion 602 based on the change request received in step S1702.

[0307] In step S1705, the information processing device 120 accepts the press of the back button 409.

[0308] In step S1706, the information processing device 120 determines whether or not it received confirmation that the back button 409 was pressed in step S1705. If it determines that it received confirmation, it proceeds to step S1613; otherwise, it proceeds to step S1707.

[0309] In step S1707, the information processing device 120 accepts the pressing of the forward button 410.

[0310] In step S1708, the information processing device 120 determines whether or not it has received confirmation of the button 410 being pressed in step S1619. If it determines that it has received confirmation, the step ends; otherwise, it proceeds to step S1702.

[0311] Figure 17(b) is a diagram showing an example of the flow in Figure 7, step S1505.

[0312] In step S1709, the information processing device 120 accepts the pressing of the back button 409.

[0313] In step S1710, the information processing device 120 determines whether or not it received confirmation that the back button 409 was pressed in step S1709. If it determines that it received confirmation, it proceeds to step S1702; otherwise, it proceeds to step S1711.

[0314] Here, we will summarize the steps S1711 to S1718.

[0315] The flow of steps S1711 to S1718 in Figure 17(b) is the same as the flow in Figure 16(a). The differences are that instead of scanning image data of a blank answer sheet, it scans image data of an answer sheet containing example answers, and the system transitions to step S1702 when the back button 409 is pressed.

[0316] Therefore, a detailed explanation of steps S1711 through S1718 will be omitted.

[0317] Figure 18(a) is a diagram showing an example of the flow in Figure 8, step S1506.

[0318] In step S1801, the information processing device 120 uses image recognition to identify the parts of the image data of the answer sheet, including the answer examples obtained in step S1718, that should be identified as answer examples. Then, it sets the scoring criteria 801, defines the clusters that will serve as the scoring criteria, and displays the scoring criteria 801.

[0319] In step S1802, the information processing device 120 automatically calculates the item names and points for each cluster based on the clusters defined by the scoring criteria 801 set in step S1801, sets up the scoring board 802, and displays it.

[0320] In step S1803, the information processing device 120 receives a request to change the position or range of the scoring criteria 801 set in step S1801 through the user's operation of a handle.

[0321] In step S1804, the information processing device 120 determines whether or not it received a request to change the position or range of the scoring criteria 801 in step S1803. If it determines that it received the request, it proceeds to step S1805; otherwise, it proceeds to step S1806.

[0322] In step S1805, the information processing device 120 performs a process to change the position and range of the scoring criteria 801 based on the change request received in step S1803.

[0323] In step S1806, the information processing device 120 accepts the press of the back button 409.

[0324] In step S1807, the information processing device 120 determines whether or not it received confirmation that the back button 409 was pressed in step S1806. If it determines that it received confirmation, it proceeds to step S1709; otherwise, it proceeds to step S1808.

[0325] In step S1808, the information processing device 120 accepts the pressing of the forward button 410.

[0326] In step S1809, the information processing device 120 determines whether or not it has received confirmation of the button 410 being pressed in step S1808. If it determines that it has received confirmation, the step ends; otherwise, it proceeds to step S1802.

[0327] Figure 18(b) is a diagram showing an example of the flow of step S1507 in Figure 10.

[0328] Here, we will summarize the steps S1810 to S1718.

[0329] The flow of steps S1810 to S1819 in Figure 18(b) is the same as the flow and diagram in Figure 17(b), except that it scans image data of an answer sheet with answers already written on it, rather than image data of a blank answer sheet or an answer sheet containing example answers.

[0330] Therefore, a detailed explanation of steps S1810 through S1819 will be omitted.

[0331] Figure 19(a) is a diagram showing an example of the flow of step S1508 in Figure 11.

[0332] In step S1901, the information processing device 120 extracts clusters that should be evaluated (scored) from the image data of the answer sheet containing the example answer acquired in step S1819, based on image recognition and scoring criteria 801. Then, it displays the cluster identification display 1101.

[0333] Let me elaborate on this step.

[0334] In this embodiment, the cluster extraction rule to be evaluated (scored) is, as shown in Figures 5 and 23 above, one in which "at least a portion of the clusters (cluster identification display 1101) relate to the answer extraction criterion 501".

[0335] Therefore, in this step, the handwritten characters on the answer sheet are identified as multiple clusters. Then, based on the answer extraction criteria 501 and intermediate calculation extraction criteria 601 defined in a portion of the answer sheet, clusters that meet the extraction rules are selected as subjects for evaluation (subject to scoring).

[0336] Furthermore, if a cluster extracted based on answer extraction criterion 501 also falls under intermediate calculation extraction criterion 601, the cluster extracted based on answer extraction criterion 501 is extracted as handwritten text written in the answer field. Therefore, even though it falls under intermediate calculation extraction criterion 601, it is excluded from being extracted as handwritten text in intermediate calculations.

[0337] Figures 11 and 12 are given as examples. The cluster identified by cluster identification display 1101c (the handwritten characters "30") is a cluster that applies to both the answer extraction criterion 501 and the intermediate calculation extraction criterion 601. However, it is extracted as handwritten characters written in the answer field and "30" is displayed in the answer display area 1204 for student ID number 0002 in Figure 12.

[0338] In other words, cluster "30" is a cluster that also falls under the intermediate calculation extraction criterion 601, but it is extracted as handwritten text written in the answer field. Therefore, it is excluded from the selection of handwritten text for intermediate calculations, and "30" is not displayed in the intermediate calculation evaluation area 1201.

[0339] In step S1902, the information processing device 120 performs automatic scoring for each cluster based on the clusters extracted by the cluster identification display 1101 set in step S1901, and displays the scoring board 802 that reflects the results.

[0340] In step S1903, the information processing device 120 receives a request to change the position or range of the cluster identification display 1101, which was set in step S1901, through the user's operation of a handle.

[0341] In step S1904, the information processing device 120 determines whether it received a request to change the position or range of the cluster identification display 1101 in step S1903. If it determines that it received the request, it proceeds to step S1905; otherwise, it proceeds to step S1806.

[0342] In step S1905, the information processing device 120 performs processing to change the cluster identification display 1101 and range based on the change request received in step S1803.

[0343] In step S1906, the information processing device 120 accepts the press of the back button 409.

[0344] In step S1907, the information processing device 120 determines whether or not it received confirmation that the back button 409 was pressed in step S1806. If it determines that it received confirmation, it proceeds to step S1810; otherwise, it proceeds to step S1908.

[0345] In step S1908, the information processing device 120 accepts the pressing of the forward button 410.

[0346] In step S1809, the information processing device 120 determines whether or not it has received confirmation of the button 410 being pressed in step S1808. If it determines that it has received confirmation, the step ends; otherwise, it proceeds to step S1902.

[0347] Figure 19(b) is a diagram showing an example of the flow in Figure 12, step S1509.

[0348] In step S1910, the information processing device 120 reads each answer and its evaluation / example answer from the data table.

[0349] The example solution shown here is the one obtained in step S1819.

[0350] Furthermore, each answer is a cluster for each student ID number extracted based on the student ID number extraction criterion 602 obtained in steps S1901 and S1905.

[0351] Furthermore, "that evaluation" refers to the scoring results for each cluster that were scored (evaluated) in steps S1902 and S1905.

[0352] In step S1911, the information processing device 120 arranges and displays each cluster read in step S1901, adjusting the location and scale of each cluster so that they are lined up in the predetermined locations as shown in Figure 12.

[0353] In step S1912, the information processing device 120 receives a user instruction to change the state of the answer evaluation area 1202 to an active state.

[0354] In step S1913, the information processing device 120 determines whether it received the user's instruction to change the answer evaluation area 1202 to an active state in step S1912. If it determines that it received the instruction, it proceeds to step S1914; otherwise, it proceeds to step S1915.

[0355] In step S1914, the information processing device 120 performs a process to change the active state of the answer evaluation area 1202 based on the change request received in step S1912. It also performs a process to change the display of the intermediate calculation evaluation area 1201 so that the cluster corresponding to the student cipher being changed is displayed.

[0356] In step S1915, the information processing device 120 receives instructions from the user to change the arrangement and scale of the clusters displayed in each answer display area 1204 through a process as shown in Figures 13 and 14.

[0357] In step S1916, the information processing device 120 determines whether it received an instruction from the user in step S1915 to change the arrangement and scale of the clusters displayed in each answer display area 1204. If it determines that it received the instruction, it proceeds to step S1917; otherwise, it proceeds to step S1918.

[0358] In step S1917, the information processing device 120 performs a process to change the arrangement and scale of the clusters displayed in the answer display area 1204 based on the change request received in step S1915.

[0359] In step S1918, the information processing device 120 receives instructions from the user to change the evaluation of the cluster being evaluated via the evaluation check 1203 or the complete answer button 1206.

[0360] In step S1919, the information processing device 120 determines whether or not it received an instruction from the user in step S1918 to change the evaluation of the cluster being evaluated. If it determines that it received the instruction, it proceeds to step S1920; otherwise, it proceeds to step S1921.

[0361] In step S1920, the information processing device 120 performs a process to change the evaluation of the cluster being evaluated based on the change request received in step S1918.

[0362] In step S1921, the information processing device 120 accepts the pressing of the back button 409.

[0363] In step S1922, the information processing device 120 determines whether or not it received confirmation that the back button 409 was pressed in step S1921. If it determines that it received confirmation, it proceeds to step S1903; otherwise, it proceeds to step S1923.

[0364] In step S1923, the information processing device 120 accepts the pressing of the forward button 410.

[0365] In step S1924, the information processing device 120 determines whether or not it has received confirmation of the button 410 being pressed in step S1923. If it determines that it has received confirmation, the step ends; otherwise, it proceeds to step S1912.

[0366] Figures 20 and 21 are configuration diagrams showing an example of a data table in this embodiment.

[0367] Figure 20(a) is a diagram showing an example of a data table for managing image data on a page-by-page basis.

[0368] The data table in Figure 20(a) receives the press of the image data acquisition button 301 in Figure 3 and manages the image data for each page of the blank answer sheet acquired in step S1608 in Figure 16.

[0369] Furthermore, the data table in Figure 20(a) receives the press of the image data acquisition button 301 in Figure 7 and manages the image data for each page of the answer sheet, including the example answer acquired in step S1718 in Figure 17.

[0370] Furthermore, the data table in Figure 20(a) receives the press of the image data acquisition button 301 in Figure 10 and manages the student ID number and page-by-page image data of the completed answer sheet acquired in step S1819 in Figure 18.

[0371] Page 2001 is stored and managed by acquiring image data on a page-by-page basis in steps S1608, S1718, and S1819 of Figure 16.

[0372] Furthermore, it would be even better to include page-specific information on the answer sheets, and have a mechanism that allows them to be correctly organized and stored in a data table even if the pages are scattered during scanning.

[0373] Student ID number 2002 is stored and managed after being extracted from the completed answer sheet obtained in step S1912 of Figure 18, based on the student ID number extraction criterion 602.

[0374] Image data name 2003 is determined and managed when image data is acquired in steps S1608, S1718, and S1819, with the name assigned to correspond to the image data type, page, and student ID number.

[0375] Figure 20(b) is a data table that manages the answer extraction criteria 501, intermediate calculation extraction criteria 601, and student ID number extraction criteria 602 for each image data page defined in Figures 5 and 6.

[0376] For the major questions (parts) 2004 and the sub-questions (questions) 2005, when the image data is acquired, the question number 401 is read using OCR or other technologies, stored, and managed.

[0377] For Question 2004 and Question 2005, it is sufficient if the divisions of the questions are appropriately classified, so a system that allows for reading according to the content being dealt with should be provided, and there are no particular restrictions on the types of numbers and symbols, such as A, B, C, αβγ, Roman numerals, etc.

[0378] The three extraction criteria 2006 are stored, modified, and managed when setting or changing the answer extraction criteria 501 in steps S1613 and S1616 of Figure 16. They are also stored, modified, and managed when setting or changing the intermediate calculation extraction criteria 601 and student ID number extraction criteria 602 in steps S1701 and S1704 of Figure 17.

[0379] Figure 21(a) is a diagram showing an example of a data table that manages information related to sample answers.

[0380] Answer example number 2101, answer example item 2102, answer example point distribution 2103, and answer example file name 2104 store and manage the answers extracted in steps S1901 and S1905, and the evaluations of answers that were evaluated or modified in steps S1902 and S1920.

[0381] Figure 22 is a diagram illustrating an example of another embodiment for appropriately extracting information to be evaluated without using the answer extraction criteria 501.

[0382] The upper diagrams in Figures 22(a), 22(b), and 22(c) correspond to Figures 13(a), 13(b), and 13(c) respectively, with basically the same configuration, but differ in that the answer extraction criteria 501 are not superimposed and set in the answer field 405.

[0383] In this configuration, for example, the cluster identification display 1101 is first specified as the starting point (point P, point P', point P'') by clicking the mouse pointer, as shown in Figure 22. Then, the mouse pointer is dragged to the desired ending point (point Q, point Q', point Q'') and the click is released. By using such range selection methods and specifying them individually for each image data of each student ID number, users can arbitrarily extract information to be evaluated.

[0384] In other words, by configuring the system to accept input for specifying the cluster identification display 1101 range from the user for each image data of each student ID number, within a portion of the image data area, it becomes possible for users to arbitrarily extract handwritten answer images to be evaluated.

[0385] Furthermore, since the extraction area can be set to match the different positions and font sizes of each test taker's answers, the data can be extracted in a format that is easy for the grader to use for scoring.

[0386] Furthermore, the cluster identification display 1101, which is applied collectively to multiple image data with completed answers, can be configured to accept input for a portion of the area of ​​one of the multiple image data files. This allows the system to reuse the input to recognize clusters at the same location and range as the input received, even when grading answer sheets from different examinees.

[0387] Furthermore, the concepts described in these two examples above can be applied not only to the answer extraction criterion 501 but also to the setting of the intermediate calculation extraction criterion 601.

[0388] The above describes a configuration in which, without using the answer extraction criterion 501 or the intermediate calculation extraction criterion 601, the user provides input to specify a region for extracting information to be evaluated from the image data (specified input acceptance) and then extracts the information.

[0389] Furthermore, configurations other than those specified are the same as those described in Figures 1 to 21, and can address the problems that could not be solved by prior art 1.

[0390] An example of an embodiment has been described above.

[0391] The information processing device 120 acquires information on example solutions with multiple solutions for a given problem, as well as information on the answer of the person who answered the problem. Then, based on the acquired information on example solutions and the answer of the person who answered the problem, the information processing device 120 generates image data that displays one of the multiple solutions to the problem and the content of the answer of the person who answered the problem. The scoring method for cases where multiple solutions exist for a single problem will be explained below with reference to Figures 8, 12, 24, and 25. In the following, one of the example solutions may be referred to as "solution."

[0392] Figures 8 and 24 are diagrams illustrating an example of a screen for receiving settings related to scoring criteria. The answer sheet display screen 406 in Figures 8 and 24 displays a scanned image of the sample answer scanned in Figure 7. The scanned sample answer includes the question number, the question text, and the solution to the problem. Specifically, the question number is "square 5" in Figure 8 and "square 6" in Figure 24. The question text is specifically the text written below "square 5" in Figure 8 and the text written below "square 6" in Figure 24.

[0393] Here, the solution to the problem includes the intermediate steps (calculation formulas) of the answer to the problem. This solution may also include diagrams used in the answer. In this embodiment, for example, a scene diagram may be used as the diagram used in the answer. The solutions to the problems shown in Figures 8 and 24 consist of a scene diagram of the intermediate calculations and calculation formulas.

[0394] In the "Square 5" problem in Figure 8, there is one solution method (one pair of diagrams and calculation formulas) for each problem statement. In contrast, in the "Square 6" problem in Figure 24, there are two solution methods (two pairs of diagrams and calculation formulas) for each problem statement.

[0395] As will be explained in more detail later, in Figure 8, the diagram enclosed by scoring criterion 801a is shown as a scene diagram included in the solution, and in Figure 24, the diagram enclosed by scoring criterion 801a and the diagram enclosed by scoring criterion 801a' are shown as scene diagrams included in the solution. In addition, in Figure 8, the formula enclosed by scoring criterion 801b is shown, and in Figure 24, the formula enclosed by scoring criterion 801b and the formula enclosed by scoring criterion 801b' are shown as calculation formulas included in the solution.

[0396] In Figure 8, the scene diagram enclosed by scoring criterion 801a and the calculation formula enclosed by scoring criterion 801b constitute one pair of solutions. Similarly, in Figure 24, the scene diagram enclosed by scoring criterion 801a and the calculation formula enclosed by scoring criterion 801b, and the scene diagram enclosed by scoring criterion 801a' and the calculation formula enclosed by scoring criterion 801b', constitute two pairs of solutions.

[0397] Here, only combinations of a diagram and a formula that correspond to each other are considered valid solutions. Therefore, for example, a pair of a diagram enclosed in scoring criterion 801a and a formula enclosed in scoring criterion 801b', or a pair of a diagram enclosed in scoring criterion 801a' and a formula enclosed in scoring criterion 801b, are not valid solution combinations. However, this is just one example of a solution, and solutions consisting only of formulas may exist; the rules are not particularly limited in this way.

[0398] The following provides a detailed explanation of the example shown in Figure 24. Problem 2401 shows the problem statement and diagram from Question 6.

[0399] Solution 2402a and Solution 2402a' are examples of solutions that are included in multiple example solutions for Problem 2401, respectively, and they show different examples of solutions (specifically, the calculation formulas and diagrams for intermediate calculations).

[0400] Solution 2402a consists of a diagram enclosed by scoring criterion 801a (Circle 1: Configuration Diagram) and a calculation formula enclosed by scoring criterion 801b (Circle 2: Intermediate Steps, Circle 3: Calculation).

[0401] Furthermore, solution 2402a in Figure 24 is structured to be solved using knowledge acquired through elementary education.

[0402] Solution 2402a' consists of a diagram enclosed by scoring criterion 801a' (circle 1: diagram of the structure) and a calculation formula enclosed by scoring criterion 801b' (circle 2: intermediate steps, circle 3: calculation).

[0403] Furthermore, solution 2402a' in Figure 24 is structured to be solved using knowledge acquired through secondary education.

[0404] The final solution 2403 is the answer to problem 2401 and consists of numbers and units enclosed in scoring criteria 801c (4 circles for numbers, 5 circles for units).

[0405] Thus, while the final solution is the same in solution 2402a and solution 2402a', the intermediate calculations and diagrams are different.

[0406] Based on the above, we will explain the scoring method for problems with multiple possible solutions by comparing Figure 12 and Figure 25.

[0407] Figure 25, like Figure 12, is a diagram illustrating an example of a screen that accepts scoring and corrections for the evaluation target.

[0408] In Figure 12, the evaluation targets are the handwritten scene diagram enclosed in handwritten intermediate calculation 1220a and the calculation formula enclosed in handwritten intermediate calculation 1220b for the "square 5" problems shown in Figures 4, 5, 6, 8, and 11, as well as the handwritten image of the final solution displayed in the answer display area 1204.

[0409] In Figure 12, images of the scene enclosed in Solution Example Intermediate Calculation 1221a and the calculation formula enclosed in Solution Example Intermediate Calculation 1221b are displayed to the right, corresponding to the handwritten scene diagrams and calculation formula images of Handwritten Intermediate Calculation 1220a and Handwritten Intermediate Calculation 1220b.

[0410] Furthermore, problem "Square 5," which corresponds to the evaluation target in Figure 12, is a problem that, as shown in Figure 8, has one question, one example solution with intermediate calculations and a diagram, and one final solution.

[0411] In Figure 25, the evaluation targets are shown as follows: a handwritten diagram of the scene enclosed in handwritten intermediate calculation 1220a and an image of the calculation formula enclosed in handwritten intermediate calculation 1220b for the "square 6" problem shown in Figure 24, and a handwritten image of the final solution displayed in the answer display area 1204.

[0412] In Figure 25, images of the diagram enclosed in Solution Example Intermediate Calculation 1221a and the calculation formula enclosed in Solution Example Intermediate Calculation 1221b are displayed as solutions, corresponding to the handwritten diagrams and calculation formula images of Handwritten Intermediate Calculation 1220a and Handwritten Intermediate Calculation 1220b.

[0413] Furthermore, problem "Square 6," which corresponds to the evaluation target in Figure 25, is a problem in which, as shown in Figure 24, one question has two pairs of intermediate steps and a diagram as an example solution, and one final solution.

[0414] Therefore, in cases like this where "there are multiple solution examples for a single problem," a mechanism is needed to display the solution example intermediate calculation 1221 on the right, which corresponds to the content of the handwritten intermediate calculation 1220 on the left.

[0415] As described above, the information processing device 120 acquires information on multiple possible solutions for a given problem, as well as information on the answers of those solvers. For example, the information processing device 120 acquires information on example solutions based on answer sheets in which the example solutions are written. In the following explanation, we will assume that information on example solutions for each problem is acquired by reading an image of an answer sheet in which the example solutions are written, as shown in Figure 24, but the process is not limited to this method as long as it is possible to acquire information on example solutions in a similar manner. For example, the information processing device 120 may acquire the intermediate steps of the example solutions as text data or the like.

[0416] The information processing device 120 can determine if multiple example answers are present in an image of an answer sheet containing example answers, using known image processing methods such as OCR, machine learning models, or video analysis. In Figure 24, solution 2402a and solution 2402a' are identified, and the intermediate steps and diagrams included in each solution are also identified.

[0417] Alternatively, for example, a user may set an area to display one example answer for a problem, and the content written in that area may be registered as an example answer. For example, when the user presses the button 803 for adding an example answer shown in Figure 8, the information processing device 120 may display a message for registering a new example answer associated with the problem. In the example in Figure 24, only solution 2402a is registered for the problem "square 6," and when the button 803 is pressed, an area for registering a new solution 2402a' (including scoring criteria 801a' and 801b') is set up in the answer sheet display screen 406 (for example, in the form of a copy of solution 2402a), and an additional scoring board 802 is also generated. The additional scoring board 802 in Figure 24 is generated with the following configuration: "Circle 1' Scene Diagram," "Circle 2' Intermediate Steps," "Circle 3' Calculations," "Circle 4 Numbers," and "Circle 5 Units." Furthermore, each of the multiple example answers registered by the information processing device 120 may be editable after registration.

[0418] The information processing device 120 acquires information about the respondent's answers. Here, as explained in step S1910 above, the information processing device 120 can read the answer sheet on which the respondent's answers are written and acquire information about the answers using video analysis technology or the like. Here, the information acquired about the answers includes the intermediate steps and diagrams in the answer.

[0419] In the example in Figure 24, only one final solution 2403 is listed, common to all solution methods. However, it is also possible to provide a separate box for listing the final solution 2403 for each solution method. In this way, the solution methods included in the example solutions may contain information about the final solution to the problem.

[0420] Furthermore, the information processing device 120 generates image data that displays one of several solutions to the problem alongside the content of the respondent's answer, based on the acquired sample solution information and the respondent's answer information. The image data generated by such an information processing device 120 will be explained below with reference to Figure 25.

[0421] In Figure 25, within the intermediate calculation evaluation area 1201, one of several solutions to the problem and the content of the respondent's answer are displayed. By displaying the solution and the answer side by side in this way, the grader can compare them and score the work. The image data generated by the information processing device 120 that displays one solution and the content of the respondent's answer in this manner may be referred to as "scoring image data" below.

[0422] First, the "display side by side" process according to this embodiment will be described below. In this embodiment, displaying two or more pieces of information simultaneously within image data is expressed as "display side by side." For example, when the solution and the answer are displayed side by side, the solution and the answer may be displayed at roughly the same height within the image data (for example, the two scene diagrams may be displayed so that the y-coordinate of the upper left corner or the y-coordinate of the center coincides) in order to make it easier for the grader to compare them. Also, when the solution and the answer are displayed side by side vertically, the solution and the answer may be displayed in parallel vertically within the image data (for example, the two scene diagrams may be displayed so that the x-coordinate of the upper left corner or the x-coordinate of the center coincides). Here, the coordinates within the image data are assumed to be coordinates where the upper left corner of the entire image data is the origin, the vertical direction is the y-axis direction, and the direction perpendicular to the y-axis is the x-axis direction.

[0423] The information displayed side-by-side may include, for example, solutions and answers for figures including a scene diagram, or solutions and answers for intermediate calculations, or other information related to the same scoring item. In Figure 25, as will be described in detail later, a scene diagram enclosed in Solution Example Intermediate Calculation 1221a, representing one of several solutions, and a handwritten scene diagram enclosed in Handwritten Intermediate Calculation 1220a, representing the content of the respondent's answer, are displayed side-by-side. Also in Figure 25, a calculation formula enclosed in Solution Example Intermediate Calculation 1221b, representing one of several solutions, and a calculation formula enclosed in Handwritten Intermediate Calculation 1220b, representing the content of the respondent's answer, are displayed side-by-side. This "displaying side-by-side" process is not particularly limited as long as the information is displayed in a way that allows the user viewing the generated image data to compare them. For example, it may also be a process where only two pieces of information to be compared (without limiting their respective heights, etc.) are displayed within a single image data.

[0424] In the scoring image data of Figure 25, one of several solution methods is displayed, consisting of a scene diagram enclosed by the aforementioned intermediate calculation example 1221a and a calculation formula enclosed by the intermediate calculation example 1221b. In addition, the scoring image data of Figure 25 displays the content of the respondent's answer, consisting of a handwritten scene diagram enclosed by the aforementioned handwritten intermediate calculation 1220a and a calculation formula enclosed by the handwritten intermediate calculation 1220b. This scoring image data may be a single image as a whole, or it may be image data that displays separate images in each of the areas of handwritten intermediate calculation 1220a, handwritten intermediate calculation 1220b, example solution intermediate calculation 1221a, and example solution intermediate calculation 1221b. It may be in any format as long as the solution and the respondent's answer can be displayed in the same way.

[0425] The information processing device 120 can select a solution to display in the scoring image data from among multiple solution methods included in the sample solution, based on the acquired sample solution information and the respondent's answer information. The following describes such a selection process.

[0426] For example, the information processing device 120 may calculate the similarity between each solution method included in the acquired sample solutions and the respondent's answer, and select the solution method with the highest similarity as the solution method to display in the scored image data. Alternatively, the information processing device 120 may use each solution method included in the acquired sample solutions to score the respondent's answer (automatic scoring), and select the solution method that yields the highest score as the solution method to display in the scored image data. Furthermore, a priority order may be set in advance for each solution method included in the sample solutions, and the solution method with the highest priority among the solutions to the problem being processed may be selected as the solution method to display in the scored image data.

[0427] Furthermore, the solution with the highest similarity in the scoring image data may be displayed, and the answer may be automatically scored using the solution displayed at this point. When automatic scoring is performed in this manner, it may be clearly indicated that automatic scoring is being performed so that it is easy for the user to understand that the scoring result is automatic, and the scoring result may be displayed in a manner that makes it clear that it is automatic scoring (for example, by using a specific color). Such processing is expected to reduce the burden on the scorer.

[0428] For example, the information processing device 120 may display a solution selected from among multiple solution methods included in the acquired example answers, based on user input, in the scoring image data. For example, the solution selected by the user from among multiple solution methods displayed in a list format may be displayed in the scoring image data, and the solution displayed in the scoring image data may be switched to another solution method in response to user input. For example, if the user clicks or taps the mouse cursor over the solution currently displayed in the scoring image data, the displayed solution may be switched to a different solution method. Even in such cases, solutions with a high similarity to the respondent's answer (for example, solutions where the calculated similarity score is above a predetermined threshold) and solutions that are not similar may be further distinguished and each may be set to be switchable.

[0429] In this way, even if a solution is not selected based on similarity, by allowing the displayed solution to be switched according to user input, it is possible to provide a system that allows graders to arbitrarily select an example solution and evaluate the intermediate steps while comparing them with the answer's intermediate steps.

[0430] This process generates and displays image data showing the selected solution and the respondent's answer, making it possible to select the most suitable example answer from multiple options and grade it accordingly.

[0431] Furthermore, after generating image data that displays one solution and the respondent's answer in this manner, the information processing device 120 may display the image data and accept input of the user's evaluation result (scoring result) of the displayed answer. The evaluation of the answer on the answer sheet in this manner can be performed in the same way as described above, referring to Figure 12, etc.

[0432] In the example in Figure 25, the handwritten intermediate calculations 1220a and 1220b included in the respondent's answer are compared with solution 2402a and solution 2402a' respectively, and a similarity score is calculated. Solution 2402a is displayed in the scoring image data as the solution with the higher similarity score. If solution 2402a' is calculated as having a higher similarity score, then solution 2402a' will be displayed in the intermediate calculation evaluation area 1201.

[0433] In Figure 25, a button 2501 for switching the solution displayed in the scoring image data and a button 2502 for displaying a different solution are located within the intermediate calculation evaluation area 1201. When the information processing device 120 receives a press of button 2501 by the user, it switches the solution displayed in the scoring image data to a different solution (in the example of Figure 25, to solution 2402a'). In this example, there are two solutions registered for the problem "square 6", so if button 2501 is pressed again after the solution has been switched once, the displayed solution will revert to the solution that was initially displayed.

[0434] Furthermore, when the information processing device 120 receives a press of button 2502 by the user, in addition to displaying the currently displayed scoring image data, it also displays alternative solutions that are not currently displayed. These alternative solutions may be displayed, for example, as a pop-up in a separate window, or a separate area may be created in the blank space within the intermediate calculation evaluation area 1201, and the alternative solutions may be displayed in that area in a way that makes it identifiable that they are alternative solutions. Such displays of alternative solutions will be turned on or off in response to the press of button 2502.

[0435] In this way, by selecting a suitable solution based on the respondent's answer and displaying it along with the answer, it becomes easier for graders to confirm whether the answer is appropriate when performing automatic scoring, and it is also possible to provide a system that makes it easier for graders to perform accurate scoring when they are doing the scoring themselves.

[0436] Alternatively, a recording medium containing a program that implements the functions of the above-described embodiment may be supplied to a system or device, and the computer (or CPU or MPU) of that system or device may read and execute the program stored on the recording medium.

[0437] At least the following can be used as recording media for supplying the program.

[0438] Specifically, these include, for example, flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EPROMs, silicon disks, solid-state drives, etc.

[0439] Furthermore, it goes without saying that the functions of the aforementioned embodiments are realized not only by the computer executing the program it has read, but also by the operating system (OS) running on the computer performing some or all of the actual processing based on the instructions of that program, thereby realizing the functions of the aforementioned embodiments.

[0440] Furthermore, it goes without saying that this also includes cases where, after a program read from a recording medium is written to the memory of a function expansion board inserted into a computer or a function expansion unit connected to a computer, the CPU or other components of the function expansion board or function expansion unit perform some or all of the actual processing based on the instructions of the program code, and the functions of the aforementioned embodiments are realized through that processing.

[0441] Furthermore, by downloading and reading the program for achieving this embodiment from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of this embodiment. Note that all configurations combining the above-described embodiments and their variations are also included in this embodiment. [Explanation of Symbols]

[0442] 120 Information Processing Devices 501 Answer extraction criteria 801 Scoring Criteria Area 1204 Answer display area

Claims

1. A first acquisition means for acquiring information on multiple solution examples for a given problem, and information on the answers of respondents to the problem. A generation means generates first image data that displays the first solution method among the plurality of solutions and the content of the respondent's answer side by side, based on the information of the example solution acquired by the first acquisition means and the information of the respondent's answer. An information processing device characterized by comprising:

2. The system further includes a selection means for selecting the first solution method from among the plurality of solutions based on the answer of the respondent, The information processing apparatus according to claim 1, characterized in that the generation means generates the first image data which displays the example solution of the first solution selected by the selection means and the content of the answer of the respondent side by side.

3. The information processing apparatus according to claim 2, characterized in that the selection means selects the first solution based on the similarity between the solution included in the plurality of solutions and the answer of the respondent.

4. The information processing apparatus according to claim 2, characterized in that the selection means selects, among the plurality of solutions, the solution that yields the highest bonus when used for automatically scoring the answer of the respondent.

5. The information processing apparatus according to claim 2, characterized in that the selection means selects the first solution method based on user input.

6. The information processing apparatus according to claim 1, characterized in that the generation means further generates a second image data which displays side by side an example solution of a second solution method, which is different from the first solution method among the plurality of solution methods, and the content of the answer of the respondent.

7. The information processing apparatus according to claim 6, characterized in that the generation means generates the second image data based on user input.

8. The information processing apparatus according to claim 1, characterized in that the plurality of solutions include intermediate steps or diagrams in the problem.

9. The information processing apparatus according to claim 1, further comprising a second acquisition means for obtaining a user's scoring result for the answer displayed in the first image data after displaying the first image data.

10. For problem 1, the process involves obtaining information on example solutions with multiple solutions for the problem, and information on the answers of the respondents to the problem. A step of generating first image data that displays the first solution method among the plurality of solutions and the content of the respondent's answer side by side, based on the acquired information of the example solution and the information of the respondent's answer, An information processing method characterized by comprising:

11. A program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 9.

Images