Information processing apparatus, information processing method, and program

Abstract

To facilitate evaluation of information extracted from image data.SOLUTION: An information processing apparatus comprising: an acquisition unit configured to acquire one or a plurality of pieces of image data; a reference determination unit configured to determine a first reference for extracting first information to be evaluated from the image data acquired by the acquisition unit to be a part of a region of the image data; An information processing apparatus comprising: extraction means for extracting first information which is located at a predetermined position with respect to the first reference and is to be evaluated; display means for displaying the first information extracted by the extraction means in a predetermined display area for displaying the first information; and evaluation input means for inputting an evaluation of the first information displayed in the predetermined display area by the display means.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, various techniques have been proposed to reduce the workload of workers who evaluate information extracted from image data.

[0003] For example, Patent Document 1 proposes a mechanism for cutting out an answer column image, which is an image of the answer column portion, based on answer column shape information included in the answer sheet image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-129182 A Summary of the Invention [Problem to be solved by the invention]

[0005] This method aims to reduce the workload of graders by extracting answer column images for the same question associated with each examinee from the image data obtained by scanning the answer sheets of multiple examinees, displaying them as a list, and grading them all at once. However, there is an issue with extracting answer column images based on answer column shape information.

[0006] This is because even if the answer sheet is cut out in accordance with the answer sheet shape information, there is a risk that an answer sheet image that is easy for the grader to grade may not be obtained.

[0007] The present invention aims to facilitate the evaluation of information extracted from image data. [Means for solving the problem]

[0008] The present invention is characterized by comprising an acquisition means for acquiring one or more image data; a standard determination means for determining a first standard for extracting first information to be evaluated from the image data acquired by the acquisition means as a part of the area of ​​the image data; an extraction means for extracting first information to be evaluated from the image data, which is located at a predetermined position relative to the first standard, using information regarding the position of the first standard determined by the standard determination means; a display means for displaying the first information extracted by the extraction means in a predetermined display area for displaying the first information; and an evaluation input means for inputting an evaluation of the first information displayed in the predetermined display area by the display means. [Effects of the Invention]

[0009] In this way, according to the present invention, it is possible to easily evaluate information extracted from image data. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a configuration diagram illustrating an example of a system configuration according to an embodiment. [Figure 2] FIG. 1 is a configuration diagram illustrating an example of a hardware configuration according to an embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of a screen that accepts an instruction to scan a blank answer sheet. [Figure 4] FIG. 10 is a diagram illustrating an example of a display screen for accepting the approval or disapproval of acquired image data. [Figure 5] FIG. 10 is a diagram illustrating an example of a display screen for determining answer extraction criteria. [Figure 6] FIG. 10 is a diagram illustrating an example of a display screen for determining interim response criteria, etc. [Figure 7] FIG. 10 is a diagram illustrating an example of a screen that accepts an instruction to scan an example answer. [Figure 8] FIG. 10 is a diagram illustrating an example of a screen for accepting settings related to scoring criteria. [Figure 9] This is a configuration diagram showing an example of setting, changing, and deleting scoring criteria items. [Figure 10] FIG. 10 is a diagram illustrating an example of a screen that accepts an instruction to scan an answer sheet on which answers have been written. [Figure 11] FIG. 10 is a diagram illustrating an example of a screen for accepting corrections regarding information of an evaluation target. [Figure 12] FIG. 10 is a diagram illustrating an example of a screen for accepting marks and corrections to an evaluation target. [Figure 13] FIG. 10 is a configuration diagram showing an example of automatic adjustment of the display position and scale of the evaluation target. [Figure 14] FIG. 10 is a configuration diagram showing an example of manual adjustment of the display position and scale of the evaluation target. [Figure 15] 10 is a flowchart illustrating a flow of information processing according to an embodiment. [Figure 16] 10 is a flowchart illustrating a flow of information processing according to an embodiment. [Figure 17] 10 is a flowchart illustrating a flow of information processing according to an embodiment. [Figure 18] 10 is a flowchart illustrating a flow of information processing according to an embodiment. [Figure 19] 10 is a flowchart illustrating a flow of information processing according to an embodiment. [Figure 20] FIG. 2 is a diagram illustrating an example of a data table according to the embodiment. [Figure 21] FIG. 2 is a diagram illustrating an example of a data table according to the embodiment. [Figure 22] FIG. 10 is a configuration diagram showing an example of another embodiment for appropriately extracting information to be evaluated without using the answer extraction criteria 501. [Figure 23] FIG. 10 is a diagram showing an example of a case where a cluster is extracted from the position reference starting point of the answer extraction criterion 501. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, the embodiments will be described in detail with reference to the drawings.

[0012] FIG. 1 is a configuration diagram showing an example of a system configuration according to an embodiment.

[0013] The image data providing device 110 is a device for providing image data to an information processing device 120, which will be described later.

[0014] As an example, in this embodiment, the scanner 111 acquires three pieces of image data: a blank answer sheet, an answer sheet with sample answers, and an answer sheet with answers already written on it, to generate image data, which is then provided to the information processing device 120 via the communication network 130.

[0015] However, this is merely an example, and image data containing information to be evaluated may be appropriately selected and used.

[0016] For example, the image data source image may be an essay written on a composition sheet, a report written on a report sheet, an illustration drawn in a work entry field, etc. The image data providing device 110 also includes various other devices that acquire, manage, and provide content to be evaluated.

[0017] In addition to the above, copyrighted works such as text, drawings, videos, program code, etc. that can be subject to evaluation can also be provided as image data and used to implement an invention.

[0018] Let us return to the image data providing device 110.

[0019] As another example of the image data providing device 110 instead of the scanner 111, for example, a portable terminal 112 can be used to provide image data of still images or moving images acquired by a camera function to the information processing device 120.

[0020] In addition, a configuration may be adopted in which image data of still images and moving images managed within the image management server 113 is provided to the information processing device 120 via the communication network 130.

[0021] Furthermore, the information processing device 120 itself may also have the function of the image data providing device 110, and may have a camera function, such as the mobile terminal 112, that can acquire image data of still images and moving images.

[0022] The network 130 is a network that connects the hardware according to the embodiment (for example, the image data providing device 110 and the information processing device 120) so that they can communicate with each other.

[0023] The communication method of the communication network 130 may be wireless or wired, and may also be a WAN (Wide Area Network) or a LAN (Local Area Network).

[0024] FIG. 2 is a configuration diagram showing an example of a hardware configuration according to the embodiment.

[0025] An example of hardware according to the embodiment includes at least an image data providing device 110 and an information processing device 120 .

[0026] Reference numeral 201 denotes a CPU, which controls each device connected to a system bus 204 by loading a program stored in a program ROM of a ROM 202 or an external memory 211 into a RAM 203 and executing the program.

[0027] The font ROM of this ROM 202 stores font data and the like, and the data ROM of ROM 202 stores various data. 203 is a RAM, which functions as the main memory, work area, etc. of the CPU 201. 202 is a keyboard controller (input controller), which controls input from 209 and a pointing device (not shown), such as a touchpad.

[0028] The input controller 205 controls the input device 209. The input device 209 also includes a touch panel (touch screen), and in such a case, the input device 209 also functions as the display unit 210 (a display device for displaying a display screen that displays images). The input device 209 is not limited to the above, and any known device can be used. To cite one example, a voice input device or a device that acquires a user's gestures using a camera or tracker and converts them into input instructions can be considered.

[0029] A video card (VC) 206 controls the display on the display unit 210. Various existing output devices can be used for the display unit 210. For example, a screen-type display or a retina-illuminated display is possible, but any output device that allows the user to obtain visual information will suffice.

[0030] A memory controller 207 controls access to an external memory 211 (such as a hard disk), and a network interface card (NIC) 208 controls communication on the communication network 130.

[0031] The image data acquisition unit controller 220 is a controller that controls the image data acquisition unit 221. The image data acquisition unit 221 is a unit that combines mechanisms required to acquire image data, and can take various units suited to the characteristics of the device. For example, in the scanner 111, it is a scan unit that scans images, and in the mobile terminal 112 and the information processing device 120, it is a camera unit that captures still images and videos.

[0032] In addition, in the case of the image management server 113 or the information processing device 120, an external memory 211 for acquiring and storing image data and a network interface card 208 for controlling connection with the communication network 130 for acquiring image data from the outside are assumed.

[0033] FIG. 3 is a diagram showing an example of a screen for accepting an instruction to scan a blank answer sheet (with no answers written on it).

[0034] In FIG. 3, a screen configured as shown in the figure is displayed on the display unit 210 of the information processing device 120, and is set up so that an instruction to acquire image information 402 can be received from the user.

[0035] 4 to 14 are also configured to be displayed on the display unit 210 of the information processing device 120 and to receive instructions from the user, similarly to FIG.

[0036] In addition, the following description of the embodiment will be given assuming, as an example, a system configuration in which a scanner 111 (image data providing device 110) is connected to a PC (information processing device 120) by wire or wirelessly.

[0037] However, the information processing device 120 can be connected to various other terminals and acquire image data. For example, various configurations that combine existing mechanisms are envisioned, such as wirelessly connecting a PC (information processing device 120) to a tablet terminal (mobile terminal 112), or connecting the image management server 113 and the PC (information processing device 120) via a network.

[0038] The image data acquisition buttons 301a to 301c are buttons for accepting an instruction from the user to acquire image data including an answer column from the image data providing device 110 that is communicably connected to the information processing device 120.

[0039] As an example, when the image data acquisition button 301a is pressed by the user, it requests the communicatively connected scanner 111 to scan a document (blank answer sheet) and send the image data obtained, thereby acquiring the image data.

[0040] In addition, the image data acquisition button 301b accepts a button press from the user and requests the mobile terminal 112, which is communicatively connected, to launch a camera app and send image data obtained by photographing a manuscript (blank answer sheet), thereby acquiring the image data.

[0041] In addition, the image data acquisition button 301c accepts a button press from the user, requests the image management server 113, which is communicatively connected, to send image data of a blank answer sheet that it manages, and acquires the image data.

[0042] The information processing device 120 itself may be provided with an image acquisition unit 221 such as a camera, and the unit may be activated to acquire an image (not shown).

[0043] In this embodiment, the information processing device 120 acquires image data by scanning a blank answer sheet (step S1608), assigns the file name "hakushi_0X.png" (X is the page number), and manages it in the data table of Figure 20(a).

[0044] FIG. 4 is a diagram showing an example of a display screen for accepting the approval or disapproval of the acquired image data.

[0045] Figure 4 shows image data of an answer sheet consisting of six pages, each of which is made up of a question number 401, a student ID number entry field 402, a question statement 403, an intermediate calculation answer space 404, and an answer field 405, displayed on an answer sheet display screen 406.

[0046] In Figure 4, question number 401 is assumed to consist of a main question with the number (here, 5) surrounded by a box, and a sub-question (Figure 21) that is expressed as a sub-number (number).

[0047] 4 shows a state in which an answer sheet for a test consisting of one part and six pages is read, and the manuscript page display 407 shows the state in which the fifth page is displayed. Note that the structure of the questions is as follows: Page 1: Main question 1 consisting of two sub-questions; Page 2: Main question 2 consisting of two sub-questions; Pages 3 to 6: Main questions 3, 4, 5, and 6 on each page.

[0048] These question numbers (main question, sub-question) are extracted using OCR or image recognition technology in the processing of step S1609-1 in Figure 16, and data tables such as those shown in Figures 20(b) and (c) are generated, stored in the main question 2001 and sub-question 2002 columns, and managed.

[0049] Returning to FIG. 4, the manuscript display screen 406 is a screen for displaying image data obtained by scanning a blank answer sheet, and displays image data corresponding to the page displayed in the manuscript page display 407.

[0050] Therefore, for example, "page 5 / 6" in FIG. 4 indicates that the fifth page of the image data of a total of six pages is being displayed on the document display screen 406.

[0051] The display forward button 408 is a button that accepts an instruction from the user to switch the page of image data displayed on the document page display 407 and the document display screen 406 .

[0052] A back button 409 in FIG. 4 is a button for accepting an instruction from the user to transition to the display screen of FIG. 3, which accepts an instruction to scan a blank answer sheet.

[0053] In FIG. 4, a displayed forward button 410 is a button for accepting an instruction from the user to transition to the display screen of FIG. 5, which defines the answer extraction criteria.

[0054] FIG. 5 is a diagram showing an example of a display screen for determining answer extraction criteria.

[0055] The answer extraction criteria 501 are criteria that are set in a part of the area of ​​the image of the blank answer sheet scanned in Figure 4 in order to extract the subject of evaluation (information to be written in the answer column in this embodiment) from the "answer sheet with answers written" as shown in Figure 11.

[0056] In this embodiment, the answer extraction criteria 501 identifies the answer field 405 using OCRoptical Character Recognition (or Reader) or image recognition technology, and defines this area as the default answer extraction criteria 501.

[0057] Here, the evaluation target is the handwritten number 30, which is enclosed in the cluster identification display 1101c in FIG. 11 and extends beyond the answer field 405 at the bottom right.

[0058] The 30 handwritten characters to be evaluated are a group or group of meaningful handwritten characters called a "cluster."

[0059] Then, as with the identification of the answer column 405, clusters can be extracted from the image of the answer sheet on which the answers have been written using algorithms such as OCR technology or image recognition technology that can identify handwritten characters as clusters. Also, clusters can be extracted by using an algorithm that groups the differences obtained from the image of the answer sheet on which the answers have been written and the image of a blank answer sheet (on which no answers have been written) into blocks of characters.

[0060] Therefore, unlike the prior art in which the image is cut out along with the answer column, in this embodiment, the answer column is not extracted, and only the handwritten answers can be extracted.

[0061] The answer extraction criteria 501 are criteria set in a part of the area of ​​the answer sheet for the purpose of extracting clusters to be evaluated from the above-mentioned clusters written on the answer sheet.

[0062] In this embodiment, the hatched area is the answer extraction standard 501, and a cluster of handwritten characters included in at least a part of this area is extracted as an evaluation target.

[0063] By setting it up in this way, it is possible to suitably extract clusters of handwritten characters to be evaluated even in cases where, in conventional technology, answers that extend beyond the answer box or are too small for the answer box are cut out according to the answer box shape information, causing problems with grading.

[0064] In this embodiment, as an example, the answer extraction criterion 501 is a rectangular area corresponding to the shape of the answer column, but it does not have to be rectangular and can be, for example, a "point," a circle, or a polygon (not shown).

[0065] For example, if a point (certain coordinates) in an area of ​​the answer sheet is set as the answer extraction criterion 501, a cluster that is at least partially contained within a 100 pixel horizontal and 30 pixel vertical area around that point may be extracted as the target for evaluation.

[0066] In any case, even if the answer protrudes from the answer column or is too small for the answer column, it is sufficient as long as the evaluation target can be appropriately extracted based on the answer extraction criteria 501.

[0067] Here, we will further add some details about the extraction of evaluation targets using Figure 23.

[0068] In this embodiment, the clusters of handwritten characters to be evaluated are extracted in the following procedure.

[0069] First, the answer columns are identified from the image data of the blank answer sheet, and the answer extraction criteria 501 are set based on the identified answer columns.

[0070] Next, handwritten characters are identified in cluster units from the completed answer sheet.

[0071] Then, from the clusters of handwritten characters identified on a cluster-by-cluster basis, clusters of handwritten characters to be evaluated (handwritten characters surrounded by cluster identification display 1101 in FIG. 23) are extracted using answer extraction criteria 501 as a starting point.

[0072] FIG. 23 is a diagram showing an example of a case where a cluster is extracted at the position reference starting point of the answer extraction criterion 501. In FIG.

[0073] In FIG. 23, the X axis is set to the right and the Y axis is set to the bottom, and an answer extraction standard 501 is set as a rectangle of 20 width x 10 height, set from the image data of the answer sheet.

[0074] The vertices a, b, c, and d of the answer extraction standard 501 are located at a(0, 0), b(20, 0), c(20, 10), and d(0, 10), respectively, and the central point o is o(10, 5).

[0075] In (a) to (e) of Figure 23, the clusters surrounded by a 20x10 rectangular cluster identification mark 1101 (vertices a'b'c'd') are in different positions, while in (f) the clusters surrounded by a 10x6 rectangular cluster identification mark 1101 are in different positions.

[0076] The center point o' of the cluster in Figure 23(a) is o'(20,10). The center point o' of the cluster in Figure 23(b) is o'(10,15). The center point o' of the cluster in Figure 23(c) is o'(10,17). The center point o' of the cluster in Figure 23(d) is o'(20,17). The center point o' of the cluster in Figure 23(e) is o'(30,15). The center point o' of the cluster in Figure 23(f) is o'(10,14).

[0077] Based on the above, we will explain which of the handwritten character clusters (a) to (f) in FIG. 23 can be extracted according to the method for extracting the handwritten character clusters to be evaluated.

[0078] As an extraction method, a case where "at least a part of the cluster (cluster identification display 1101) relates to (overlaps, overlaps, matches) the answer extraction criterion 501" will be described.

[0079] Note that this extraction method is merely an example for the purpose of explanation.

[0080] As another example, the extraction method may be set as a default, or multiple default extraction methods may be prepared and the user may select one of them, or a specific extraction method may be specified by the user.

[0081] The following is one example of how the selection or specific extraction method of rules can be received from the user. This is a method in which a mechanism for receiving the selection or input of the extraction method (such as a button, toggle, dropdown, slider, picker, or text field) is provided in the UI shown in Figures 8, 12, and 13. Alternatively, the configuration may be such that the selection or input is received on an independent screen (not shown).

[0082] As an example, a case will be described in which the answer extraction criteria 501 and the cluster identification display 1101 each have grid point data, and the extraction of a cluster is determined by matching the grid points of both.

[0083] For example, consider the condition for a cluster extraction method as "there is a match of lattice points of 2 dots vertically and 2 dots horizontally or more."

[0084] In this case, the positional relationship between the answer extraction criteria 501 and the cluster identification display 1101 that satisfies the conditions is established only in FIG. 23(a).

[0085] This is because, in Figure 23(a), the answer extraction standard 501 and the cluster identification display 1101 match at lattice points of 5 vertical dots and 10 horizontal dots in the area (10,5)(20,5)(20,10)(10,10). In other words, the condition of "matching at lattice points of 2 vertical dots or more and 2 horizontal dots or more" is met.

[0086] On the other hand, in FIG. 23(b), grid points of 1 dot vertically and 20 dots horizontally meet at (0,10) to (20,10).

[0087] In addition, in FIG. 23(e), at (20,10), grid points of 1 dot vertically and 1 dot horizontally abut each other.

[0088] In other words, in the cases of FIG. 23(b) and FIG. 23(e), the condition that "there is a matching of lattice points of 2 dots or more vertically and 2 dots or more horizontally" is not met, and therefore no clusters are extracted.

[0089] Therefore, if the user inputs a change to the cluster extraction method, such as "Extract a cluster when there is a match of lattice points of at least 1 dot vertically and 1 dot horizontally," then Figure 23(b) and Figure 23(e) will satisfy the condition. In other words, they are cases in which a cluster is extracted.

[0090] 23(c), 23(d), and 23(f), there is no point where the grid points of the answer extraction criteria 501 and the cluster identification display 1101 match, so the cluster is not extracted as a target for evaluation.

[0091] Next, the extraction method can be set to "the distance between the center point of each cluster and the answer extraction standard 501 is set to within 11".

[0092] In this case, the distance between the centers of the cluster and the answer extraction criterion 501 is approximately 11.2 in Figure 23(a), 12 in Figure 23(c), approximately 15.6 in Figure 23(d), and approximately 15.6 in Figure 23(e), and in these cases the cluster is not extracted as a target for evaluation.

[0093] On the other hand, in FIG. 23(b), the distance between the centers of the cluster and the answer extraction standard 501 is 10, and in FIG. 23(f), it is 11, and in these cases the cluster is extracted as a target for evaluation.

[0094] The extraction method can also be set to "the shortest distance between each side or point of the cluster and the answer extraction standard 501 is 1 or less."

[0095] In this case, in FIG. 23(a), the cluster and the answer extraction criteria 501 overlap in plane, so the cluster is extracted as a target for evaluation.

[0096] In addition, in FIG. 23(b), the cluster and the answer extraction criterion 501 are in contact with each other at an edge, so the cluster is extracted as a target for evaluation.

[0097] In addition, in FIG. 23(e), the cluster and the answer extraction criterion 501 are in contact at a point, so the cluster is extracted as a target for evaluation.

[0098] In addition, in FIG. 23(f), the shortest distance between the cluster and the answer extraction criterion 501 is 1, so the cluster is extracted as a target for evaluation.

[0099] On the other hand, in FIG. 23(c) and FIG. 23(d), the shortest distance between the cluster and the answer extraction criterion 501 is 2, so in these cases the cluster is not extracted as a target for evaluation.

[0100] As described above, there are several possible methods for extracting evaluation targets based on the answer extraction criteria 501. It is more preferable that the grader be able to freely select which method to use.

[0101] As another embodiment, instead of setting the answer extraction criteria 501, it is possible to accept an arbitrary range specification from the user for each image data of the answer sheets on which answers have been written, and extract clusters to be evaluated. Alternatively, it is also possible to accept an arbitrary cluster specification from the user, and extract clusters to be evaluated.

[0102] To explain this using FIG. 23 as an example, image data with answers written in it is displayed on the display screen, and the system is set up so that the user can arbitrarily specify the range within the dotted line of the cluster identification display 1101.

[0103] By providing the clusters in this way, the user can arbitrarily select and extract any of the clusters in FIGS. 23(a) to 23(f) as clusters to be evaluated.

[0104] Furthermore, this embodiment is not limited to extracting handwritten answers, but can also accept range and cluster specifications from the user for intermediate calculations and scene diagrams, as described below, and extract clusters to be evaluated.

[0105] Next, the adjustment of the answer extraction criteria 501 will be described.

[0106] In the embodiment of Figure 5, the answer extraction standard 501 has an object for the user to specify an area, and handles are provided for adjusting the size and position, and the user can operate it by dragging and dropping, allowing the user to adjust it as desired.

[0107] For example, Figure 5 shows the situation when the user is adjusting the area by placing the mouse cursor on the handles indicated by "circles" located at the center of the sides and four vertices of the answer extraction criterion 501 and dragging them.

[0108] The above is an example in which the user can arbitrarily determine the criteria for the answer extraction criteria 501 by accepting user operations such as dragging and dropping. If there is a function that can automatically set the initial position appropriately, the burden on the worker will be reduced, so it is desirable to have such a function.

[0109] Next, the setting of the default answer extraction criteria 501 will be explained.

[0110] For example, in the example shown in Figure 5, there is a mechanism that detects character information that can be obtained by OCR, such as "answer" and "answer," as well as information that is often used as the answer section of test questions, such as "square boxes" and "underlines," and determines the initial position based on the position of the detected information.

[0111] Another method is to use a mechanism that allows the initial position of the answer extraction criteria 501 to be set more flexibly, for example, by using a trained model.

[0112] A back button 409 in FIG. 5 is a button for accepting an instruction from the user to transition to the display screen of FIG. 4, which accepts the approval or disapproval of the acquired image data.

[0113] Further, a forward button 410 in FIG. 5 is a button for receiving an instruction from the user to transition to a display screen shown in FIG. 6, in which answer criteria and the like are determined along the way.

[0114] FIG. 6 is a diagram showing an example of a display screen for determining interim response criteria and the like.

[0115] The intermediate calculation extraction criteria 601 is an area that serves as a criterion for extracting information to be evaluated as intermediate calculations.

[0116] As shown in Figure 6, like the answer extraction standard 501, the intermediate calculation extraction standard 601 also has handles for adjusting the size and position of the object that allows the user to specify the area, and can be adjusted as desired by dragging and dropping.

[0117] If the answer extraction criterion 501 is included in the area of ​​the intermediate calculation extraction criterion 601, the area is set to the answer extraction criterion 501, not the intermediate calculation extraction criterion 601.

[0118] Similarly to the answer extraction criteria 501, the intermediate calculation extraction criteria 601 can use various methods for extracting forms.

[0119] Furthermore, it is desirable that the intermediate calculation extraction criteria 601, like the answer extraction criteria 501, have a function that allows the initial position to be automatically set appropriately.

[0120] The student ID number extraction criteria 602 is an area that serves as a criterion for extracting a personal identification number such as a student ID number at the school or cram school to which the test taker belongs, or an examination number.

[0121] As with the answer extraction criteria 501, the student ID number extraction criteria 602 can use various existing methods for extracting shapes according to the circumstances when implementing the invention, and it is desirable that there is a function that can automatically set the initial position appropriately.

[0122] A back button 409 on the display screen shown in FIG. 6 is a button for accepting an instruction from the user to transition to the display screen of FIG. 5 for determining the answer extraction criteria.

[0123] Further, a forward button 410 on the display screen shown in FIG. 6 is a button for accepting an instruction from the user to transition to the display screen shown in FIG. 7, which accepts an instruction to scan an answer example.

[0124] This button is used to determine the intermediate calculation standard 601 and to receive an instruction from the user to transition to the next screen (FIG. 7).

[0125] FIG. 7 is a diagram showing an example of a display screen for accepting an instruction to scan an example answer.

[0126] The display screen shown in Figure 7 is a screen for accepting instructions from the user to acquire image data using the image data acquisition button 301, similar to the display screen shown in Figure 3, but differs from Figure 3 in that it accepts instructions to acquire image data including sample answers.

[0127] Further, a back button 409 on the display screen shown in FIG. 7 is a button for accepting an instruction from the user to transition to the display screen of FIG. 6 for determining interim answer criteria and the like.

[0128] FIG. 8 is a diagram showing an example of a screen for accepting settings related to the scoring criteria.

[0129] Figure 8 shows the screen displaying image data including the sample answers acquired in Figure 7, as well as answer extraction criteria 501, intermediate calculation extraction criteria 601, and student ID number extraction criteria 602 as auxiliary displays.

[0130] However, the display of this auxiliary display may be omitted, or the auxiliary display may be provided so that it can be switched on and off.

[0131] The scoring criteria 801 is an object that defines "clusters" for assigning partial points based on sample answers.

[0132] "Cluster" is a word that means "group," "gathering," or "assembly." However, in this embodiment, the information within the range enclosed by the scoring criteria 801 is treated as one information group (= cluster).

[0133] For example, in the display screen shown in Fig. 8, the scoring criteria 801a surrounds the "scenery map" in the area of ​​the intermediate calculation criteria 601. This is then defined as one group of information (=cluster) to be evaluated, and if there is a description that follows this content, it is defined that a score of 2 out of 10 points will be given. In this embodiment, the "scene diagram" includes all drawings (line diagrams, graphs, etc.) used in carrying out the calculations, and in Figure 12 described below, where it is written "scene diagram" for convenience, the drawings used in carrying out the calculations are displayed.

[0134] 8, the scoring criteria 801b surrounds the "intermediate formula" and "calculation result (calculation)" in the area of ​​the intermediate calculation criteria 601, and defines this as one group of information (=cluster) to be evaluated. If there is a description in line with this content, it is defined that an evaluation of 3 points for the "intermediate formula" and 2 points for the "calculation result (calculation)" will be given out of a maximum of 10 points.

[0135] Furthermore, on the display screen shown in FIG. 8, the marking criteria 801c surrounds the "answer numbers" and "answer units" that are within the area of ​​the answer extraction criteria 501 or in the vicinity thereof.

[0136] The information within the enclosed area is considered to be one group of information (= cluster) to be evaluated, and if the description is in line with this content, 2 points out of 10 will be given for the "answer number," and if the "answer unit" is also correct, an additional point will be given.

[0137] The scoring board 802 displayed on the screen is a board for defining the details of the clusters defined by the scoring criteria 801a to 801c. The numbers "1"* to "5"* are the numbers of the clusters.

[0138] (*Since the circled numbers in Figure 8 cannot be used in the specification, they are instead expressed as "Circle 1", "Circle 2", ... "Circle 6". The same applies to the following explanations.) Using the board, you can define at least the number, item name, and point allocation, and it is also possible to assign multiple evaluation items to one cluster. Details on setting, changing, and deleting items will be described later in Figure 9.

[0139] A back button 409 in FIG. 8 is a button for accepting an instruction from the user to transition the display screen to the display screen of FIG. 7, which accepts an instruction to scan an example answer.

[0140] Further, a forward button 410 in FIG. 8 is a button for accepting an instruction from the user to transition to the display screen of FIG. 10, which accepts an instruction to scan the answer sheet on which the answers have been written on the display screen.

[0141] FIG. 9 is a configuration diagram showing an example of setting, changing, and deleting items of the scoring criteria.

[0142] FIG. 9(a) is a configuration diagram showing two points, the grading criteria 801a and the scoring board 802, extracted from FIG.

[0143] The delete item button 901 is a button for accepting an instruction from the user to delete an item from a cluster defined by the scoring criteria 801a.

[0144] If there is only one item, the specification may be such that an error message is displayed to inform the user that the item cannot be deleted because there is only one item, or the specification may be such that the scoring criteria 801a is deleted as received.

[0145] The add item button 902 is a button that accepts an instruction from the user to add an item to the cluster defined by the scoring criteria 801a. Instructions can also be accepted in the same way for 801b and 801c.

[0146] In Fig. 9(a), when the add item button 902 is pressed, the screen transitions to Fig. 9(b), and the item "2" linked to the scoring criteria 801a is added to the scoring criteria 801a and the scoring board 802. The screen transitions to Fig. 9(b).

[0147] In this case, the items linked to grading criteria 801b, "Mark 2 Intermediate Formula" and "Mark 3 Calculation", and the items linked to grading criteria 801c, "Mark 4 Number" and "Mark 5 Unit", will have their numbers moved down to "Mark 3 Intermediate Formula", "Mark 4 Calculation", "Mark 5 Number", and "Mark 6 Unit".

[0148] Furthermore, in Figure 9(b), when a click on an added item is received, the selection items are pulled down as shown in Figure 9(c), and it becomes possible to receive an item selection from the user (item input may also be received).

[0149] Furthermore, when a click on the score is accepted in Figure 9(c), the second cell from the top of the score column becomes active, as shown in Figure 9(d), and the score can be entered. In this case, "2" was entered.

[0150] 9(d), when the item deletion button 901 is pressed, the item marked with a circle 2 is deleted, and the state transitions to the same state as that shown in FIG. 9(a).

[0151] In this case, the items linked to grading criteria 801b, "Mark 3 Intermediate Formula" and "Mark 4 Calculation", and the items linked to grading criteria 801c, "Mark 5 Number" and "Mark 6 Unit", will have their numbers moved up to "Mark 2 Intermediate Formula", "Mark 3 Calculation", "Mark 4 Number", and "Mark 5 Unit".

[0152] FIG. 10 is a diagram showing an example of a screen that accepts an instruction from an examinee to scan an answer sheet on which answers have been written.

[0153] The screen shown in Figure 10, like the screen shown in Figure 3, accepts instructions from the user to acquire image data using the image data acquisition button 301, but differs from Figure 3 in that it accepts instructions to acquire image data with answers already written on it.

[0154] A back button 409 is a button for accepting an instruction from the user to transition to the display screen of FIG. 8, which accepts settings related to the grading criteria.

[0155] FIG. 11 is a diagram showing an example of a screen for accepting corrections regarding information to be evaluated.

[0156] In Figure 11, answer extraction criteria 501, intermediate calculation extraction criteria 601, and student ID number extraction criteria 602 are displayed as auxiliary displays, but these auxiliary displays may be omitted or may be configured to be switchable on / off.

[0157] Cluster identification displays 1101a to 1101c are clusters that have been identified as targets for evaluation based on the answer extraction criteria 501 and the intermediate calculation extraction criteria 601.

[0158] For example, the cluster identification display 1101a is an object for identifying and displaying the cluster to be evaluated, which is defined as a scene diagram based on the grading criteria 801a among the information handwritten near the intermediate calculation extraction criteria 601 on the blank answer sheet shown in Figure 8.

[0159] In addition, the cluster identification display 1101b is a cluster to be evaluated, which is defined as the intermediate formula and its calculation result based on the grading criteria 801b among the information handwritten (drawn) around the intermediate calculation extraction criteria 601 on the blank answer sheet shown in Figure 8.

[0160] In addition, the cluster identification display 1101c is a cluster to be evaluated, which is defined as the "numbers" and "units" of the answer based on the scoring criteria 801c among the information handwritten (drawn) around the answer extraction criteria 501 on the blank answer sheet shown in Figure 8.

[0161] Here, the answer column 405, the answer extraction criteria 501, and the cluster identification display 1101c will be further described.

[0162] In this embodiment, the answer extraction criteria 501 are set so as to overlap the answer column 405, and in FIG. 11, the handwritten answer to be evaluated is written outside the answer column 405.

[0163] If handwritten answers are extracted from image data in this state using the method of Patent Document 1 based on the "answer column shape information," the image that is extracted will simply be the image inside the answer column 405, i.e., the difficult-to-read image shown in i) of Figure 14(b), making evaluation difficult.

[0164] On the other hand, the extraction criteria 501 function as a "criteria" for extracting information to be evaluated, and therefore, clusters to be evaluated can be extracted from the surrounding vicinity.

[0165] Therefore, as shown in Figure 11, even if part of the handwritten answer extends beyond the extraction criteria 501, the cluster identification display 1101c can extract the cluster to be evaluated based on the answer extraction criteria 501 in an appropriate form that does not interfere with the evaluation.

[0166] Then, based on the answer extraction criteria 501, clusters to be evaluated can be extracted in a suitable form that does not impede the evaluation, and another example will be described later with reference to FIG.

[0167] 11, the score board 802 displays the evaluation (score) of the cluster identified by the cluster identification display 1101 calculated based on the answer extraction criteria 501 and intermediate calculation extraction criteria 601.

[0168] Specifically, for example, the cluster of cluster identification display 1101a is determined to correspond to the contents of scoring criteria 801a using techniques such as OCR and machine learning algorithms, and the answer as a scene diagram is determined to be appropriate.

[0169] The "Mark 1 Scenery Drawing" is assigned two points.

[0170] Similarly, the cluster of cluster identification display 1101b is determined to correspond to the contents of scoring criteria 801b using techniques such as OCR and machine learning algorithms, and the intermediate formulas and their calculations are determined to be appropriate.

[0171] Three points are allocated to the "2nd mark intermediate formula" and two points to the "3rd mark calculation" (correctness of the calculation in the intermediate formula).

[0172] Similarly, the cluster in cluster identification display 1101c is determined to correspond to the content of scoring criteria 801c using techniques such as character recognition and machine learning algorithms, and it is determined that ``4 numbers in a circle'' (numbers in the answer) is appropriate, and ``5 units in a circle'' (units in the answer) is inappropriate.

[0173] An answer with "4 numbers in a circle" is assigned 2 points, and an answer with "5 units in a circle" is assigned 0 points.

[0174] More preferably, similar to the scoring criteria 801 in FIG. 8, the cluster identification display 1101 is provided so that the range of the automatically assigned clusters can be changed at the user's discretion by accepting the operation of a handle, for example.

[0175] The attendance number identification display 1102 is a cluster identified as a student ID number based on the student ID number extraction criteria 602. The part marked with an "*" in the attendance number identification display 1102 indicates that the handwritten characters have been read as "0002" through video analysis.

[0176] More preferably, the evaluation of each cluster is reflected in advance in a program of an information processing device, and the grader can double-check and confirm or correct the content, as shown in the scoring board 802 of Fig. 11. In this embodiment, this can be implemented as shown in Fig. 12.

[0177] This is because it is expected to reduce the grading burden and reduce grading errors compared to evaluation by a grader alone.

[0178] The student ID number display counter 1103 is a counter that displays the student ID number read by the attendance number identification display 1102 in association with the image data of the answer sheet on which the student ID number is displayed.

[0179] The counter forward button 1104 is a button that, when pressed by the user, receives an instruction to switch the display of image data of the answer sheet linked to the student ID number read by the attendance number identification display 1102 .

[0180] In the state of Figure 11, if the "△" button is pressed, the display will switch to the image data of the answer sheet for the previous student ID number, that is, student ID number 0001 (page 5 / 6).

[0181] Also, in the state of FIG. 11, if the "▽" button is pressed, the display will switch to the image data of the answer sheet for the next student ID number, that is, student ID number 0003 (page 5 / 6).

[0182] A back button 409 in FIG. 11 is a button for accepting an instruction from the user to transition to the display screen of FIG. 10, which accepts an instruction to scan the answer sheet on which the answers have been written.

[0183] Further, a forward button 410 in FIG. 11 is a button for accepting an instruction from the user to transition to the display screen of FIG. 12, which accepts grading and correction of the evaluation target.

[0184] FIG. 12 is a diagram showing an example of a screen for accepting marks and corrections for evaluation targets.

[0185] The intermediate calculation evaluation area 1201 is an area where the cluster extracted as the object of evaluation and the cluster of the answer example are displayed side by side.

[0186] In FIG. 12, the clusters (handwritten scene diagrams and intermediate calculation formulas) extracted as the targets of evaluation are the clusters defined by the cluster identification displays 1101a and 1101b read in FIG. 10 and determined in FIG. In addition, in FIG. 12, the clusters of the example answers (scenery diagrams and intermediate calculation formulas) are the clusters that are read in FIG. 7 and defined by the grading criteria 801a and 801b in FIG.

[0187] When arranging the answers side by side, as in the example of "10 + 20 = 30 minutes" in Figure 13, it is more preferable to use technology such as OCR or machine learning algorithms to provide a function that allows adjustment of the order and scale so that the handwritten answers correspond to the sample answers.

[0188] The answer evaluation area 1202 is an area that displays the clusters defined by the cluster identification display 1101c that were read in Figure 10 and determined in Figure 11 in the same way as the intermediate calculation evaluation area 1201, arranged in order of student ID number, and ready to accept grading from the user.

[0189] In addition, multiple "clusters extracted as evaluation targets" with different student ID numbers corresponding to the same page (pages 5 / 6 in FIG. 12) are displayed in the answer evaluation area 1202. However, if there are multiple questions on the same page, the numbers 401 of the main questions or the numbers of each sub-question may be used as separators.

[0190] Furthermore, among the multiple clusters with different student ID numbers displayed in the answer evaluation area 1202, the cluster displaying the answer corresponding to the student ID number for which the handwritten scene diagram or intermediate calculation formula is displayed in the intermediate calculation evaluation area 1201 is displayed in an identifiable manner. (In this example, the outer frame is highlighted with a bold frame, indicating that it is "identifiable.") As a modification of this example, the intermediate calculation evaluation area 1201 does not necessarily have to be an integrated area as shown in FIG. 12, and may be displayed as a pop-up or separate window, for example.

[0191] This is because providing it in this way has the advantage of allowing the answer evaluation area 1202 to be wider.

[0192] The evaluation check 1203 is an object that accepts from the user an evaluation of the clusters (handwritten scene diagram, intermediate calculation formula, and answer) that are the subject of evaluation and are displayed in the intermediate calculation evaluation area 1201 and the answer evaluation area 1202. By default, the evaluation check 1203 reflects the results displayed on the scoring board in FIG.

[0193] Note that the evaluation check 1203 in FIG. 12 is configured to determine whether or not the answer is correct using a check box, but this is merely one example of a mechanism for accepting an evaluation input from the user and determining the evaluation.

[0194] For example, in cases where more detailed scoring is required from the grader, such as in long-form written questions in Japanese language, various existing input acceptance means can be used to suit the type of evaluation required for the answers to the questions, such as by configuring the system to accept numerical input in the range of 0 to 10 points.

[0195] The answer display area 1204 is an area (display area) where clusters (handwritten answers in this case) corresponding to the student ID numbers assigned to the answer evaluation areas 1202 are displayed.

[0196] The clusters displayed in the answer display areas 1204 are arranged so that the display position and scale can be adjusted automatically or manually so that they fit within the answer display areas 1204.

[0197] The automatic position and scale adjustment function for handwritten answers will be described later in FIG. 13, and the manual position and scale adjustment function (adjusted in response to user instructions) will be described later in FIG. 14.

[0198] The evaluation object 1205 is an object that classifies the results of the evaluation into three levels and displays them to the user in an identifiable manner.

[0199] In Figure 12, a perfect answer is displayed as "○", a partial score is displayed as "△", and a complete error is displayed as "×".

[0200] Furthermore, in the case of "△" where partial points have been obtained, the score of the marking result given in the evaluation check 1203 is also displayed on the bottom right side so that the total partial points can be identified.

[0201] The complete answer button 1206 is a button for receiving an evaluation from the user that the answer to be evaluated is a complete answer. Therefore, when the user presses this button, the evaluation object 1205 becomes "○".

[0202] The strikethrough 1207 is an object that is automatically or manually (installed upon receiving a user's request) superimposed on the handwritten answer displayed in the answer display area 1204 to identify the incorrect answer.

[0203] For example, in Figure 12, the answer for student ID number 0001 is "30m," but the correct answer is in minutes, not meters, so the incorrect "m" is crossed out.

[0204] Additionally, the answer for student ID number 0004 is "25 minutes," but the correct answer is "30 minutes," so the incorrect "25" is crossed out.

[0205] The unit correct answer notation 1208 is an object that identifiably displays the correct answer regarding units among the incorrect answer portions of the answer.

[0206] The numerical correct answer notation 1209 is also an object that identifiably displays the correct numerical answer among the incorrect answer portions of the answer.

[0207] It is preferable that if the answer (units, numbers, descriptions, drawings, etc.) is incorrect, the answer or an example answer is written in the area inside the triangle.

[0208] This is because by surrounding important points with triangular objects like this, it becomes easier for graders and users who receive graded answer sheets to recognize the points that need attention.

[0209] The complete correct answer notation 1210 is an object that can identify an example answer, and is displayed under the answer with a strikethrough 1207 superimposed if the answer displayed in the answer display area 1204 is incorrect in both the number and the unit.

[0210] Furthermore, the complete correct answer notation 1210 is an object that can identify the example answer displayed in the blank answer display area 1204 when nothing is written.

[0211] In this way, if a handwritten answer that is not correct is entered in any of the unit correct answer notation 1208, number correct answer notation 1209, and complete correct answer notation 1210, a strikethrough 1207 is displayed below the superimposed answer.

[0212] Furthermore, when no correct answer is entered in the unit correct answer notation 1208, the number correct answer notation 1209, or the complete correct answer notation 1210, an example answer is displayed in the blank answer display area 1204 or in the margin of the answer display area 1204.

[0213] This is done so that the grader or the user who receives the graded answer sheet can clearly see the details of the evaluation, such as whether the answer was given but there were incorrect parts and therefore no points were earned, or whether the answer was not given and therefore no points were earned.

[0214] Also, if the numbers are correct but the units are missing, it is preferable to display the cluster left-justified rather than centered, so that there is enough space to write the correct unit notation 1208, as shown in the examples for student ID numbers 0002 and 0008.

[0215] By setting it up in this way, the grader or the user who receives the graded answer sheet can easily see and understand that the answer was correct but the credits were forgotten.

[0216] The evaluation object 1205 and the strikethrough 1207 are examples of objects that can identify the evaluation of the evaluation target, and the unit correct answer notation 1208, the numeric correct answer notation 1209, and the complete correct answer notation 1210 are examples of sample answers, that is, additional information that adds to the evaluation.

[0217] These objects can be reused to the benefit of test takers.

[0218] Specifically, this involves generating a "graded answer sheet image for return."

[0219] First, the images of the "answer columns and clusters" in the image of the completed answer sheet are identified.

[0220] Then, this is replaced with the object displayed in the answer evaluation area 1202 corresponding to the student ID number, and a "graded answer sheet image to be returned" is generated.

[0221] The objects referred to here are the "answer display area 1204 and cluster" and the "evaluation object 1205, strikethrough 1207, unit correct answer notation 1208, numeric correct answer notation 1209, and complete correct answer notation 1210."

[0222] An example will be given based on Figures 11 and 12. In the image of the completed answer sheet for student ID number 0002 displayed on answer sheet display screen 406 in Figure 11, the answer column 405 overlapping with answer extraction criteria 501 and the cluster surrounded by cluster identification display 1101c are the data to be replaced (deleted).

[0223] Then, the answer display area 1204 displayed in the answer evaluation area 1202 of the student ID number 0002 in FIG. 12 is placed as replacement data so as to overlap the position of the answer column 405.

[0224] Furthermore, the handwritten characters "30", the correct unit answer notation 1208 "minutes", and the evaluation object 1205 (triangular object) are arranged as replacement data in the same layout and scale as the answer display area 1204 of the answer evaluation area 1202 of student ID number 0002 in Figure 12.

[0225] This image data is generated as a "graded answer sheet image for return" to be fed back to each examinee as test results.

[0226] There are no particular restrictions on the specific method of providing feedback. For example, the feedback may be sent to the examinee's terminal, displayed on a display device, or printed out and handed over on paper.

[0227] As described above, by making the information displayed in the answer evaluation area 1202 in Figure 12 reusable as replacement information, test takers can receive feedback in the form of image data in which the answers have been corrected to make them easier to see, known as a "graded answer sheet image for return."

[0228] A back button 409 in FIG. 12 is a button for accepting an instruction from the user to transition to the display screen of FIG. 11, which accepts corrections regarding the information to be evaluated.

[0229] FIG. 13 is a configuration diagram showing an example of automatic adjustment of the display position and scale of the evaluation object.

[0230] The configuration diagram above the arrow in Figure 13(a) shows that the answer extraction criteria 501, cluster identification display 1101c, and the clusters (handwritten answers) to be evaluated defined by the cluster identification display 1101c are extracted from Figure 11 and displayed in the answer display area 1204.

[0231] In addition, the configuration diagram below the arrow in Figure 13(a) is an example of an embodiment of the present invention, in which the answer display area 1204 for student ID number 0002, the evaluation object 1205, and the unit correct answer notation 1208 are extracted from Figure 12.

[0232] In this way, in FIG. 13( a ), clusters (handwritten answers) to be evaluated, which are defined by the cluster identification display 1101 c based on the answer extraction criteria 501 , are extracted and displayed in the answer display area 1204 .

[0233] This handwritten answer defined by the cluster identification display 1101c is written so that the right and bottom sides extend beyond the answer extraction criteria 501. Therefore, in order to display this answer in the answer display area 1204 in an easy-to-read manner, it is necessary to adjust the scale and position.

[0234] Then, the information processing device's calculation process identifies that the answer does not have units and adjusts the scale so that it fits within the frame of answer display area 1204. The handwritten answer is displayed left-justified to leave a margin on the right, and a unit correct answer notation 1208 is displayed to the right of that. The evaluation object 1205 is also displayed so that the unit correct answer notation 1208 is located inside the area.

[0235] FIG. 13(b) is a configuration diagram showing an embodiment of student ID number 0004.

[0236] In Figure 13(b), the configuration diagram above the arrow shows the answer extraction criteria 501 and the cluster identification display 1101c, and the clusters (handwritten answers) to be evaluated defined by the cluster identification display 1101c are extracted and displayed in the answer display area 1204. In other words, it is the same as Figure 13(a).

[0237] In addition, the configuration diagram below the arrow in Figure 13(b) is an embodiment of the present invention, in which the answer display area 1204 for student ID number 0004, the evaluation object 1205, the strikethrough 1207, and the numerical correct answer notation 1209 are extracted from Figure 12.

[0238] In this way, in FIG. 13(b), the cluster (handwritten answer) of student ID number 0004 defined by the cluster identification display 1101c is extracted based on the answer extraction criteria 501. In addition, the configuration diagram below the arrow in Figure 13(b) is an example of an embodiment of the present invention, in which the answer display area 1204 for student ID number 0004, the evaluation object 1205, the strikethrough 1207, and the numerical correct answer notation 1209 are extracted from Figure 12.

[0239] In this way, in FIG. 13(b), the cluster (handwritten answer) of student ID number 0004 defined by the cluster identification display 1101c is extracted based on the answer extraction criteria 501.

[0240] Then, in order to fit within the frame of the answer display area 1204, the information processing device performs calculations to identify that the units are correct but the numbers are incorrect, and the score is evaluated as 0 points for both the numbers and units (because it is not appropriate to add points even if only the units are correct).The handwritten answer is then displayed centered, and a strikethrough 1207 is superimposed on the incorrect answer "25," with the correct numerical answer notation 1209 displayed below it.

[0241] FIG. 13(c) is a configuration diagram showing an embodiment of student ID number 0005.

[0242] In Figure 13(c), the configuration diagram above the arrow shows the answer extraction criteria 501 and cluster identification display 1101c, and the clusters (handwritten answers) to be evaluated defined by the cluster identification display 1101c are extracted and displayed in the answer display area 1204. In other words, it is the same as Figure 13(a) and Figure 13(b).

[0243] 13(c) is an example of an embodiment of the present invention, in which the answer display area 1204 for student ID number 0005 and the evaluation object 1205 are extracted from FIG.

[0244] In this way, in FIG. 13(b), the cluster (handwritten answer) of student ID number 0004 defined by the cluster identification display 1101c is extracted based on the answer extraction criteria 501.

[0245] This handwritten answer defined by the cluster identification display 1101c is written in such a way that the upper half extends beyond the answer extraction standard 501. Therefore, this answer also needs to be displayed with its scale and position adjusted so that it can be easily seen in the answer display area 1204.

[0246] Then, the information processing device performs calculations to identify whether the answer's numbers and units are correct so that it fits within the frame of the answer display area 1204, and the answer is scored as 2 points for the numbers and 1 point for the units, and the handwritten answer is displayed centered along with the evaluation object 1205.

[0247] FIG. 14 is a configuration diagram showing an example of manual adjustment of the display position and scale of the evaluation target.

[0248] FIG. 14(a) is a modified example of the answer evaluation area 1202 for student ID number 0002 in FIG.

[0249] The difference is in the following two buttons:

[0250] When pressed, the first number assignment button 1401 superimposes a strikethrough 1207 on the number if there is a handwritten answer in the answer display area 1204, as in the example of student ID number 0004 in Figure 12, and displays a numerical correct answer notation 1209 below it.

[0251] Furthermore, when the number assignment button 1401 is pressed, if there is no handwritten answer in the answer display area 1204, a complete correct answer notation 1210 is written in the answer display area 1204, as in the case of student ID number 0003 in FIG.

[0252] When pressed, the second credit assignment button 1402 displays a strikethrough 1207 over the credit portion if there is a handwritten answer in the answer display area 1204, as in the example of student ID number 0001 in Figure 12, and displays a correct credit notation 1208 below it.

[0253] In addition, when the credit assignment button 1402 is pressed, if the handwritten answer in the answer display area 1204 does not have credits, the button writes the correct credit answer notation 1208 in the margin to the right of the handwritten answer in the answer display area 1204, as in the example of student ID number 0002 in Figure 12.

[0254] FIG. 14(b)i) is a configuration diagram showing only the answer display area 1204 portion of FIG. 14(a).

[0255] When the user's selection in the answer display area 1204 is accepted in Figure 14(b)i), the cluster to be evaluated (answer cluster 1403) extracted in the cluster identification display 1101 in Figure 11 is displayed as shown in Figure 14(b)ii).

[0256] At this time, the part that protrudes from the answer display area 1204 is displayed in a identifiable manner when the answer cluster 1403 accepts the selection and becomes active. For example, in Figure 14(b)ii), the part that does not protrude is displayed in a dark color, and the part that protrudes is displayed in a light color.

[0257] This answer cluster 1403 has a handle that can be operated by the user to manually adjust the scale and position.

[0258] For example, FIG. 14(b) iii) shows an example of how the answer cluster 1403 is dragged from ii) so that the answer display area 1204 and the upper left corner of the answer cluster 1403 correspond to each other.

[0259] Also, Figure 14(b) iii) to vi) show an example of how the scale is changed by moving the handle so that the bottom side of the answer display area 1204 corresponds to the bottom right corner of the answer cluster 1403.

[0260] On the other hand, these actions are realized by double-clicking in the transition from Figure 14(b) ii) to vii). In Figure 14(b) ii), when the user double-clicks on the answer cluster 1403, the processes in Figure 14(b) ii) to vi) are automatically executed.

[0261] Figure 14(c) is a configuration diagram showing an example of when, in the state of Figure 14(b) vi) or vii), the evaluation of answer cluster 1403 is accepted by evaluation check 1203 and the user presses number assignment button 1401.

[0262] FIG. 15 is a block diagram showing an example of the main flow of the present invention.

[0263] In step S1501, the information processing device 120 acquires image data of a blank answer sheet from the image data providing device 110.

[0264] In step S1502, the information processing device 120 receives from the user whether or not the image data acquired from the image data providing device 110 in step S1501 is acceptable.

[0265] In step S1503, the information processing device 120 sets the answer extraction criteria 501 using the image data acquired from the image data providing device 110 in step S1501.

[0266] In step S1504, the information processing device 120 sets the intermediate calculation extraction criteria 601 and the student ID number extraction criteria 602 using the image data acquired from the image data providing device 110 in step S1501.

[0267] In step S1505, the information processing device 120 acquires image data of an answer sheet including sample answers from the image data providing device 110.

[0268] In step S1506, the information processing device 120 sets a cluster that will be the scoring criterion, using the image data acquired from the image data providing device 110 in step S1505.

[0269] In step S1507, the information processing device 120 accesses the image data providing device 110 and acquires image data of the answer sheet on which the answers have been written.

[0270] In step S1508, the information processing device 120 accepts corrections related to the information (cluster) to be evaluated, based on the image data acquired from the image data providing device 110 in step S1507.

[0271] In step S1508, the information processing device 120 accepts marks and corrections for the evaluation target based on the image data acquired from the image data providing device 110 in step S1507.

[0272] 16 to 19 are block diagrams showing an example of a subflow of the present invention.

[0273] FIG. 16(a) is a configuration diagram showing an example of the flow of step S1501 in FIG.

[0274] In step S1601, the information processing device 120 accepts from the user a device that sends a request to acquire image data of a blank answer sheet.

[0275] In step S1602, the information processing device 120 determines whether or not the request was accepted in step S1601. If it is determined that the request was accepted, the process proceeds to step S1603, and if it is determined that the request was not accepted, the process proceeds to step S1601.

[0276] In step S1603, the information processing device 120 sends a request to acquire and transmit image data to the device accepted in step S1601.

[0277] In the flow of this figure, only the case where a request is sent to the scanner 111 will be described.

[0278] This is because when the information processing device 120 sends a request to another device to obtain or send image data, a similar flow occurs in which the receiving device sends image data to the information processing device 120 in response to the request.

[0279] In step S1604, the scanner 111 accepts the request to acquire and transmit the image data sent by the information processing device 120 in step S1603.

[0280] In step S1605, the scanner 111 determines whether or not the request was accepted in step S1604. If it is determined that the request was accepted, the process proceeds to step S1606, and if it is determined that the request was not accepted, the process proceeds to step S1604.

[0281] In step S1606, the scanner 111 acquires image data of the blank answer sheet based on the request received in step S1604.

[0282] In step S1607, the scanner 111 transmits the image data acquired in step S1606 to the information processing device 120.

[0283] In step S1608, the information processing device 120 receives the image data of the blank answer sheet sent by the scanner 111 in step S1607.

[0284] FIG. 16(b) is a configuration diagram showing an example of the flow of step S1502 in FIG.

[0285] In step S1609-1, the information processing device 120 extracts question numbers (main questions and sub-questions) from the image data of the blank answer sheet received in step S1608, and generates a data table for managing the question numbers (main questions and sub-questions).

[0286] In step S1609-2, the information processing device 120 accepts pressing of the back button 409.

[0287] In step S1610, the information processing device 120 determines whether or not pressing of the back button 409 was accepted in step S1609. If it is determined that it was accepted, the process proceeds to step S1601, and if it is determined that it was not accepted, the process proceeds to step S1611.

[0288] In step S1611, the information processing device 120 accepts pressing of the forward button 410.

[0289] In step S1612, the information processing device 120 determines whether or not pressing the proceed button 410 in step S1611 has been accepted. If it is determined that the pressing has been accepted, the step ends, and if it is determined that the pressing has not been accepted, the process proceeds to step S1609.

[0290] FIG. 16(c) is a block diagram showing an example of the flow of step S1503 in FIG.

[0291] In step S1613, the information processing device 120 uses image recognition to determine the areas to be identified as answer columns in the image data of the blank answer sheet acquired in step S1608, and sets the answer extraction criteria 501.

[0292] A further explanation of this step will be provided.

[0293] In this embodiment, first, the answer columns on the blank answer sheet are identified.

[0294] Then, answer extraction criteria 501 are set at positions corresponding to these answer columns.

[0295] Here, these processes are expressed as a single step.

[0296] In step S1614, the information processing device 120 accepts a request to change the position or range of the answer extraction criteria 501 set in step S1613 through the user's operation of the handle.

[0297] In step S1615, the information processing device 120 determines whether or not a request to change the position or range of the answer extraction criterion 501 has been received in step S1614. If it is determined that the request has been received, the process proceeds to step S1616, and if it is determined that the request has not been received, the process proceeds to step S1617.

[0298] In step S1616, the information processing device 120 performs processing to change the position and range of the answer extraction criteria 501 based on the change request received in step S1615.

[0299] In step S1617, the information processing device 120 accepts that the back button 409 has been pressed.

[0300] In step S1618, the information processing device 120 determines whether or not pressing of the back button 409 was accepted in step S1617. If it is determined that the pressing of the back button 409 was accepted, the process proceeds to step S1609, and if it is determined that the pressing of the back button 409 was not accepted, the process proceeds to step S1619.

[0301] In step S1619, the information processing device 120 accepts pressing of the forward button 410.

[0302] In step S1620, the information processing device 120 determines whether or not pressing the forward button 410 in step S1619 has been accepted. If it is determined that the pressing has been accepted, the step ends, and if it is determined that the pressing has not been accepted, the process proceeds to step S1614.

[0303] FIG. 17(a) is a configuration diagram showing an example of the flow of step S1504 in FIG.

[0304] In step S1701, the information processing device 120 uses image recognition to determine the areas to be identified as answer columns in the image data of the blank answer sheet acquired in step S1608, and sets intermediate calculation extraction criteria 601 and student ID number extraction criteria 602.

[0305] In step S1702, the information processing device 120 accepts a request to change the position or range of the intermediate calculation extraction criteria 601 or the student ID number extraction criteria 602 set in step S1701 through the user's handle operation.

[0306] In step S1703, the information processing device 120 determines whether or not a request to change the position or range of the intermediate calculation extraction criteria 601 or the student ID number extraction criteria 602 was accepted in step S1702. If it is determined that the request was accepted, the process proceeds to step S1704, and if it is determined that the request was not accepted, the process proceeds to step S1705.

[0307] In step S1704, the information processing device 120 performs processing to change the positions and ranges of the intermediate calculation extraction criteria 601 and the student ID number extraction criteria 602 based on the change request received in step S1702.

[0308] In step S1705, the information processing device 120 accepts pressing of the back button 409.

[0309] In step S1706, the information processing device 120 determines whether or not pressing of the back button 409 was accepted in step S1705. If it is determined that it was accepted, the process proceeds to step S1613, and if it is determined that it was not accepted, the process proceeds to step S1707.

[0310] In step S1707, the information processing device 120 accepts pressing of the forward button 410.

[0311] In step S1708, the information processing device 120 determines whether or not pressing of the forward button 410 in step S1619 has been accepted. If it is determined that the pressing of the forward button 410 has been accepted, the step ends, whereas if it is determined that the pressing of the forward button 410 has not been accepted, the process proceeds to step S1702.

[0312] FIG. 17(b) is a configuration diagram showing an example of the flow of step S1505 in FIG.

[0313] In step S1709, the information processing device 120 accepts pressing of the back button 409.

[0314] In step S1710, the information processing device 120 determines whether or not pressing the back button 409 was accepted in step S1709. If it is determined that the pressing was accepted, the process proceeds to step S1702, and if it is determined that the pressing was not accepted, the process proceeds to step S1711.

[0315] Here, steps S1711 to S1718 will be explained together.

[0316] The flow of steps S1711 to S1718 in Fig. 17(b) is similar to the flow of Fig. 16(a), except that image data of an answer sheet containing sample answers is scanned instead of image data of a blank answer sheet, and that the process proceeds to step S1702 when the back button 409 is pressed.

[0317] Therefore, detailed explanation of steps S1711 to S1718 will be omitted.

[0318] FIG. 18(a) is a configuration diagram showing an example of the flow of step S1506 in FIG.

[0319] In step S1801, the information processing device 120 uses image recognition to determine the portions of the image data of the answer sheet containing the sample answers acquired in step S1718 that should be identified as answer examples. Then, it sets the grading criteria 801, defines clusters that will be the grading criteria, and displays the grading criteria 801.

[0320] In step S1802, the information processing device 120 automatically calculates the item name and score for each cluster based on the clusters defined in the scoring criteria 801 set in step S1801, and sets and displays the scoring board 802.

[0321] In step S1803, the information processing device 120 accepts a request to change the position or range of the scoring criteria 801 set in step S1801 through the user's operation of the handle.

[0322] In step S1804, the information processing device 120 determines whether or not a request to change the position or range of the scoring criteria 801 was received in step S1803. If it is determined that the request was received, the process proceeds to step S1805, and if it is determined that the request was not received, the process proceeds to step S1806.

[0323] In step S1805, the information processing device 120 performs processing to change the position and range of the scoring criteria 801 based on the change request received in step S1803.

[0324] In step S1806, the information processing device 120 accepts pressing of the back button 409.

[0325] In step S1807, the information processing device 120 determines whether or not pressing the back button 409 was accepted in step S1806. If it is determined that the pressing was accepted, the process proceeds to step S1709, and if it is determined that the pressing was not accepted, the process proceeds to step S1808.

[0326] In step S1808, the information processing device 120 accepts pressing of the forward button 410.

[0327] In step S1809, the information processing device 120 determines whether or not pressing the proceed button 410 in step S1808 has been accepted. If it is determined that the pressing has been accepted, the step ends, and if it is determined that the pressing has not been accepted, the process proceeds to step S1802.

[0328] FIG. 18(b) is a configuration diagram showing an example of the flow of step S1507 in FIG.

[0329] Here, steps S1810 to S1718 will be explained together.

[0330] The flow of steps S1810 to S1819 in Figure 18(b) is similar to the flow and diagram in Figure 17(b), except that image data of an answer sheet with answers written on it is scanned, rather than image data of a blank answer sheet or an answer sheet containing sample answers.

[0331] Therefore, detailed explanation of steps S1810 to S1819 will be omitted.

[0332] FIG. 19(a) is a configuration diagram showing an example of the flow of step S1508 in FIG.

[0333] In step S1901, the information processing device 120 extracts clusters to be evaluated (graded) from the image data of the answer sheet including the sample answers acquired in step S1819 by image recognition of the portions to be identified as sample answers based on the grading criteria 801. Then, a cluster identification display 1101 is displayed.

[0334] A further explanation of this step will be provided.

[0335] In this embodiment, the cluster extraction rule to be evaluated (scored) is one in which "at least a part of the cluster (cluster identification display 1101) relates to the answer extraction criteria 501," as shown in Figures 5 and 23 above.

[0336] Therefore, in this step, the handwritten characters written on the answer sheet are identified as multiple clusters. Then, based on the answer extraction criteria 501 and intermediate calculation extraction criteria 601 defined in a part of the answer sheet area, clusters that meet the extraction rules are extracted as evaluation targets (score targets).

[0337] If a cluster extracted based on the answer extraction criteria 501 also falls under the intermediate calculation extraction criteria 601, the cluster extracted based on the answer extraction criteria 501 is extracted as handwritten characters written in the answer field. Therefore, although the cluster falls under the intermediate calculation extraction criteria 601, processing is performed to exclude it from the targets for extraction as handwritten characters for intermediate calculations.

[0338] Take Figures 11 and 12 as examples. The cluster in cluster identification display 1101c (handwritten characters "30") is a cluster that falls under both answer extraction criteria 501 and intermediate calculation extraction criteria 601. However, it is extracted as handwritten characters written in the answer field, and is displayed as "30" in answer display area 1204 for student ID number 0002 in Figure 12.

[0339] In other words, although the cluster "30" is a cluster that falls under the intermediate calculation extraction criteria 601, it is extracted as handwritten characters written in the answer column. Therefore, it is excluded from the targets to be extracted as handwritten characters for intermediate calculations, and "30" is not displayed in the intermediate calculation evaluation area 1201.

[0340] In step S1902, the information processing device 120 executes automatic scoring for each cluster based on the clusters extracted by the cluster identification display 1101 set in step S1901, and displays the scoring board 802 reflecting the results.

[0341] In step S1903, the information processing device 120 accepts a request to change the position or range of the cluster identification display 1101 set in step S1901 through the user's operation of the handle.

[0342] In step S1904, the information processing device 120 determines whether or not a request to change the position or range of the cluster identification display 1101 was accepted in step S1903. If it is determined that the request was accepted, the process proceeds to step S1905, and if it is determined that the request was not accepted, the process proceeds to step S1806.

[0343] In step S1905, the information processing device 120 performs processing to change the cluster identification display 1101 or the range based on the change request received in step S1803.

[0344] In step S1906, the information processing device 120 accepts pressing of the back button 409.

[0345] In step S1907, the information processing device 120 determines whether or not pressing of the back button 409 was accepted in step S1806. If it is determined that it was accepted, the process proceeds to step S1810, and if it is determined that it was not accepted, the process proceeds to step S1908.

[0346] In step S1908, the information processing device 120 accepts pressing of the forward button 410.

[0347] In step S1809, the information processing device 120 determines whether or not pressing the proceed button 410 in step S1808 has been accepted. If it is determined that the pressing has been accepted, the step ends, and if it is determined that the pressing has not been accepted, the process proceeds to step S1902.

[0348] FIG. 19(b) is a block diagram showing an example of the flow of step S1509 in FIG.

[0349] In step S1910, the information processing device 120 reads each answer and its evaluation and example answer from the data table.

[0350] Here, the sample answer is the sample answer acquired in step S1819.

[0351] Each answer is a cluster for each student ID number extracted based on the student ID number extraction criteria 602 acquired in steps S1901 and S1905.

[0352] Moreover, "the evaluation" refers to the result of the scoring for each cluster that was performed in steps S1902 and S1905 (the evaluation was determined).

[0353] In step S1911, the information processing device 120 arranges and displays the clusters read in step S1901 while adjusting the location and scale of each cluster so that they are lined up in the predetermined locations as described with reference to FIG.

[0354] In step S1912, the information processing device 120 receives an instruction from the user to change the answer evaluation area 1202 to an active state.

[0355] In step S1913, the information processing device 120 determines whether or not an instruction to change the answer evaluation area 1202 to an active state has been received from the user in step S1912. If it is determined that the instruction has been received, the process proceeds to step S1914, and if it is determined that the instruction has not been received, the process proceeds to step S1915.

[0356] In step S1914, the information processing device 120 changes the answer evaluation area 1202 to be activated based on the change request received in step S1912. Also, the information processing device 120 changes the display of the intermediate calculation evaluation area 1201 so that the cluster corresponding to the student registration code to be changed is displayed.

[0357] In step S1915, the information processing device 120 receives an instruction from the user to change the arrangement and scale of the clusters displayed in each answer display area 1204 through the process shown in FIG. 13 or FIG.

[0358] In step S1916, the information processing device 120 determines whether or not an instruction to change the arrangement and scale of the clusters displayed in each answer display area 1204 has been received from the user in step S1915. If it is determined that the instruction has been received, the process proceeds to step S1917, and if it is determined that the instruction has not been received, the process proceeds to step S1918. In step S1917, the information processing device 120 performs processing to change the arrangement and scale of the clusters displayed in the answer display area 1204 based on the change request received in step S1915.

[0359] In step S1918, the information processing device 120 receives an instruction from the user via the evaluation check 1203 or the complete answer button 1206 to change the evaluation of the cluster that is the evaluation target.

[0360] In step S1919, the information processing device 120 determines whether or not an instruction to change the evaluation of the cluster being evaluated has been received from the user in step S1918. If it is determined that the instruction has been received, the process proceeds to step S1920, and if it is determined that the instruction has not been received, the process proceeds to step S1921.

[0361] In step S1920, the information processing device 120 performs processing to change the evaluation of the cluster that is the evaluation target, based on the change request received in step S1918.

[0362] In step S1921, the information processing device 120 accepts pressing of the back button 409.

[0363] In step S1922, the information processing device 120 determines whether or not pressing of the back button 409 was accepted in step S1921. If it is determined that the pressing of the back button 409 was accepted, the process proceeds to step S1903, and if it is determined that the pressing of the back button 409 was not accepted, the process proceeds to step S1923. In step S1923, the information processing device 120 accepts pressing of the forward button 410. In step S1824, the information processing device 120 determines whether or not pressing of the forward button 410 in step S1923 has been accepted. If it is determined that the pressing of the forward button 410 has been accepted, the step ends, and if it is determined that the pressing of the forward button 410 has not been accepted, the process proceeds to step S1912.

[0364] 20 and 21 are diagrams showing an example of a data table according to the present invention.

[0365] FIG. 20(a) is a diagram showing an example of a data table for managing image data for each page.

[0366] The data table in FIG. 20(a) manages image data for each page of the blank answer sheet acquired in step S1608 in FIG. 16 upon receiving a press of the image data acquisition button 301 in FIG.

[0367] 17. The data table in FIG. 20(a) manages image data for each page of the answer sheet, including the sample answer, acquired in step S1718 in FIG. 17 upon pressing of the image data acquisition button 301 in FIG.

[0368] 18. The data table in FIG. 20(a) receives the press of the image data acquisition button 301 in FIG. 10 and manages the student ID number and image data for each page of the completed answer sheet acquired in step S1819 in FIG.

[0369] Page 2001 is stored and managed by acquiring image data in page units in steps S1608, S1718, and S1819 in FIG.

[0370] More preferably, it would be better to provide information on the answer sheets that can identify the pages, so that even if the pages are disjointed when scanned, they can be correctly organized and stored in the data table.

[0371] The student ID number 2002 is extracted from the completed answer sheet obtained in step S1912 of FIG. 18 based on the student ID number extraction criteria 602, and is then stored and managed.

[0372] The image data name 2003 is determined and managed by assigning a name corresponding to the type of image data, page, and student ID number when the image data is acquired in steps S1608, S1718, and S1819.

[0373] FIG. 20(b) is a data table that manages the answer extraction criteria 501, intermediate calculation extraction criteria 601, and student ID number extraction criteria 602 for each page of image data defined in FIG. 5 and FIG. When image data of the main questions (parts) 2004 and sub-questions (questions) 2005 is acquired, the question number 401 is read using technology such as OCR, and the data is stored and managed.

[0374] For main question 2004 and sub-question 2005, it is sufficient to classify the divisions of the questions appropriately, so it is assumed that a system that can be read according to the content being dealt with can be provided, and the type of numbers or symbols, such as A, B, C, α, β, γ, or Roman numerals, is not particularly important.

[0375] The three extraction criteria 2006 are stored, changed, and managed when the answer extraction criteria 501 are set or changed in step S1613 or step S1616 in Fig. 16. They are also stored, changed, and managed when the intermediate calculation extraction criteria 601 and the student ID number extraction criteria 602 are set or changed in step S1701 or step S1704 in Fig. 17.

[0376] FIG. 21(a) is a diagram showing an example of a data table for managing information related to sample answers.

[0377] The answer example number 2101, answer example item 2102, answer example point allocation 2103, and answer example file name 2104 store and manage the answers extracted in steps S1901 and S1905 and the evaluations of the answers evaluated or changed in steps S1902 and S1920.

[0378] FIG. 22 is a configuration diagram showing an example of another embodiment for appropriately extracting information to be evaluated without using the answer extraction criteria 501.

[0379] The upper figures of Figures 22(a), 22(b), and 22(c) correspond to Figures 13(a), 13(b), and 13(c), respectively, with the basic same configuration, but differ in that the answer extraction criteria 501 are not superimposed and set in the set answer column 405.

[0380] In such a configuration, for example, first, the cluster identification display 1101 is designated as the starting point (point P, point P', point P'') by clicking the mouse pointer as shown in FIG. 22. Then, the mouse pointer is dragged to the position (point Q, point Q', point Q'') that is desired to be designated as the ending point, and the click is released. By using such a range designation method and individually designating each piece of image data for each student ID number, the user can arbitrarily extract information to be evaluated.

[0381] In other words, by configuring the input for specifying the cluster identification display 1101 range to be received from the user in a portion of the area of ​​the image data for each student ID number, the user can freely extract the handwritten answer image to be evaluated.

[0382] Furthermore, since the extraction area can be set according to the position and character size of the answer, which differ for each examiner, the test can be extracted in a way that is easy for the grader to grade.

[0383] Furthermore, the input of the cluster identification display 1101 to be applied collectively to multiple image data with answers written on it can be configured to be accepted in a part of the area of ​​one of the multiple image data. This makes it possible to recognize and reuse the same position and range of the accepted input when grading answer sheets of different examinees.

[0384] The concepts of the above two embodiments can be applied not only to the answer extraction criteria 501 but also to the setting of the intermediate calculation extraction criteria 601 in the same manner.

[0385] The above describes a configuration in which, without using the answer extraction criteria 501 or intermediate calculation extraction criteria 601, the user inputs a designation of an area for extracting information to be evaluated contained in image data, by accepting the designation input into a part of the image data area (designation input acceptance).

[0386] The configuration other than this condition is the same as the configuration explained with reference to FIGS. 1 to 21, and can deal with the problems that could not be solved by Prior Art 1.

[0387] An example of an embodiment has been described above.

[0388] In addition, a recording medium on which a program that realizes the functions of the above-mentioned embodiment is recorded may be supplied to a system or device, and the computer (or CPU or MPU) of that system or device may read and execute the program stored on the recording medium.

[0389] As a recording medium for supplying the program, at least the following can be used.

[0390] Specifically, examples include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EPROMs, silicon disks, and solid state drives.

[0391] Furthermore, it goes without saying that not only are the functions of the above-mentioned embodiments realized by the computer executing a program it has read, but also cases are included in which an OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the program, and the functions of the above-mentioned embodiments are realized through that processing.

[0392] Furthermore, it goes without saying that this also includes cases where a program read from a recording medium is written into a memory provided on a function expansion board inserted into a computer or a function expansion unit connected to the computer, and then a CPU or the like provided on the function expansion board or function expansion unit performs some or all of the actual processing based on the instructions of the program code, thereby realizing the functions of the above-mentioned embodiments.

[0393] Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. Note that the present invention also includes configurations that combine the above-mentioned embodiments and their modified examples. [Explanation of symbols]

[0394] 120 Information processing equipment 501 Answer extraction criteria 801 Scoring Criteria Area 1204 Answer display area

Claims

1. acquiring means for acquiring one or more image data; a criterion determining means for determining a first criterion for extracting first information to be evaluated from the image data acquired by the acquiring means as a part of an area of ​​the image data; an extraction means for extracting, from the image data, first information that is located at a predetermined position with respect to the first reference and that is to be evaluated, using information about the position of the first reference determined by the reference determination means; a display means for displaying the first information extracted by the extraction means in a predetermined display area for displaying the first information; The information processing device further comprises an evaluation input means for inputting an evaluation of the first information displayed in the predetermined display area by the display means.

2. 2. The information processing device according to claim 1, wherein the display means is configured to adjust the position or scale of the display of the first information in the predetermined display area so that each piece of first information fits within the respective predetermined display area.

3. the evaluation means determines additional information to be displayed on the display means, the additional information being determined in response to each of the first information together with each of the first information; 2. The information processing apparatus according to claim 1, wherein the display means displays the additional information added by the evaluation determining means together with each of the first information.

4. The information processing device according to claim 3, characterized in that the additional information includes at least one of an object that can identify the evaluation of the first information determined by the evaluation determination means, and information that adds to the evaluation of the first information.

5. the standard determination means determines a second standard for extracting second information to be evaluated from the image data acquired by the acquisition means, in a part of an area of ​​the image data; the extraction means extracts second information to be evaluated from the image data in the vicinity of the second reference determined by the reference determination means, excluding the first information; the display means displays the one or more pieces of second information extracted by the extraction means in respective predetermined display areas for displaying the second information; 2. The information processing apparatus according to claim 1, wherein the evaluation determining means determines an evaluation of each of the second information items displayed in the predetermined display area by the display means.

6. acquiring one or more image data; a criterion determination step of determining a first criterion for extracting first information to be evaluated from the image data acquired in the acquisition step as a part of an area of ​​the image data; an extraction step of extracting, from the image data, first information that is located at a predetermined position with respect to the first reference and that is to be evaluated, using information about the position of the first reference determined by the reference determination means; a display step of displaying the first information extracted in the extraction step in a predetermined display area of ​​a display means for displaying the first information; An information processing method comprising: an evaluation input step of inputting an evaluation of the first information displayed in the predetermined display area in the display step.

7. A program for causing a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 5.

Images